Partial text search in GAE datastore (Java)

Question

I am trying to query my (GAE Java) datastore's text field and expecting results including partial matches. For example, results for the search string "test" should include "test, test1, more tests, etc". Below is the Java code snippet I am using.

        DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
        Filter testFilter = new FilterPredicate("test", FilterOperator.GREATER_THAN_OR_EQUAL, testStr);
        Query testQuery = new Query("testEntity").setFilter(testFilter);
        List<Entity> testResults = datastore.prepare(testQuery).asList(FetchOptions.Builder.withLimit(100));

My test results are totally irrelavant. I am new to GAE and Java and I am sure I am not on the right track. I have searched for the right approach in SO and other sites and found solutions for Python, but couldn't find any for Java. The Java examples I came across are all using queries with integer values. Please suggest.

@JohnnyZ, I didn't get a chance to try it out yet. However, from my research so far, this seems to be the best approach. I may be wrong. I will update this post once I try it out. Please share your findings too. — MemoryLeak
Hello, I tried this approach and it worked well. Note that you have to create a document that contains every available searchable item. For instance, to search for a user named Joe Smith, I made a single document taht was "j o e s m i t h jo oe es sm mi it th joe oes esm smi mit ith" etc. — Johnny Z
@JohnnyZ, thanks much for the update. Although it sounds like a tedious and very inefficient way to achieve the goal considering the number of records a datastore can have, I am glad to know that there IS a way to do partial text search. I really hope there is an easier way to do this. Thanks again for trying it out and sharing your findings. — MemoryLeak
Its not as much as you think. One document per search result. See my answer below. — Johnny Z

Johnny Z Johnny Z · Accepted Answer · 2015-03-04T20:42:54

My example is searching for users of my application by name. I created one document for each user, and that document contains every possible string that you can search for.

For example, the document for user "John Smith" has one string: search input delimited by spaces: "joh ohn smi smit mith (etc)".

Here the code I used to get this to work. The "id" is the id for the user in my backend datastore.

private void createSearchableUserDoc(String id, String displayName) {
    List<String> substrings = buildAllSubstrings(displayName);
    String combinedString = combine(substrings, " ");
    // The input for this looks like "CHR CHRI CHRIS HRI HRIS" etc...
    createUserDocument(id, combinedString);
}

private List<String> buildAllSubstrings(String displayName) {
    List<String> substrings = new ArrayList<String>();
    for (String word : displayName.split(" ")) {
        int wordSize = 1;
        while (true) {
            for (int i = 0; i < word.length() - wordSize + 1; i++) {
                substrings.add(word.substring(i, i + wordSize));
            }
            if (wordSize == word.length())
                break;
            wordSize++;
        }
    }
    return substrings;
}

private String combine(List<String> strings, String glue) {
    int k = strings.size();
    if (k == 0)
        return null;
    StringBuilder out = new StringBuilder();
    out.append(strings.get(0));
    for (int x = 1; x < k; ++x)
        out.append(glue).append(strings.get(x));
    return out.toString();
}

private void createUserDocument(String id, String searchableSubstring) {
    Builder docBuilder = Document
            .newBuilder()
            .setId(id)
            .addField(
                    Field.newBuilder().setName("Display_Name")
                            .setText(searchableSubstring));

    addDocToIndex(docBuilder.build());

}

private void addDocToIndex(Document document) {

    Index index = getUserDocIndex();

    try {
        index.put(document);
    } catch (PutException e) {
        log.severe("Error putting document in index... trying again.");
        if (StatusCode.TRANSIENT_ERROR.equals(e.getOperationResult().getCode())) {
            index.put(document);
        }
    }
}

public static Index getUserDocIndex() {
    IndexSpec indexSpec = IndexSpec.newBuilder().setName("USER_DOC_INDEX").build();
    Index index = SearchServiceFactory.getSearchService().getIndex(indexSpec);
    return index;
}

To perform a search I did this:

    Query query = Query.newBuilder().build("Display_Name" + "=" + searchText);

    Index userDocIndex = getUserDocIndex();
    Results<ScoredDocument> matchingUsers = userDocIndex.search(query);

Partial text search in GAE datastore (Java)

2 Answers