App engine - easy text search

Question

I was hoping to implement an easy, but effective text search for App Engine that I could use until official text search capabilities for app engine are released. I see there are libraries out there, but its always a hassle to install something new. I'm wondering if this is a valid strategy:

1) Break each property that needs to be text-searchable into a set(list) of text fragments 2) Save record with these lists added 3) When searching, just use equality filters on the list properties

For example, if I had a record:

{
  firstName="Jon";
  lastName="Doe";
}

I could save a property like this:

{
  firstName="Jon";
  lastName="Doe";

  // not case sensative:
  firstNameSearchable=["j","o", "n","jo","on","jon"];   
  lastNameSerachable=["D","o","e","do","oe","doe"];
}

Then to search, I could do this and expect it to return the above record:

//pseudo-code:
SELECT person 
WHERE firstNameSearchable=="jo" AND
      lastNameSearchable=="oe"

Is this how text searches are implemented? How do you keep the index from getting out of control, especially if you have a paragraph or something? Is there some other compression strategy that is usually used? I suppose if I just want something simple, this might work, but its nice to know the problems that I might run into.

Update:::

Ok, so it turns out this concept is probably legitimate. This blog post also refers to it: http://googleappengine.blogspot.com/2010/04/making-your-app-searchable-using-self.html

Note: the source code in the blog post above does not work with the current version of Lucene. I installed the older version (2.9.3) as a quick fix since google is supposed to come out with their own text search for app engine soon enough anyway.

The solution suggested in the response below is a nice quick fix, but due to big table's limitations, only works if you are querying on one field because you can only use non-equality operators on one property in a query:

db.GqlQuery("SELECT * FROM MyModel WHERE prop >= :1 AND prop < :2", "abc", u"abc" + u"\ufffd")

If you want to query on more than one property, you can save indexes for each property. In my case, I'm using this for some auto-suggest functionality on small text fields, not actually searching for word and phrase matches in a document (you can use the blog post's implementation above for this). It turns out this is pretty simple and I don't really need a library for it. Also, I anticipate that if someone is searching for "Larry" they'll start by typing "La..." as opposed to starting in the middle of the word: "arry". So if the property is for a person's name or something similar, the index only has the substrings starting with the first letter, so the index for "Larry" would just be {"l", "la", "lar", "larr", "larry"}

I did something different for data like phone numbers, where you may want to search for one starting from the beginning or middle digits. In this case, I just stored the entire set of substrings starting with strings of length 3, so the phone number "123-456-7890" would be: {"123","234", "345", ..... "123456789", "234567890", "1234567890"}, a total of (10*((10+1)/2))-(10+9) = 41 indexes... actually what I did was a little more complex in order to remove some unlikely to-be-used substrings, but you get the idea.

Then your query would be: (Pseaudo Code) SELECT * from Person WHERE firstNameSearchIndex == "lar" phonenumberSearchIndex == "1234"

The way that app engine works is that if the query substrings match any of the substrings in the property, then that is counted as a match.

Is inventing your own full text search really simpler than installing a library? Or using google.appengine.ext.db.searchablemodel? — Nick Johnson
Hi Nick, what is google.appengine.ext.db.searchablemodel? I can only find fleeting references to it online. — Chris Dutrow

Drew Sears Drew Sears · Accepted Answer · 2010-10-11T17:32:23

In practice, this won't scale. For a string of n characters, you need n factorial index entries. A 500 character string would need 1.2 * 10^1134 indexes to capture all possible substrings. You will die of old age before your entity finishes writing to the datastore.

Implementations like search.SearchableModel create one index entry per word, which is a bit more realistic. You can't search for arbitrary substrings, but there is a trick that lets you match prefixes:

From the docs:

db.GqlQuery("SELECT * FROM MyModel WHERE prop >= :1 AND prop < :2", "abc", u"abc" + u"\ufffd")

This matches every MyModel entity with a string property prop that begins with the characters abc. The unicode string u"\ufffd" represents the largest possible Unicode character. When the property values are sorted in an index, the values that fall in this range are all of the values that begin with the given prefix.

App engine - easy text search

1 Answers