Grails lucene search taking inordinate amount of time

Question

For background, I am using Grails v2.2.1 and Searchable plugin (v0.6.4) for my application, although I am a newbie when it comes to configuring Lucene.

Logs show the search taking 26 mili seconds but the compass transaction takes about 15 seconds to return:

2013-04-23 00:40:34,269 DEBUG grails.plugin.searchable.internal. compass.search.DefaultSearchMethod - query: [+kind:band +name:snoop], [4] hits, took [26] millis

2013-04-23 00:40:49,965 DEBUG org.compass.core.transaction.LocalTransaction - Committing local transaction on thread [http-bio-8080-exec-10] Compass [1176020804] Session [2089649487]

This seems to be more an issue with Compass than Lucene as the query executes quickly but the Compass mapping pegs my Java process at near 100% CPU and hangs for way too long.

I have about 3500 domain objects indexed and my domain model looks like the following: I've tried to index only the fields name and id but it seems to map everything in the domain when seen through Luke.

package com.bandbot

class Band {
    def slugGeneratorService
    static searchable = {
        mapping {
            spellCheck "exclude"
            only: ['name', 'id']
        }
    }
    String name
    String biography
    String profileImage
    String slug
    String similarBands // this will store bands/url/pic in the form of Kanye West::url::img.png~Queen::url::img.png
    boolean onTour // is this band currently touring? (Info from lastfm)
    String mbid // This band's MusicBrainz ID see @ http://musicbrainz.org/doc/MusicBrainz_Identifier
    String bandUrl
    String lastFMUrl // stores the lastfm url
    Date dateOfInception
    Date dateDisbanded
    Date lastUpdated

    static belongsTo = [Genre, BandbotUser]

    static hasMany = [ events : Event, genres : Genre ]

    def beforeInsert() {
        lastUpdated = new Date()
        this.slug = slugGeneratorService.generateSlug(this.class, "slug", name)
    }

    def beforeUpdate() {
        lastUpdated = new Date()
        if (isDirty('name')) {
            this.slug = slugGeneratorService.generateSlug(this.class, "slug", name)
        }
    }

    static constraints = {
        name(nullable: false, blank: false, unique: true)
        slug(nullable: true)
        bandUrl(nullable: true)
        dateDisbanded(nullable: true)
        mbid(nullable: true)
        dateOfInception(nullable: true)
        biography(nullable: true)
        similarBands(nullable: true)
        lastUpdated(nullable: true)
        lastFMUrl(nullable: true)
        kind( display: false )
    }
    static mapping = {
        onTour defaultValue: false
        biography type: 'text'
        similarBands type: 'text'
    }

    String toString(){name}

}

My search logic in my controller for bands:

def search() {
    if (!params.q?.trim()) {
        return [:]
    }
    try {
        def searchResult


        if (params.sort) {
            searchResult = searchableService.search(
                    params.q.trim(),
                    [offset: params.offset ? params.int('offset') : 0,
                            max: params.max ? params.int('max') : 10,
                    sort: params.sort, order: params.order? params.order : 'asc']
                    )
        }
        else {
            searchResult = searchableService.search(
                    params.q.trim(),
                    [offset: params.offset ? params.int('offset') : 0,
                            max: params.max ? params.int('max') : 10]
                    )
        }

        return [searchResult: searchResult, params: params]

    } catch (SearchEngineQueryParseException ex) {
        return [parseException: true, params: params]
    }

}

Any ideas would be greatly appreciated. This is for a self-learning project of mine and I really want to do search the right way. :) Thanks, Kevin

Answer 1

I was having the same problem using the searchable plugin on a recent Grails application I was developing. I had two domain objects, with a one to many relationship, that I was indexing to be searched. For simplicity I'm just showing the Domain objects with their fields and relationships. I am not showing any mapping or constraint information. Here are my original classes

class CodeValue{
    static searchable ={
        only:['value', 'description']
        value boost: 2.0
    }
    String value
    String description
    static belongsTo = [codeset: CodeSet]
}
class CodeSet{
    static searchable ={
        only:['name', 'description']
        name boost: 2.0
    }

    String name
    String description
    static hasMany = [codeValues:CodeValue]
}

Searches for CodeValues were taking > 17 seconds. I did have over 1000 CodeValue objects indexed, but a 17 second search time was unacceptable. I figured out what was causing the slow search times and it appeared to be related to the Compass functionality built into the Grails Searchable plugin.

As part of the search, all matched objects are marshalled into the index. For a set of Domain objects in the 100’s the time to perform this marshalling is not too bad, however, when you get into the 1000’s it takes a substantial amount of time. Perhaps the time is also related to complexity of the object being marshalled? Anyway, I found a blog post from a guy having a similar problem as me.

http://webcache.googleusercontent.com/search?q=cache:lebHKgX2yXUJ:blog.hououji.info/archives/165+&cd=10&hl=en&ct=clnk&gl=us

To summarize the article when he was searching 1000+ objects the search time was greater 15 seconds. Just like what I was experiencing.

He mentioned two things:

1)Setting the "supportUnmarshall" option to false, default value is true, in the “static searchable“ configuration in the domain object. By setting this option search matches are not marshalled into the index, however search time is very fast. The down fall to setting this option to false is the results will not contain the objects unmarshalled from the Index, and you have to fetch the matching domain object from the database using the id returned as part of the search result. You would think this would be bad, but it is not, and using this method my search results are actually displayed much faster than before. Here is the URL for the info on setting the supportUnmarshall option http://grails.org/Searchable+Plugin+-+Mapping+-+Class+Mapping. It is the last option in the “Options” section.

2)Enable "reload" in the defaultMethodOptions of the Searchable.groovy configuration file. So put something like this in the Searchable.groovy file:

defaultMethodOptions = [
    search: [reload: true, escape: false, offset: 0, max: 25, defaultOperator: "and"],
    suggestQuery: [userFriendly: true]
]

You will need to add the Searchable Config plugin to update this value. Details for adding and editing the Searchable.groovy config file can be found on the web page for the Grail Searchable Plugin. Since I do not have a high enough reputation. I cannot post more than two links, so you will need to go to the web page for the Grails Searchable plugin, and look up the documentation on how to install the Searchable Config Plugin.

To give an example of the performance improvement. Previously, searches on CodeValues took 17+ seconds to complete. They are now completed in 0.002 seconds.

The final code I wrote looked like this:

class CodeValue{
    static searchable ={
        only:['value', 'description']
        value boost: 2.0
        supportUnmarshall false
    }
    String value
    String description
    static belongsTo = [codeset: CodeSet]
}
class CodeSet{
    static searchable ={
        only:['name', 'description']
        name boost: 2.0
        supportUnmarshall false
    }

    String name
    String description
    static hasMany = [codeValues:CodeValue]
}