solr shingle query matching keyword tokenized field

0
votes

I'm using Solr 5.5 and I have problem that I'm hoping to find a solution here.

I have a field that I've created using the below setting:

<field name="exactName_noAlias_en_US" type="text_exact_query_tokenized" indexed="true" stored="false"/>

<fieldtype name="text_exact_query_tokenized" class="solr.TextField" positionIncrementGap="100">
       <analyzer type="index">
            <tokenizer class="solr.KeywordTokenizerFactory"/> 
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.ASCIIFoldingFilterFactory" preserveOriginal="true"/>
         </analyzer>
          <analyzer type="query">
            <tokenizer class="solr.WhitespaceTokenizerFactory"/>                    
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.ASCIIFoldingFilterFactory" preserveOriginal="true"/>
            <filter class="solr.ShingleFilterFactory" maxShingleSize="10"/>
         </analyzer>
    </fieldtype>

This field can have value like: "Justin Bieber"

And my expectation is as follows:

For a query "The artist Justin Bieber is a teen heartthrob", I'd like it to match this document. And queries like "An artist named Bieber Justin is canadian" or "The name Justin is so common" aren't supposed to find a match.

I see that using the default '/select' request handler isn't finding the match when I issue a query "Justin Bieber" even though it is an exact match. But a different field was set as the default field for the '/select' RH, so, I tried to create another RH using the below curl:

curl http://localhost/solr/performer/config -H 'Content-type:application/json'  -d '{"add-requesthandler" : {"name": "/exactName","class":"solr.SearchHandler","defaults":{ "echoParams":"explicit" ,"rows":10, "df":"exactName_noAlias_en_US", "q.op":"AND" },"useParams":"x"}}'

It created the RH I wanted but my query still didn't match the required document.

Kindly suggest a solution to this problem.

Here is a screenshot from the Analysis screen.

Below is a snippet of the 'debug' section of the response for query : "/exactName?q=exactName_noAlias_en_US:Justin%20Bieber&wt=json&indent=true&debug=true"

"debug":{
    "rawquerystring":"exactName_noAlias_en_US:Justin Bieber",
    "querystring":"exactName_noAlias_en_US:Justin Bieber",
    "parsedquery":"+exactName_noAlias_en_US:justin +exactName_noAlias_en_US:bieber",
    "parsedquery_toString":"+exactName_noAlias_en_US:justin +exactName_noAlias_en_US:bieber",
    "explain":{},

And below is the snippet of the 'debug' section of the response for query : "/select?q=exactName_noAlias_en_US:Justin%20Bieber&wt=json&indent=true&debug=true"

"debug":{
    "rawquerystring":"exactName_noAlias_en_US:Justin Bieber",
    "querystring":"exactName_noAlias_en_US:Justin Bieber",
    "parsedquery":"+exactName_noAlias_en_US:justin +searchKeywords_en_US:bieber",
    "parsedquery_toString":"+exactName_noAlias_en_US:justin +searchKeywords_en_US:bieber",
    "explain":{},

And below is the snippet of the 'debug' section of the response for the phrase query with /select RH: "/select?q=exactName_noAlias_en_US:"Justin%20Bieber"&wt=json&indent=true&debug=true

"debug":{
    "rawquerystring":"exactName_noAlias_en_US:\"Justin Bieber\"",
    "querystring":"exactName_noAlias_en_US:\"Justin Bieber\"",
    "parsedquery":"MultiPhraseQuery(exactName_noAlias_en_US:\"(justin justin bieber) bieber\")",
    "parsedquery_toString":"exactName_noAlias_en_US:\"(justin justin bieber) bieber\"",
    "explain":{},

And below is the snippet of the 'debug' section of the response for the phrase query with /exactName RH: "/exactName?q=exactName_noAlias_en_US:"Justin%20Bieber"&wt=json&indent=true&debug=true

"debug":{
"rawquerystring":"exactName_noAlias_en_US:\"Justin Bieber\"",
"querystring":"exactName_noAlias_en_US:\"Justin Bieber\"",
"parsedquery":"MultiPhraseQuery(exactName_noAlias_en_US:\"(justin justin bieber) bieber\")",
"parsedquery_toString":"exactName_noAlias_en_US:\"(justin justin bieber) bieber\"",
"explain":{},

Below is the query and the corresponding debug section with the whitespace in the query escaped:

select?q=Justin\ Beiber&df=exactName_noAlias_en_US

Debug:

"rawquerystring":"Justin\\ Beiber",
"querystring":"Justin\\ Beiber",
"parsedquery":"+((exactName_noAlias_en_US:justin exactName_noAlias_en_US:justin beiber)/no_coord) +exactName_noAlias_en_US:beiber",
"parsedquery_toString":"+(exactName_noAlias_en_US:justin exactName_noAlias_en_US:justin beiber) +exactName_noAlias_en_US:beiber",
"explain":{},
2
solrtokenizeexact-matchshingles
Have you looked at the output under the "Analysis" page under the Admin page? That would allow you to enter both the indexed content and the query, and see which tokens are generated on both sides and whether you get the same on both sides. For debugging supplying the default field or the query fields in the URLs are usually easier than changing or adding request handlers as well. - MatsLindh
Yes, I did. I see exactly what I wanted in that tool. But I don't see the same behaviour when I issue the query via http using the /select RH or the custom one '/exactName' that I mentioned above. - oveflown
Could you include the output from the analysis stage? Also remember that any changes to the index part of an analyzer will require reindexing that document. - MatsLindh
@MatsLindh, I included the screenshot of the Analysis for a sample query. And yes, I've reindexed my collection after having made the configuration changes. - oveflown

2 Answers

0
votes

The parsedquery debug information for the last entry shows that you're searching in two different fields - one is the field you provided, the other is the default search field you've set earlier. As neither have a single token with the content you're searching in each field, you don't get a hit. This is also the reason why your analysis shows a hit, but you don't get a hit when you're searching - the actual query isn't the same as you provided for the analysis page.

+exactName_noAlias_en_US:justin +searchKeywords_en_US:bieber"
 ^^^^^^^^^^^^^^^^^^^^^^^         ^^^^^^^^^^^^^^^^^^^^

The first entry shows that you're searching for two tokens, then requesting the intersection of those two sets of documents - as neither contains that single token, you don't get a hit.

Searching for exactName_noAlias_en_US:"Justin Bieber" might give you the answer you're looking for.

0
votes

In order to use shingles, you have to prevent parser to parse query and apply default operator where space is. That can be done by escaping space. Possible queries are:

  • q=exactName_noAlias_en_US:(Justin\ Bieber)
  • q=Justin\ Bieber&df=exactName_noAlias_en_US
  • q=Justin\ Bieber (if you have df defined in your handler)

Few additional notes:

  • there is discrepancy within field naming - in your config you define field 'exactMatch', you define default field 'exactName' and you query 'exactName_noAlias_en_US'
  • you don't need new query handler to override df - you can provide that in your query
  • you don't need df if you explicitly provide field name