I'm having real problems getting a query using field boosting to work with Elasticsearch. I've been through the Nest docs on the topic, but they weren't particularly helpful, so my code is really based on the solution to this question: Elastic Search using NEST Field Boosting.
If I run the following query I get one result, as expected:
var matches =
_client.Search<SearchableMerchant>(
s => s.From((page - 1) * pageSize)
.Size(pageSize)
.QueryString("*test*")
.MinScore(1)
);
However, if I try to use field boosting, using the following, I get no matches:
var matches =
_client.Search<SearchableMerchant>(
s => s.From((page - 1) * pageSize)
.Size(pageSize)
.Query(q => q
.Boosting(bq => bq
.Positive(pq => pq
.CustomScore(cbf => cbf
.Query(cbfq => cbfq
.QueryString(
qs => qs
.OnFieldsWithBoost(d => d
.Add("opportunities.acquirerLocationMID", Math.Pow(2, 17))
.Add("opportunities.amexMID", Math.Pow(2, 16))
.Add("opportunities.epayMID", Math.Pow(2, 16))
.Add("v1MerchantId", Math.Pow(2, 16))
.Add("locatorId", Math.Pow(2, 15))
.Add("opportunities.opportunityLocatorId", Math.Pow(2, 14))
.Add("businessName", Math.Pow(2, 13))
.Add("searchablePhone", Math.Pow(2, 12))
.Add("address.postCodeDetails.postCode.postCode", Math.Pow(2, 11))
.Add("contacts.contact.searchableEmailAddress", Math.Pow(2, 11))
.Add("contacts.contact.searchableMainPhone", Math.Pow(2, 10))
.Add("contacts.contact.searchableMobilePhone", Math.Pow(2, 10))
.Add("contacts.contact.fullName", Math.Pow(2, 9))
.Add("contacts.contact.surname", Math.Pow(2, 8))
.Add("contacts.contact.firstName", Math.Pow(2, 7))
.Add("searchableAddress", Math.Pow(2, 6))
.Add("ownershipUser.username", Math.Pow(2, 5))
.Add("ownershipUser.searchableFullName", Math.Pow(2, 4))
.Add("ownershipUser.lastName", Math.Pow(2, 3))
.Add("ownershipUser.firstName", Math.Pow(2, 2))
.Add("opportunities.depositAccount", Math.Pow(2, 1))
.Add("opportunities.depositIban", Math.Pow(2, 1))
.Add("opportunities.feesAccount", Math.Pow(2, 1))
.Add("opportunities.feesIban", Math.Pow(2, 1))
// TODO: Company registration number - somewhere in legal methinks
)
.Query(
"*test*"
)
)
)
)
)
.Negative(nq => nq
.Filtered(nfq => nfq
.Query(qq => qq.MatchAll())
.Filter(f =>
f.Missing("opportunities.acquirerLocationMID")
&& f.Missing("opportunities.amexMID")
&& f.Missing("opportunities.epayMID")
&& f.Missing("v1MerchantId")
&& f.Missing("locatorId")
&& f.Missing("opportunities.opportunityLocatorId")
&& f.Missing("businessName")
&& f.Missing("searchablePhone")
&& f.Missing("address.postCodeDetails.postCode.postCode")
&& f.Missing("contacts.contact.searchableEmailAddress")
&& f.Missing("contacts.contact.searchableMainPhone")
&& f.Missing("contacts.contact.searchableMobilePhone")
&& f.Missing("contacts.contact.fullName")
&& f.Missing("contacts.contact.surname")
&& f.Missing("contacts.contact.firstName")
&& f.Missing("searchableAddress")
&& f.Missing("ownershipUser.username")
&& f.Missing("ownershipUser.searchableFullName")
&& f.Missing("ownershipUser.lastName")
&& f.Missing("ownershipUser.firstName")
&& f.Missing("opportunities.depositAccount")
&& f.Missing("opportunities.depositIban")
&& f.Missing("opportunities.feesAccount")
&& f.Missing("opportunities.feesIban")
)
)
)
.NegativeBoost(0.01)
)
)
.MinScore(1)
);
I realise this code could be better structured, but right now I just want to get a field boosting query working - I'll tidy it later.
Here are some things I've tried:
The Nest docs are silent on the topic of whether you can use OnFieldsWithBoost with property names. I.e., is this OK?
.OnFieldsWithBoost(d => d .Add("businessName", Math.Pow(2, 13))
As opposed this?
.OnFieldsWithBoost(d => d
.Add(m => m.businessName, Math.Pow(2, 13))
The reason I ask is that I have sub-properties I want to boost that sit within collections. For example, opportunities.opportunityLocatorId. Opportunities obviously being the collection, and I want to match where any object in that collection has a matching value for its opportunityLocatorId field.
This works with fields - you can use a lambda or a string - but does it work with boosting?
No idea, but I've tried it both ways, slimming down the query to just include a boost for businessName, since this is the field that should match the string 'test', but still no results come back.
I've also tried getting rid of the .Negative clause, just in case that was matching something it shouldn't. It's there to deboost any query where no match is found in any of the fields listed in the .Positive clause. Still no results.
I've also upped the .NegativeBoost value to 1 (i.e., no effect, so any results shouldn't be filtered down to a score below 1 that didn't start out with such a low score) but, again, no dice.
Here's the content of my index, just so you can see that the businessName field should match 'test' with the second query, as it does with the first:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "merchantv2",
"_type" : "searchablemerchant",
"_id" : "00000000-0000-0000-0000-000000000000",
"_score" : 1.0,
"_source":{"merchantGuid":"00000000-0000-0000-0000-000000000000","v1MerchantId":0,"locatorId":"0","address":{"addressGuid":"00000000-0000-0000-0000-000000000000","postCodeDetails":{"postCodeKey":0,"postalDistrict":{"postalDistrictKey":0,"postalDistrict":""},"postalLocation":"0","latitude":0.0,"longitude":0.0,"townName":"None","countyKey":0,"countryKey":0,"postCode":{"postCodeKey":0,"postCode":" 0"}},"county":{"countyKey":0,"countyName":"","countryKey":0,"recStatus":3,"countryKeyValue":0},"countryKey":0,"addressTypeKey":0,"updateDate":"0001-01-01T00:00:00+00:00","createdDate":"2016-01-07T19:46:28.4463+00:00"},"searchableAddress":" 0","searchablePhone":"","searchableFax":"","businessName":"","contacts":[],"opportunities":[{"opportunityGuid":"00000000-0000-0000-0000-000000000000","merchantGuid":"00000000-0000-0000-0000-000000000000","location":{"locationGuid":"00000000-0000-0000-0000-000000000000","tradingAddress":{"verified":false,"addressGuid":"00000000-0000-0000-0000-000000000000","postCodeDetails":{"postCodeKey":0,"postalDistrict":{"postalDistrictKey":0,"postalDistrict":""},"postalLocation":"0","latitude":0.0,"longitude":0.0,"townName":"None","countyKey":0,"countryKey":0,"postCode":{"postCodeKey":0,"postCode":" 0"}},"county":{"countyKey":0,"countyName":"","countryKey":0,"recStatus":3,"countryKeyValue":0},"countryKey":0,"addressTypeKey":0,"updateDate":"0001-01-01T00:00:00+00:00","createdDate":"2016-01-07T19:46:28.4463+00:00"}},"opportunityLocatorId":"000000"}]}
}, {
"_index" : "merchantv2",
"_type" : "searchablemerchant",
"_id" : "5f55fe61-ca65-e411-93f3-0cc47a07ef4a",
"_score" : 1.0,
"_source":{"merchantGuid":"5f55fe61-ca65-e411-93f3-0cc47a07ef4a","locatorId":"PM227Z02","address":{"addressGuid":"5c55fe61-ca65-e411-93f3-0cc47a07ef4a","houseNumber":"242","streetName":"Acklam Road","houseName":"","flatAptSuite":"","townName":"London","postCodeDetails":{"postCodeKey":1,"postalDistrict":{"postalDistrictKey":2782,"postalDistrict":"W10"},"postalLocation":"5JJ","latitude":51.52094651,"longitude":-0.20149990,"townName":"London","countyKey":0,"countryKey":224,"postCode":{"postCodeKey":1,"postCode":"W10 5JJ"}},"county":{"countyKey":626,"countyName":"Kensington And Chelsea","countryKey":224,"recStatus":1,"countryKeyValue":224},"countryKey":224,"addressTypeKey":0,"updateDate":"0001-01-01T00:00:00+00:00","createdDate":"2016-01-07T19:46:28.4653+00:00"},"searchableAddress":"242 Acklam Road, London, Kensington And Chelsea, W10 5JJ","searchablePhone":"+44 2031954484","searchableFax":"","businessName":"Test Merchant","contacts":[],"opportunities":[]}
} ]
}
}
I'm using Elasticsearch 1.7.1 and Nest 1.7.1 on Windows 7 (yeah, I know, but it's what the client uses) with .NET 4.5.1.
I've also tried capturing the traffic going between my Web API and elasticsearch, but to no avail. Possibly a config issue but neither Fiddler nor Wireshark/npcap is able to capture the traffic between these two, both running on the local machine, so I can't see the actual request being sent to elasticsearch, which I suspect would be helpful. Basically I was wondering if any error was coming back from Elasticsearch that Nest was swallowing.
Well... the intuition turned out to be correct. Here's a sample of what's appearing in the elasticsearch log file:
[2016-01-08 10:14:01,534][DEBUG][action.search.type ] [Rocket Racer] All shards failed for phase: [query]
org.elasticsearch.search.SearchParseException: [user][4]: from[0],size[20]: Parse Failure [Failed to parse source [{
"from": 0,
"size": 20,
"min_score": 1.0,
"query": {
"boosting": {
"positive": {
"custom_score": {
"query": {
"query_string": {
"query": "*test*",
"fields": [
"opportunities.acquirerLocationMID^131072",
"opportunities.amexMID^65536",
"opportunities.epayMID^65536",
"v1MerchantId^65536",
"locatorId^32768",
"opportunities.opportunityLocatorId^16384",
"businessName^8192",
"searchablePhone^4096",
"address.postCodeDetails.postCode.postCode^2048",
"contacts.contact.searchableEmailAddress^2048",
"contacts.contact.searchableMainPhone^1024",
"contacts.contact.searchableMobilePhone^1024",
"contacts.contact.fullName^512",
"contacts.contact.surname^256",
"contacts.contact.firstName^128",
"searchableAddress^64",
"ownershipUser.username^32",
"ownershipUser.searchableFullName^16",
"ownershipUser.lastName^8",
"ownershipUser.firstName^4",
"opportunities.depositAccount^2",
"opportunities.depositIban^2",
"opportunities.feesAccount^2",
"opportunities.feesIban^2"
]
}
}
}
},
"negative": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"missing": {
"field": "opportunities.acquirerLocationMID"
}
},
{
"missing": {
"field": "opportunities.amexMID"
}
},
{
"missing": {
"field": "opportunities.epayMID"
}
},
{
"missing": {
"field": "v1MerchantId"
}
},
{
"missing": {
"field": "locatorId"
}
},
{
"missing": {
"field": "opportunities.opportunityLocatorId"
}
},
{
"missing": {
"field": "businessName"
}
},
{
"missing": {
"field": "searchablePhone"
}
},
{
"missing": {
"field": "address.postCodeDetails.postCode.postCode"
}
},
{
"missing": {
"field": "contacts.contact.searchableEmailAddress"
}
},
{
"missing": {
"field": "contacts.contact.searchableMainPhone"
}
},
{
"missing": {
"field": "contacts.contact.searchableMobilePhone"
}
},
{
"missing": {
"field": "contacts.contact.fullName"
}
},
{
"missing": {
"field": "contacts.contact.surname"
}
},
{
"missing": {
"field": "contacts.contact.firstName"
}
},
{
"missing": {
"field": "searchableAddress"
}
},
{
"missing": {
"field": "ownershipUser.username"
}
},
{
"missing": {
"field": "ownershipUser.searchableFullName"
}
},
{
"missing": {
"field": "ownershipUser.lastName"
}
},
{
"missing": {
"field": "ownershipUser.firstName"
}
},
{
"missing": {
"field": "opportunities.depositAccount"
}
},
{
"missing": {
"field": "opportunities.depositIban"
}
},
{
"missing": {
"field": "opportunities.feesAccount"
}
},
{
"missing": {
"field": "opportunities.feesIban"
}
}
]
}
}
}
},
"negative_boost": 0.01
}
}
}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.index.query.QueryParsingException: [user] No query registered for [custom_score]
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:303)
at org.elasticsearch.index.query.BoostingQueryParser.parse(BoostingQueryParser.java:63)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:305)
at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:731)
... 9 more
So what am I doing wrong? Does anyone know how to fix the second query, which elasticsearch clearly doesn't like? And is there some way of getting any erorrs out of Nest? I'd expect an exception, but that's not happening - it just returns silently with an empty match collection, and there's no property on the collection that indicates something went wrong.
Any help gratefully received.
Thanks!
Bart
