How do I update my deprecated custom score query with a function score query?

0
votes

I'm upgrading from elasticsearch 0.90.10 and NEST 0.12.0 to elasticsearch 1.3.4 and NEST 1.1.2, and I'm trying to replace a custom score query, which was deprecated in 0.90.4, with a function score query.

This is my old code:

descriptor.Search<IndexedProduct>(s => s
    .Index(index)
    .From(((page ?? 1) - 1) * paging)
    .Size(paging)
    .Fields(f => f.ID, f => f.NumberOfSalesLastTwelveMonths, f => f.Description, f => f.itemIsInStock)
    .Query(qsc => qsc
        .CustomScore(sc => sc
            .Query(q => q
                .QueryString(qs => qs
                    .OnFieldsWithBoost(f => f
                        .Add(entry => entry.Description, 2.0)
                        .Add(entry => entry.LongDescription, 1.0)
                        .Add(entry => entry.ShortFamilyAndNumber, 4.0)
                        .Add(entry => entry.FamilyAndNumber, 4.0)
                        .Add(entry => entry.Suppliers, 1.5)
                        .Add(entry => entry.CleanSupplierReferences, 4.0)
                        .Add(entry => entry.SupplierReferences, 4.0)
                        .Add(entry => entry.SupplierBrands, 2.0)
                        .Add(entry => entry.Categories, 1.5)
                        .Add(entry => entry.ParameterValues, 1.0)
                        .Add(entry => entry.KeyWords, 2.0)
                        .Add(entry => entry.EANReferencesWMS, 4.0)
                    )
                    .Query(CleanUpText(EscapeSearchString(searchquery)))
                    .Operator(Operator.and)
                )
            )
            .Script("_score + ((_score > 0.99 ) ? doc['numberOfSalesLastTwelveMonths'].value : 0) + ((_score > 0.99 ) ? doc['itemIsInStock'].value : 0)")
        )
    ));

This is my new code:

descriptor.Search<IndexedProduct>(s => s
    .Index(index)
    .From(((page ?? 1) - 1) * paging)
    .Size(paging)
    .Fields(f => f.ID, f => f.NumberOfSalesLastTwelveMonths, f => f.Description, f => f.itemIsInStock)
    .Query(qsc => qsc
        .FunctionScore(fs => fs
            .Query(q => q
                .QueryString(qs => qs
                    .OnFieldsWithBoost(f => f
                        .Add(entry => entry.Description, 2.0)
                        .Add(entry => entry.LongDescription, 1.0)
                        .Add(entry => entry.ShortFamilyAndNumber, 4.0)
                        .Add(entry => entry.FamilyAndNumber, 4.0)
                        .Add(entry => entry.Suppliers, 1.5)
                        .Add(entry => entry.CleanSupplierReferences, 4.0)
                        .Add(entry => entry.SupplierReferences, 4.0)
                        .Add(entry => entry.SupplierBrands, 2.0)
                        .Add(entry => entry.Categories, 1.5)
                        .Add(entry => entry.ParameterValues, 1.0)
                        .Add(entry => entry.KeyWords, 2.0)
                        .Add(entry => entry.EANReferencesWMS, 4.0)
                    )
                    .Query(CleanUpText(EscapeSearchString(searchquery)))
                    .DefaultOperator(Operator.And)
                )
            )
            .ScriptScore(sc => sc
                .Script("_score + ((_score > 0.99 ) ? doc['numberOfSalesLastTwelveMonths'].value : 0) + ((_score > 0.99 ) ? doc['itemIsInStock'].value : 0)")
            )
        )
    )
);

The new code fails with a SearchParseException, this is the query (from the .log file):

[{
    "from": 0,
    "size": 10,
    "fields": ["iD", "numberOfSalesLastTwelveMonths", "description", "itemIsInStock"],
    "query": {
        "function_score": {
            "query": {
                "query_string": {
                    "query": "*",
                    "fields": ["description^2", "longDescription^1", "shortFamilyAndNumber^4", "familyAndNumber^4", "suppliers^1.5", "cleanSupplierReferences^4", "supplierReferences^4", "supplierBrands^2", "categories^1.5", "parameterValues^1", "keyWords^2", "eANReferencesWMS^4"],
                    "default_operator": "and"
                }
            },
            "script_score": {
                "script": "_score + ((_score > 0.99 ) ? doc['numberOfSalesLastTwelveMonths'].value : 0) + ((_score > 0.99 ) ? doc['itemIsInStock'].value : 0)"
            }
        }
    }
}]

Full log entry:

[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7a87107e] lastShard [true]
org.elasticsearch.search.SearchParseException: [sta_products_public_nl_635490239970034081][0]: from[0],size[10]: Parse Failure [Failed to parse source [{"from":0,"size":10,"fields":["iD","numberOfSalesLastTwelveMonths","description","itemIsInStock"],"query":{"function_score":{"query":{"query_string":{"query":"*","fields":["description^2","longDescription^1","shortFamilyAndNumber^4","familyAndNumber^4","suppliers^1.5","cleanSupplierReferences^4","supplierReferences^4","supplierBrands^2","categories^1.5","parameterValues^1","keyWords^2","eANReferencesWMS^4"],"default_operator":"and"}},"script_score":{"script":"_score + ((_score > 0.99 ) ? doc['numberOfSalesLastTwelveMonths'].value : 0) + ((_score > 0.99 ) ? doc['itemIsInStock'].value : 0)"}}}}]]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:660)
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:516)
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:488)
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
        at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
        at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
        at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.index.query.QueryParsingException: [sta_products_public_nl_635490239970034081] script_score the script could not be loaded
        at org.elasticsearch.index.query.functionscore.script.ScriptScoreFunctionParser.parse(ScriptScoreFunctionParser.java:97)
        at org.elasticsearch.index.query.functionscore.FunctionScoreQueryParser.parse(FunctionScoreQueryParser.java:128)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:239)
        at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:342)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:268)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:263)
        at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:644)
        ... 9 more
Caused by: org.elasticsearch.script.ScriptException: dynamic scripting for [mvel] disabled
        at org.elasticsearch.script.ScriptService.verifyDynamicScripting(ScriptService.java:374)
        at org.elasticsearch.script.ScriptService.compile(ScriptService.java:335)
        at org.elasticsearch.script.ScriptService.search(ScriptService.java:509)
        at org.elasticsearch.index.query.functionscore.script.ScriptScoreFunctionParser.parse(ScriptScoreFunctionParser.java:94)
        ... 16 more

If I omit the FunctionScore like this, it works fine:

descriptor.Search<IndexedProduct>(s => s
    .Index(index)
    .From(((page ?? 1) - 1) * paging)
    .Size(paging)
    .Fields(f => f.ID, f => f.NumberOfSalesLastTwelveMonths, f => f.Description, f => f.itemIsInStock)
    .Query(q => q
        .QueryString(qs => qs
            .OnFieldsWithBoost(f => f
                .Add(entry => entry.Description, 2.0)
                .Add(entry => entry.LongDescription, 1.0)
                .Add(entry => entry.ShortFamilyAndNumber, 4.0)
                .Add(entry => entry.FamilyAndNumber, 4.0)
                .Add(entry => entry.Suppliers, 1.5)
                .Add(entry => entry.CleanSupplierReferences, 4.0)
                .Add(entry => entry.SupplierReferences, 4.0)
                .Add(entry => entry.SupplierBrands, 2.0)
                .Add(entry => entry.Categories, 1.5)
                .Add(entry => entry.ParameterValues, 1.0)
                .Add(entry => entry.KeyWords, 2.0)
                .Add(entry => entry.EANReferencesWMS, 4.0)
            )
            .Query(CleanUpText(EscapeSearchString(searchquery)))
            .DefaultOperator(Operator.And)
        )
    )

This is the query (captured via Wireshark, I couldn't get the log/slowlog configured to log all queries):

{
    "index": "sta_products_public_nl",
    "type": "indexedproduct"
}
{
    "from": 0,
    "size": 10,
    "fields": ["iD", "numberOfSalesLastTwelveMonths", "description", "itemIsInStock"],
    "query": {
        "query_string": {
            "query": "*",
            "fields": ["description^2", "longDescription^1", "shortFamilyAndNumber^4", "familyAndNumber^4", "suppliers^1.5", "cleanSupplierReferences^4", "supplierReferences^4", "supplierBrands^2", "categories^1.5", "parameterValues^1", "keyWords^2", "eANReferencesWMS^4"],
            "default_operator": "and"
        }
    }
}

What am I doing wrong?

2
elasticsearchnest
a very easy way to log all queries while developing is to use ipv4.fiddler instead of localhost while fiddler is running. – Martijn Laarman
It would be helpful to see the java SearchParseException that is returned in the response for .Search() – Martijn Laarman
@MartijnLaarman I've added the full log entry. – user247702

2 Answers

2
votes

Ok the big hint is here (should have clicked with me immediately to be honest :))

Caused by: org.elasticsearch.script.ScriptException: dynamic scripting for [mvel] disabled

Elasticsearch used to enable scripting by default but since folks were exposing elasticsearch to the world (equivelent of exposing a db to the world) they were getting bit by this.

We released a blog post detailing how you can re-enable scripting:

http://www.elasticsearch.org/blog/scripting/

1
votes

Adding 

script.disable_dynamic: false

does not fix the problem always, I already have this in my elasticsearch.yml yet I continue to receive the error out put :

Caused by: org.elasticsearch.script.ScriptException: dynamic scripting for [mvel] disabled
    at org.elasticsearch.script.ScriptService.verifyDynamicScripting(ScriptService.java:374)
    at org.elasticsearch.script.ScriptService.compile(ScriptService.java:335)
    at org.elasticsearch.script.ScriptService.search(ScriptService.java:509)
    at org.elasticsearch.index.query.functionscore.script.ScriptScoreFunctionParser.parse(ScriptScoreFunctionParser.java:94)