Lucene 5.0.0 - search string with special characters

2
votes

I am using Lucene version 5.0.0. In my search string, there is a minus character like “test-”. I read that the minus sign is a special character in Lucene. So I have to escape that sign, as in the queryparser documentation:

Escaping Special Characters: Lucene supports escaping special characters that are part of the query syntax. The current list special characters are: 

- + - && || ! ( ) { } [ ] ^ " ~ * ? : \ /`

To escape these character use the \ before the character. For example to search for (1+1):2 use the query:

\(1\+1\)\:2

To do that I use the QueryParser.escape method:

query = parser.parse(QueryParser.escape(searchString));

I use the classic Analyzer because I noticed that the standard Analyzer has some problems with escaping special characters.

The problem is that the Parser deletes the special characters and so the Query has the term content:test

How can I set up the parser and searcher to search for the real value “test-“? I also created my own query with the content test- but that also didn’t work. I recieved 0 results but my index has entries like:

  • Test-VRF
  • Test-IPLS

I am really confused about this problem.

2
lucenespecial-characters

2 Answers

1
votes

While escaping special characters for the queryparser deals with part of the problem, it doesn't help with analysis.

Neither classic nor standard analyzer will keep punctuation in the indexed form of the field. For each of these examples, the indexed form will be in two terms:

  • test and vrf
  • test and ipls

This is why a manually constructed query for "test-" finds nothing. That term does not exist in the index.

The goal of these analyzers is to attempt to index words. As such, punctuation is mostly eliminated, and is not searchable. A phrase query for "test vrf" or "test-vrf" or "test_vrf" are all effectively identical. If that is not what you need, you'll need to look to other analyzers.

1
votes

The goal to fix this issue is to store the value content in an NOT_ANALYZED way. Field fieldType = new Field(key.toLowerCase(),value, Field.Store.YES, Field.Index.NOT_ANALYZED);

Someone who has the same problem has to take care how to store the contents in the index. To request the result create a query in this way searchString = QueryParser.escape(searchString); and use for example a WhitespaceAnalyzer.