Elasticsearch: JSON to get frequency of specific term

0
votes

I think this is a straightforward application, yet I cannot find a recipe on the internet.

Can you suggest a JSON query to send through python to an Elasticsearch instance that would return the frequency of a specific term in a certain field?

I guess it should be possible by some tweak of the Term Vector API, but it seems not straightforward.

I would not mind to get both the absolute frequency and the number of document containing the term.

2
elasticsearchword-frequencyterm-vectors
So there is no direct way? I have to get the docids first and then either count them or aggregate the tf over all of them for each term? – Radio Controlled

2 Answers

1
votes

If you have the ids, you can use Multivectors API https://www.elastic.co/guide/en/elasticsearch/reference/6.8/docs-multi-termvectors.html

curl -X POST "localhost:9200/index/type/_mtermvectors?pretty" -H 'Content-Type: application/json' -d' 
{
    "ids" : ["your_document_id1","your_document_id2"],      
    "parameters": {
        "fields": [
                "your_field"       
        ],
        "term_statistics": true
    }
}
'

You can even pass an artifical document with the terms you want to analyse. As stated here (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-termvectors.html), make sure term_statistics is set up to true, so you can get this info across your index:

  • total term frequency (how often a term occurs in all documents)
  • document frequency (the number of documents containing the current term)
0
votes

There is actually a simple solution, goes like this:

from elasticsearch import Elasticsearch as ES
from copy import deepcopy as copy
import sys

_field = sys.argv[1]
_terms = sys.argv[2:]

_timeout = 60
_gate    = 'some.gate.org/'
_index   = 'some_index'
_client  = ES([_gate],scheme='http',port=80,timeout=_timeout) #or however to get connection

_body= {"doc": {_field: None}, "term_statistics" : True, "field_statistics" : True, "positions": False, "offsets": False}

for term in terms_:
    body   = copy(_body); body["doc"][_field] = term
    result = _client.termvectors(index=_index,body=body)
    print 'documents with', term, ':', result['term_vectors'][_field]['terms'][term]['doc_freq']
    print 'frequency of  ', term, ':', result['term_vectors'][_field]['terms'][term]['ttf']