How to get the total word count per document in SOLR?

Question

I would like to retrieve some summary statistics from the text documents I have indexed in Solr. In particular, the word count per document.

For example, I have the following three documents indexed:

{
 "id":"1",
 "text":["This is the text in document 1"]},
{
 "id":"2",
 "text":["some text in document 2"]},
{
 "id":"3",
 "text":["and document 3"]}

I would like to get the total number of words per each individual document:

"1",7,
"2",5,
"3",3,

What query can I use to get such a result?

I am new to Solr and I am aware that I can use facets to get the count of the individual words over all documents using something like:
http://localhost:8983/solr/corename/select?q=*&facet=true&facet.field=text&facet.mincount=1
But how to get the total word count per document is not clear to me.

I appreciate your help!

How exact do you need it? What do you consider a word count? The number of tokens (which can be greater or less than the number of "words" - depending your definition)? — MatsLindh
Thank you for your comment. You are right, I should have phrased it as term or token count. I want to count the number of tokens per document, generated by the tokenizer assigned to the field in question. — TMS

uylmz uylmz · Accepted Answer · 2020-08-31T12:52:37

If you do a faceted search over id and an inner facet over text, the inner facet count will give the number of words in that document with that id. But text field type must be text_general or something equivalent (tokenized).

If you only want to count "distinct" words per document id, it is actually much easier:

{
"query": "*:*",
"facet": {
    "document": {
        "type": "terms",
        "field": "id",
        "facet": {
            "wordCount": "unique(message)"
        }
    }
}
}

Gives distinct word count per document. Following gives all words and all counts per document but it's up to you to sum them to get total amount (also it's an expensive call)

{
"query": "*:*",
"facet": {
    "document": {
        "type": "terms",
        "field": "id",
        "facet": {
            "wordCount": {
                "type": "terms",
                "field": "message",
                "limit": -1
            }
        }
    }
}
}

@MatsLindth's comment is something to consider too. Solr and you might not agree on what's a "word". Tokenizer is configurable to a point but depending on your needs it might not be very easy.

How to get the total word count per document in SOLR?

1 Answers