ElasticSearch has far less docs than what I added

1
votes

I am adding 11378 documents to an index in ElasticSearch. But the number of documents it shows in the stats is only only 225. However the index_total under indexing has the correct number, that is 11378. When I search for a particular word, it returns 13 hits (docs). When I use a like query in SQL server, I am getting 178 records(docs). I am not understanding what I am doing wrong. I added the documents to the index, first with PUT command, and then later with POST. Bot the HTTP methods led to the same stats.

Can someone please explain what is happening? Any links or points are appreciated.

Thanks.

1
elasticsearch
How do you define the IDs of your documents? - Val
I have concatenated 3 integer values to generate an unique id. I have checked whether there are dupes in them through a group by query in SQL and I did not find any. - dotcoder
Can you share the client code you're using to index your documents? - Val
I have stopped generating the ID and let the ES do it itself. I then checked the docs which is 11378. So there seem to be something wrong with the way I am generating the ID. The search yielded 191 hits which is good. - dotcoder
Glad you figured it out! When you have less docs than what you think, the most common issue is that the ID is not generated correctly and you have collisions. - Val

1 Answers

2
votes

When you have less docs than what you think, the most common issue is that the ID is not generated correctly and you have collisions.

So you have two solutions:

  1. make sure that the IDs you're generating are unique
  2. let ES generate its own IDs