Azure search indexer indexes entire Cosmos db data source every time

Question

I have Azure Search Index based on top of Cosmos Db. All worked as a charm while my data source was configured without custom query (indexer crawled only new data on each run).

Now I need to define a query like the following:

SELECT c.id, 1 as itemType, c._ts FROM c WHERE c._ts >= @HighWaterMark ORDER BY c._ts

The issue is that each time my indexer runs it collects a complete data source. Data source is 4000 items, the number of new items is 5, data is not changed.

I found that using the following query respects previous progress (indexer works as it did without the query):

SELECT * FROM c WHERE c._ts >= @HighWaterMark ORDER BY c._ts

Also I tried to load @HighWaterMark as one of my fields and found that @HighWaterMark is always -1.

UPDATE

My data is unchangeable. Therefore I didn't use data change detection policy. But enabling this policy makes the indexer work as intended. I haven't found any info about such dependency between policy and indexer work in Microsoft docs.

Answer 1

You can define a new indexer and specify the query the way you need:

more info: https://docs.microsoft.com/en-us/azure/search/search-howto-index-cosmosdb

However, if you want to keep the Azure Cognitive Search Index updated, a better approach would be combine Cosmos Db Change Feed + Azure Functions, then, send the updated / new documents do your Azure Cognitive Search Index.

More info:

https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed-functions

https://azure.microsoft.com/en-us/resources/videos/azure-cosmosdb-change-feed/