Azure Data Explorer (ADX) Filtered ingestion

0
votes

It is common that JSON log files contain many types of records together. In a Root Cause Analysis, we often want to explore only a specific type of records, which is just a fraction of the data.

Today, as ADX does not support filtered ingestion, we have no other choice but to repeat the following pattern:

  1. Load all the records to a "raw" table. (Its bad as 90% of that data is junk)

  2. Connect the row table to a derived table with an update policy and with that execute a filter which will get only the records we care about to the derived table.

  3. purge the "raw" table.

  4. Execute the analytics against the derived table.

Is there a plan to support filtered ingestion? Is there a better way to do this, than what we do today?

1
azure-data-explorer

1 Answers

2
votes

you can submit/upvote a feature request for what you call "filtered ingestion" @ https://aka.ms/adx.uservoice.

with regards to steps 1-4 in your question:

  • if you're actually calling 'purge' command in #3, that's both redundant and very costly.
  • you can simply set a retention policy with a soft delete period of 0 (zero) on the source table of the update policy, to guarantee that the raw data is never persisted/committed.