Table Encoding Policy difference between Kusto/ADX clusters

0
votes

I have the exact the same table in 2 different ADX/Kusto clusters -- the data/schema is identical but if I calculate ExtentSize for 1 day of data the difference between the two is enormous. While the table on cluster has 10TB , the table on the other has 15TB. That's a big difference. When I checked encoding policies on both the table , there is a slight difference. The table on the first cluster has the following encoding policy:-

"ColumnIndexRangeGranularity": 0,
"ShardFieldCompressionCodec": "DEFAULT",

Whereas the table on the other cluster has the following:-

"ColumnIndexRangeGranularity": 32,
"ShardFieldCompressionCodec": "LZ4",

My goal is bring down size of other table to the same size as the first table, so I can do with less cache policy. So ideally I would like to change these two parameters. But when I fire the following command, it has no effect on encoding policy of the table:-

.alter table MyTable policy encoding   @'{ "ShardFieldCompressionCodec": "Default" }'

There is no error also.

So I have two questions.

  1. Does ADX simply ignore whenever we try to change encoding policy of a table? These encoding policies we have not explicitly set , when the table got created these were assigned by clusters by default.
  2. Does it mean that if we have the same table in 2 different clsuters with the same schema/table -- their extent size will always vary (because the underlying table compression is different)?
1
azure-data-explorer

1 Answers

1
votes

An encoding policy can be set on different entities:

  • column - only affects data ingested to the column after the change in policy.
  • table - only affects columns that will be added to the table after the change in policy.
  • database – only affects tables that will be created in the database after the change in policy.

It appears what you're interested in doing is altering the column level policy, and not the table level policy. Though, you should understand that will not change the encoding for data that has already been ingested.