high iops when insert data to ClickHouse

Question

i am facing a performance issue when insert data to clickhouse. we inserted 100,000 rows per batch, interval time is 30s, but as the cardinality of field of partition key is big, so 100,000 rows will be inserted into 2000+ partitions, then it reached the limitation of iops which is 150, even if we increase limitation of iops of ebs to 2000, still reached the limitation. How can i reduce iops if i cannot reduce number of partitions? the version of clickhouse is 20.5 Thanks in advance!

Answer 1

2000 partitions ~ it's 2000 * number_of_columns * 2(4) files ( * 2 iops) per insert.

try to enable Compact parts Compact — All columns are stored in one file in a filesystem. table SETTINGS min_bytes_for_wide_part = '10M'

or try engine=Buffer