Kafka compression, how to limit it to some listed topics? How to use compressed.topics property using clients produce API?

0
votes

I've enabled kafka message compression from client's producer API like :

configProperties.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip")

It works well, however if we want to limit the above compression to some listed topics, we've notice a way :

configProperties.put("compressed.topics", "topicA,topicB")

from this link : https://cwiki.apache.org/confluence/display/KAFKA/Compression . But it is not working, means in my test case I've enabled compression and I didn't listed my topic in compressed.topics's args. still the compression been applied. please help.

1
apache-kafkacompression
You need to set compression.code according to that wiki page, but I would first check the official configurations on the Kafka website since that wiki page seems to be dated back to Kafka 0.7 release - OneCricketeer

1 Answers

2
votes

Compression is handled by several parameters in Kafka:

  • On the brokers, compression.type configures the default for the entire cluster. It allows the values gzip, snappy, lz4, uncompressed or producer. The first three are self-explanatory, they're just different compression methods (snappy/lz4 being the recommended ones), uncompressed is self-explanatory, and the default producer, which basically is just using whatever compression (or not) the producer decided to use. In this case, Kafka just sees the messages as byte arrays, and doesn't try to decode them.
  • At the topic level on the broker, you can also specify the same configurations as above, gzip, snappy, lz4, uncompresssed or producer. They're exactly the same, but act as overrides specifically for the topic you set them to.
  • On the producer side, in code, you can also set compression.type, and the possible values are gzip, snappy, lz4 and none, where this last one is the default.

You should be looking at the recent documentation for Kakfa. You don't specify which Kafka version you're using, but assuming a recent one, I would look at these docs:

https://kafka.apache.org/20/documentation.html

In those, there is no mention of compression.topics as a configuration option. So, like @cricket_007 mentioned, this is an outdated/stale wiki page that you shouldn't be using.