How to disable headers from Consuming in kafka

1
votes

Is there any option to disable Kafka headers being consumed from consumer. In my case I wrote a consumer to consume messages from a Kafka topic published by an upstream system. My processing doesn't require any information from headers and the published headers are heavy weight (bigger than the message itself in size). So my consumer is taking longer time than expected.

Any option that I can only consume message content leaving headers so that it saves time to transfer the headers over network and de-serialize them at consumer. Your help is appreciated.

1
apache-kafkakafka-consumer-api
Kafka has very minimal headers. It's just a message, which are simply byte arrays, and an optional key. And then maybe an offset and a timestamp. If you have large headers that your upstream application is sending, then it's up to that application to stop sending them. – mjuarez
@mjuarez correct. the headers were added by the upstream application and i don't have any control to stop them or ask them to stop publishing headers. Because the published events are being consumed by multiple dowstreams like mine. – Reddy

1 Answers

1
votes

Every message is a Record with Headers (as of Kafka 0.11).

length: varint
attributes: int8
    bit 0~7: unused
timestampDelta: varint
offsetDelta: varint
keyLength: varint
key: byte[]
valueLen: varint
value: byte[]
Headers => [Header]

Record Header

headerKeyLength: varint
headerKey: String
headerValueLength: varint
Value: byte[]

Even if you ignore deserializing them, they will still be sent over the wire as part of the Record's TCP packet body.

You could try using a Kafka 0.10.2 client version, for example, which might drop the header entirely, because they just weren't part of the API, but YMMV.

As mentioned in the comments, the most reliable way here would be to stop sending such heavy information in the upstream application. Or the middle-ground would be to compress, and/or binary encode that data.