New Kafka consumer ignoring earliest offset

0
votes

I have a kafka topic(XYZ) with just one partition and one consumer(C1) running on RHEL6 machine. I have copied the same setup on to a RHEL7 machine and stopped the C1 and started the new consumer(C2) using the same group id as the C1. C2 is able to connect, exchange heartbeat messages and print following debug statements repeatedly but not able to consume messages. Received zero records. It uses "earliest" offset reset but seems to ignore it. note: all consumer share the same code version.

Logs:

Fetcher: [Consumer clientId=testclient-0, groupId=mcu] Fetch READ_UNCOMMITTED request at offset 1001 for partition XYZ-0 returned fetch data (error=none, highWaterMark=1001, lastStableOffset=1001, logStartOffset=10 ....

Adding READ_UNCOMMITTED request for partiion... Sending READ_UNCOMMITTED request for partition... Resetting offset for partition XYZ-0 to the committed offset 1001

My question is why it is resetting offset to the latest offset value and not consume message from beginning.

Following is what i have tried to resolve the issue.

  1. I am able to consume message from C2 consumer from another topic on same cluster.
  2. I am able to consume message from same topic XYZ from another consumer C3 on a different RHEL 6 machine with exactly same configuration.
  3. I have used a new Group id but no success
  4. Produced message after staring the consumer first but no success. Note: Kafka client vaersion 1.0

Any help would be highly appreciated. Thank you.

1
apache-kafkakafka-consumer-api
Can you describe the old and new consumer groups? Can you show what the GetOffsetShell tool reports as the earliest offset? – OneCricketeer
Set -Dorg.xerial.snappy.tempdir=/some/other/path/with/execpermissions/ in jvm args and it worked. Broker unpack the library to a suitable filesystem with exec permissions. stackoverflow.com/questions/64987526/… – Rajat Swain
Other option is give full permissions to /tmp/ folder on the consumer host. – Rajat Swain
Please post your answer below rather than as a comment – OneCricketeer

1 Answers

0
votes

I was able to fix it after setting the following jvm argument.

-Dorg.xerial.snappy.tempdir=/some/other/path/with/execpermissions/

The new host where I set up the consumer didn't have write permissions to /tmp dir(default) which is needed to unpack snappy library. Also make sure /tmp has space left for these libraries. The RHEL6 hosts with old consumers had write permissions. The compression type was not set on topic configuration so it used snappy by default. Other topics where I could consume from had compression type set to producer, so it didn't need snappy libraries and hence worked. The issue had nothing to do with RHEL 6 or RHEL7 version.

More info: Kafka Broker throws error for clients that produce snappy compressed messages