Spring kafka batch listener prints whole consumer record

Question

Config

    props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000);
    props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 10);
    props.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 50*1024*1024);
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

prints the whole payload, any help in getting of it and consumer the data would be helpful ConsumerRecord(topic = xxxxxxx, partition = 2, offset = 1512343, CreateTime = 1591460009853, serialized key size = 8, serialized value size = 9506789.......

Log size is too big to print and analyze

Edit 2: could see more this exception in that big log Caused by: org.springframework.kafka.KafkaException: Interrupted while queuing ack

>prints the whole payload - it's not clear what you mean, or what software is "printing" it. If it's your code, you can modify what's printed. Spring only logs the whole record in DEBUG logs. We could add an option to prevent that, if that's what you mean. Exactly "where" is it being "printed"? — Gary Russell
this is an error log, something went wrong, assuming this could be of message size — mkumar
Can you edit the question to show the log (minus the record) so I can see exactly which logger(s) need to be updated? — Gary Russell
Thanks Russell, still i am hitting a rock, why these occurred in first place,love the message not to be printed just metadata is good enough for any audit/bug — mkumar
The fix should be released Wednesday. That error is because you are calling Acknowledgment.acknowledge() on a foreign thread with MANUAL_IMMEDIATE AckMode; and that foreign thread has been interrupted, preventing the queueing of the ack. — Gary Russell

Gary Russell Gary Russell · Accepted Answer · 2020-06-08T17:47:22

I have opened a new feature request.

could see more this exception in that big log Caused by: org.springframework.kafka.KafkaException: Interrupted while queuing ack

That error is because you are calling Acknowledgment.acknowledge() on a foreign thread with MANUAL_IMMEDIATE AckMode; and that foreign thread has been interrupted, preventing the queueing of the ack.

It's best to call it on the listener thread, if possible.

Spring kafka batch listener prints whole consumer record

1 Answers