I encountered below error I tried to push a large file to Kafka using Kafka console producer

0
votes

kafka-console-producer --broker-list localhost:9092 --topic com-hftl2 < file.txt

[2019-12-25 16:09:24,476] ERROR Error when sending message to topic com-hftl2 with key: null, value: 20299831 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.RecordTooLargeException: The message is 20299919 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

1
apache-kafka
What is your max.request.size configuration value? – Progman

1 Answers

0
votes

That command will treat each line of a file as its own message.

Kafka defaults to only allowing a max record size of 1MB... If you really plan on sending data much larger than a few MB, the pattern most recommend is to use a shared filesystem such as NFS or S3, then send the filepath in the record and expect consumers to then download the file separately