Apache kafka Error while appending records to topic

Question

I am trying to consume 10 millions rows csv file of size(600MB) via connect api. The connection starts consuption completes for 3.7million records .After that i am getting the below error.

[2018-11-01 07:28:49,889] ERROR Error while appending records to topic-test-0 in dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)
java.io.IOException: No space left on device
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
        at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:95)
        at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:151)
        at kafka.log.LogSegment.append(LogSegment.scala:138)
        at kafka.log.Log.$anonfun$append$2(Log.scala:868)
        at kafka.log.Log.maybeHandleIOException(Log.scala:1837)
        at kafka.log.Log.append(Log.scala:752)
        at kafka.log.Log.appendAsLeader(Log.scala:722)
        at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:634)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
        at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:257)
        at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:622)
        at kafka.server.ReplicaManager.$anonfun$appendToLocalLog$2(ReplicaManager.scala:745)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
        at scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:138)
        at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:236)
        at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:229)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:138)
        at scala.collection.TraversableLike.map(TraversableLike.scala:234)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:733)
        at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:472)
        at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:489)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:106)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
        at java.lang.Thread.run(Thread.java:748)
[2018-11-01 07:28:49,893] INFO [ReplicaManager broker=0] Stopping serving replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
[2018-11-01 07:28:49,897] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions __consumer_offsets-22,__consumer_offsets-30,__consumer_offsets-8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-7,__consumer_offsets-9,__consumer_offsets-46,topic-test-0,__consumer_offsets-25,__consumer_offsets

i have single topic name topic-test

Machine Specs:

OS : CentOs 7
Ram : 16GB
HD : 80GB

i see some blogs talking about log.dirs is server.property but things are not clear as how it wants the input. Also am i to create partition ? I didnot do it thinking it is the same data file.

log.dirs is the directory you want to save your data. tmp is ok for experimenting but will delete everything when you reboot. Also in your case the space left in TMP depends on how you configured your volumes, you can check how much space you have with the command df -h — Paizo
@Paizo yes i did the same df -h and changed my log.dirs to a space which had more space and it works. I am looking for kafka logs to delete on it' own as of now i see for for consumption of 635MB file there is a log of approx 10 GB — souradeep majumdar

souradeep majumdar souradeep majumdar · Accepted Answer · 2018-11-05T06:52:04

ERROR Error while appending records to topic-test-0 in dir /tmp/kafka-logs (kafka.server.LogDirFailureChannel)java.io.IOException: No space left on device This comes when you are consuming a huge file or stream in kafka topic. Go to default log directory /tmp/kafka-logs Then,

[root@ENT-CL-015243 kafka-logs]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhel6u4x64-lv_root   61G   8.4G   49G  15% /
tmpfs                    7.7G     0  7.7G   0% /dev/shm
/dev/sda1                    485M   37M  423M   9% /boot
/dev/mapper/vg_rhel6u4x64-lv_home   2.0G   68M  1.9G   4% /home
/dev/mapper/vg_rhel6u4x64-lv_tmp    4.0G  315M  3.5G   9% /tmp
/dev/mapper/vg_rhel6u4x64-lv_var    7.9G  252M  7.3G   4% /var

As you can see in my case only 3.5Gb of /tmp space was available and I was facing this issue. I create a /klogs in root and changed log.dirs=/klogs/kafka-logs in server.properties

Apache kafka Error while appending records to topic

1 Answers