Flume NullPointerExceptions on checkpoint

2
votes

I've setup a file to file source/sink , just as a test of basic flume functionality.

Im currently using the "exec" source, with the command being "tail -F mytmpfile".

In my script, I continuously echo "....." >> mytmpfile , so that the tail command constitutes a stream.

However, I've started seeing the following exception in the flume logs:

java.lang. IllegalStateException: Channel closed [channel=c1]. Due to java.lang.NullPointerException: null at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:353) at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122) at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:183) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:895) at org.apache.flume.channel.file.Log.replay(Log.java:406) at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:303) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) ... 1 more

Any thoughts on where this NullPointerException is coming from? It appears from scanning the code that maybe it related to a missing folder or directory. But I cant find the exact line on the git hub branches.

This is using apache-flume-1.3.1.23-...

1
streamnullpointerexceptionflume
It seems that the error comes from "file" channel (flume.apache.org/FlumeUserGuide.html#file-channel), not "exec" source. Its code is here: git-wip-us.apache.org/repos/…. Why don't you use standard "memory" channel (flume.apache.org/FlumeUserGuide.html#memory-channel)? Could you show your config? - Dmitry
The source above helped me realize that the error was in my configuration file, which want referencing the channels correctly - jayunit100
I've ironed out a lot of these kind of problems through remote debugging- try adding -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=xxxx,suspend=n to the JVM flags in flume-env-sh. - Chris Mowforth

1 Answers

2
votes

In the past I've had problems with file channels, and they've normally boiled down to two problems:

1) If you're running multiple agents on the same box, make sure you configure them to have separate dataDirs and checkpointDir.

2) On Linux boxes, check that your tmpfs isn't near its capacity. If it's getting full, flume will complain. Try stopping the flume agent, unmount tmpfs, enlarge it, remount and restart the agent.