Flume memory channel is full on start-up

Question

I am having problems with the Flume memory channel. I ran a Flume agent, which overwhelmed the memory channel and the logs started spilling out "The channel is full, and cannot write data now. The source will try again after 250 milliseconds"

So far so good. I stop the agent, edit the flume.conf to increase the capacity and try again. The problem is that already at startup Flume spills out the same message:

`16/05/14 00:21:48 INFO node.Application: Starting new configuration:
 { sourceRunners:{s1=EventDrivenSourceRunner: { source:Spool  
  Directory source s1: { spoolDir: /home/vagrant/logs } }} 
  sinkRunners:{kafka-avro-sink2=SinkRunner: { 
  policy:org.apache.flume.sink.DefaultSinkProcessor@63203b59 
  counterGroup:{ name:null counters:{} } }, kafka-avro-
   sink1=SinkRunner: { 
   policy:org.apache.flume.sink.DefaultSinkProcessor@591882e6 
  counterGroup:{ name:null counters:{} } }} channels:
  {mem1=org.apache.flume.channel.MemoryChannel{name: mem1}} }
  16/05/14 00:21:48 INFO node.Application: Starting Channel mem1
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:       
  Monitored counter group for type: CHANNEL, name: mem1: Successfully  
  registered new MBean.
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Component type: CHANNEL, name: mem1 started
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro- 
  sink2
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro-sink1
  16/05/14 00:21:48 INFO node.Application: Starting Source s1
  16/05/14 00:21:48 INFO source.SpoolDirectorySource:  
  SpoolDirectorySource source starting with directory:  
  /home/vagrant/logs
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Monitored counter group for type: SOURCE, name: s1: Successfully  
  registered new MBean.
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
   Component type: SOURCE, name: s1 started
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after 
  250 milliseconds
  16/05/14 00:21:49 INFO avro.ReliableSpoolingFileEventReader: Last  
  read was never committed - resetting mark position.
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after  
  500 milliseconds`

So the channel is already full when I start the agent. How can I manually reset or clear it?

I have searched for hours for a similar problem in vain. A slightly enraging issue is that https://flume.apache.org/ is so full of text that it basically gets indexed by google for pretty much anything flume related. So a search on "flume channel is always full" for example will invariably give https://flume.apache.org/ as a number one result.

Hi have you tried xms and xmx options while starting memory channel through your agent start-up script ? — Ram Ghadiyaram

Ram Ghadiyaram Ram Ghadiyaram · Accepted Answer · 2016-05-16T14:19:05

Seems you are not able to allocate sufficient memory to your channel. where as your incoming rate of event is higher than memory available for memory channel.

try using like below example : xmx is sample option you can gradually increase and see what is best value of memory allocation based on your number of agents running parllely.

For Ex:

flume-ng agent -n $agentnumber -c ../../config/conf/ -f ../../config/conf/youragentconf.conf -Xmx3g

Moreover, you need to also check other configuration parameters in your agent configuration file by referring this Memory channel from Flume Guide.

Flume memory channel is full on start-up

1 Answers