Stopping Flume Agent

2
votes

I have a requirement where I want to run Flume agent with spooling directory as source. After all the files from the spool directory is copied to HDFS(sink) I want the agent to stop as I know all the files are pushed to channel. Also I want to run this steps for different spooling directories each time and stop the agent when all files from the directory are marked as .COMPLETED. Is there any way to stop the flume agent?

3
hadoopflumeflume-ng
This is not the use case of flume - Farooque
Ok.May be I can a level down. Let me explain to you what I am trying to achieve.I have a process of ETL. When a user gives input directory I first copy it to HDFS using put command and then run MapReduce job on it.What I was trying is to explore if there is efficient way of pushing data to HDFS than using put command.So I was trying to explore flume to achieve that.But the problem is each time the spooling directory will change as users may want to load data from different directory. Does this fit into the use case of Flume?If not is there any other component available for doing this? - Aditya Calangutkar

3 Answers

0
votes

For now I could suggest that on running flume agent you open the flume agent terminal. Then on this terminal execute ctrl+c and agent is gone.

0
votes

2 ways to stop the Flume agent:

  1. Go to the terminal where Flume agent is running and press ctrl+C to forcefully kill the agent
  2. Run jps from any terminal and look for 'Application' process. Note down its process id and then run kill -9 to terminate the process
0
votes

Open another duplicate session window , then use below command .

ps –ef | grep flume

take out the process_id, and use below command to kill

kill -9 process_id

This worked for me.