Piping Apache server log using Apache Flume

1
votes

I'm using Apache Flume version 1.3 and referring to Apache Flume User guide.

My objective is to pipe Apache server logsdirectly into the Apache Flume Agent's Channel. If my understanding is correct, Apache server won't be creating local files viz. server.log, error.log but directly writing the corresponding output to the agent's channel; even if it writes in logs, the Agent won't be reading from those files. But at least the Apache Server's documentation doesn't provide an example for the same.

My queries are :

  1. I referred this link which pertains to Cloudera Flume AND NOT Apache Flume. I tried this but as expected, the Apache server gave an error :

    /bin/sh: flume: not found
piped log program 'flume node_nowatch -1 -s -n apache -c \\'apache:console|agentBESink("collector");\\'' failed unexpectedly
piped log program 'flume node_nowatch -1 -s -n apache -c \\'apache:console|agentDFOSink("collector");\\'' failed unexpectedly

I'm doubtful of Apache Flume and Cloudera Flume being in-sync.

  1. As per the Apache Server's documentation : Apache httpd is capable of writing error and access log files through a pipe to another process, rather than directly to a file. How do I achieve this for a running Apache Flume Agent i.e what must be the entry in the Apache conf

CustomLog "| ?" common

1
apacheapache2flume

1 Answers

1
votes

I was trying a similar usecase. Your issue is that the flume command does not exist (anymore). The newer versions of flume use the command flume-ng.

What I did in the /etc/httpd/conf/httpd.conf:

# Default behaviour, but daily-rollover logging (|| does not spawn a new shell) 
CustomLog "||/usr/sbin/rotatelogs /var/log/httpd/access_log.%Y-%m-%d 86400" combined
# Send logging to local flume-agent
CustomLog "||/usr/bin/flume-ng avro-client -H localhost -p 10000" combined

The flume-agent config has an avro source bind on localhost and listens on port 10000. Also an file-channel and in our case an HDFS-sink.

Note that when the Flume-agent crashes or you restart Flume you will lose some logging events. If you want a more durable solution you need to pickup (old/rotated) log-files and move/remove when processed.