How do I set up AWS kinesishose `agent.json`?

Question

I'm trying to test it as shown below architecture

1. The region is virginia

2. I created an EC2 instance and Access Management (IAM) role configured with permission to write to Amazon Kinesis Firehose and Amazon CloudWatch

   OS amazon Linux : amzn-ami-hvm-2017.09.1.20180115-x86_64-gp2 (ami-97785bed))

3. Install the Fake Apache Log Generator and Runs a script

   pwd -> /home/ec2-user/Fake-Apache-Log-Generator-master/access_log_*.log

   Run script -> python apache-fake-log-gen.py -n 0 -o LOG

4. Install the Amazon Kinesis Agent on EC2 instance

5. create Amazon kinesishose of delivery stream under the name "ymtestdatastream3"

6. agent.json configured on the EC2 instance as below

{

 "cloudwatch.endpoint": "monitoring.us-east-1.amazonaws.com",
  "cloudwatch.emitMetrics": true,
  "firehose.endpoint": "firehose.us-east-1.amazonaws.com",
  "flows": [
      {
      "filePattern": "/home/ec2-user/Fake-Apache-Log-Generator-master/access_log*.log",
      "deliveryStream": "ymtestdatastream3",
      "dataProcessingOptions": [
          {
          "initialPostion": "START_OF_FILE",
          "maxBufferAgeMillis":"2000",
          "optionName": "LOGTOJSON",
          "logFormat": "COMBINEDAPACHELOG"
          }
          ]
      }   
  ]

}

However, there is no data in AWS s3.

/var/log/aws-kinesis-agent/aws-kinesis-agent.log is as follows

localhost (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 3330023ms 2018-02-09 10:56:41.120+0900 localhost (FileTailer[fh:ymtestdatastream3:/home/ec2-user/Fake-Apache-Log-Generator-master/access_log*.log].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:ymtestdatastream3:/home/ec2-user/Fake-Apache-Log-Generator-master/access_log*.log]: Tailer Progress: Tailer has parsed 0 records (0bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.

How do I set up AWS kinesishose agent.json?

help me please

Answer 1

EC2 by default sets /home with highly restricted permissions because that's the default for ssh keys. So the kinesis agent is not even able to get to your home directory. Since you are writing in a restricted path it doesn't work. Edit your faker python program to write logs elsewhere - and ensure you have permissions granted to the entire path till the log file(s). good luck!