AWS Kinesis - Kinesis Agent not parsing records

Question

I've created a kinesis stream, installed a kinesis agent on my EC2 instance and configured it to read files from a certain directory.

Kinesis Agent Configuration file:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "kinesis.eu-central-1.amazonaws.com",
  "firehose.endpoint": "firehose.eu-central-1.amazonaws.com",
  "awsAccessKeyId": "ACCESSKEYID",
  "awsSecretAccessKey": "SECRETACCESSKEY",

  "flows": [
    {
      "filePattern": "/home/ec2-user/workspace/connect_s3/documents/*.log",
      "kinesisStream": "EntryPoint",
      "partitionKeyOption": "RANDOM"
    }
]
}

I've also given my EC2 instance the required IAM policies for Kinesis (I added the access keys later on for debug purposes).

Then I started my agent, added files and tailed the /var/log/aws-kinesis-agent/aws-kinesis-agent.log file, but nothing happens.

My agent starts correctly, but no matter how many files I add to the /home/ec2-user/workspace/connect_s3/documents/ directory, the agent can't seem to parse anything; it returns:

Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.

I've been thinking it might be a permission problem, but I changed all permissions and everything is in 755 file permission.

However, in my AWS console, the steam metrics seem to be monitoring that things are happening IMAGE HERE -> (https://i.stack.imgur.com/n2ily.png)

As anyone encountered problems similar to this one ? In advance, thank you for reading.

PickleRick PickleRick · Accepted Answer · 2020-10-30T14:56:21

I found the problem literally seconds after posting this (sorry, it took me hours total to fix the problem).

For anyone having similar issues & if you're try this from an EC2 instance:

Remove your Source directory (filePattern) from /ec2-user -> put it in /var/
Add the "initialPosition": "START_OF_FILE" argument.

Here's my final agent.json file state:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "kinesis.eu-central-1.amazonaws.com",
  "firehose.endpoint": "firehose.eu-central-1.amazonaws.com",
  "awsAccessKeyId": "********",
  "awsSecretAccessKey": "**************",

  "flows": [
    {
      "filePattern": "/var/documents/*.txt",
      "kinesisStream": "EntryPoint",
      "partitionKeyOption": "RANDOM",
      "initialPosition": "START_OF_FILE"
    }
]
}

Also, original answer came from here:

Kinesis agent not parsing the file

AWS Kinesis - Kinesis Agent not parsing records

1 Answers