AWS-CloudWatch: InvalidSequenceTokenException

Question

I have a php worker where i log events to AWS could watch. Unfortunately i got the following error when i try to submit it.

InvalidSequenceTokenException Error executing "PutLogEvents" on "https://logs.eu-west-1.amazonaws.com"; AWS HTTP error: Client error: POST https://logs.eu-west-1.amazonaws.com resulted in a 400 Bad Request response: {"__type":"InvalidSequenceTokenException","expectedSequenceToken":"999999999999990356407851919528174 (truncated...) InvalidSequenceTokenException (client): The given sequenceToken is invalid. The next expected sequenceToken is: 495599999999988500356407851919528174642 - {"__type":"InvalidSequenceTokenException","expectedSequenceToken":"495573099999999900356407851919528174642","message":"The given sequenceToken is invalid. The next expected sequenceToken is: 495579999999900356407851919528174642"}

and this is my code

 $date = new DateTime();
 $instance= = new CloudWatchLogsClient([
                'region' => 'eu-west-1',
                'version' => 'latest',
                'credentials' => [
                    'key' => 'XXX',
                    'secret' => 'XXXX'
                ]
            ]);
        $instance->putLogEvents([
                'logGroupName' => "WorkerLog",
                'logStreamName' => "log",
                'logEvents' => [
                    [
                        'timestamp' => $date->getTimestamp(),
                        'message' => "test log"
                    ]
                ]
            ]);

Answer 1

http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html

You must include a sequence token with your request. If you don't have one you must use describeLogStreams (http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogStreams.html) to get the stream sequence.

When you make a call to putLogEvents you will get the nextToken in the response. You also must be ready for the case in which someone else pushes to the stream and invalidates the nextToken. (in this case you need to describe the stream again to get an updated token).

Answer 2

The DescribeLogStreams does not support the same volume call as PutLogEvent. You may got throttled if calling it frequently.

The recommended way to do it is calling the PutLogEvents directly and catch the InvalidSequenceTokenException. Then retry the PutLogEvents with Sequence Token in the InvalidSequenceTokenException message.

Correct sequence token can be found in the expectedSequenceToken field of InvalidSequenceTokenException

    try {
        $result = $client->describeLogStreams([
            'logGroupName' => $logGroupName,
            'logStreamNamePrefix' => $logStreamName,
        ]);

        $uploadSequenceToken = $logStreams[0]['uploadSequenceToken'];

        $client->putLogEvents([
            'logGroupName' => $logGroupName,
            'logStreamName' => $logStreamName,
            'logEvents' => [
                [
                    'timestamp' => $timestamp,
                    'message' => $message
                ],
            ],
            'sequenceToken' => $uploadSequenceToken,
        ]);
    } catch (\InvalidSequenceTokenException $e) {
        $client->putLogEvents([
            'logGroupName' => $logGroupName,
            'logStreamName' => $logStreamName,
            'logEvents' => [
                [
                    'timestamp' => $timestamp,
                    'message' => $message
                ],
            ],
            'sequenceToken' => $e->expectedSequenceToken,
        ]);
    }

Answer 3

This is my working solution: before send new putLogEvents you must take the last uploadSequenceToken.

    try {
        $client = \Aws\CloudWatchLogs\CloudWatchLogsClient::factory($configCloudWatch);

        $logStreamName = 'testLogStream';
        $logGroupName = 'testGroupName';

        $result = $client->describeLogStreams([
            'logGroupName' => $logGroupName,
            'logStreamNamePrefix' => $logStreamName,
        ]);
        $logStreams=$result->get('logStreams');
        if (!$logStreams)
            throw new \Exception('No log stream found');
        if (count($logStreams)!=1)
            throw new \Exception('Multiple log stream found');
        $uploadSequenceToken = $logStreams[0]['uploadSequenceToken'];

        $client->putLogEvents([
            'logGroupName' => $logGroupName,
            'logStreamName' => $logStreamName,
            'logEvents' => [
                [
                    'timestamp' => round(microtime(true) * 1000),
                    // message is required
                    'message' => json_encode([ ... ]
                    ),
                ],
            ],
            'sequenceToken' => $uploadSequenceToken,
        ]);
    } catch (\Exception $e) {
        \Log::error(__METHOD__, ['exception' => $e]);
    }