AWS Cloudwatch Log filtered params available on Alarm notification

Question

I'm using Cloudwatch and Lambda to monitor logs. I have setup some filters that are triggered when a specific kind of error appears in the log. Is it possible to send on the Cloudwatch Alarm SNS the params that are filtered on the Log?

For example:

Receive Error :

[2017-06-29 17:58:52] prod.ERROR: ErrorException: Undefined variable: message X

Metric Filter :

[date,info="*.ERROR:",error]

Alarm :

IS OK when <= 0

Notification sent to SNS and Lambda to trigger different notification agents.

The SNS Event gives me this message , but I would like to access the filtered vars:

{
    "AlarmName": "PHP_ERROR",
    "AlarmDescription": null,
    "AWSAccountId": "xxxxxxxxx",
    "NewStateValue": "OK",
    "NewStateReason": "Threshold Crossed: no datapoints were received for 1 period and 1 missing datapoint was treated as [NonBreaching]",
    "StateChangeTime": "2017-06-29T17:09:12.336+0000",
    "Region": "EU - Ireland",
    "OldStateValue": "ALARM",
    "Trigger": {
        "MetricName": "PHP_ERROR",
        "Namespace": "Logs",
        "StatisticType": "Statistic",
        "Statistic": "SUM",
        "Unit": null,
        "Dimensions": [],
        "Period": 60,
        "EvaluationPeriods": 1,
        "ComparisonOperator": "GreaterThanOrEqualToThreshold",
        "Threshold": 0,
        "TreatMissingData": "- TreatMissingData:                    NonBreaching",
        "EvaluateLowSampleCountPercentile": ""
    }
}

Thanks,

Efren Efren · Accepted Answer · 2018-12-18T01:25:37

Unfortunately the alarm is only looking at a metric to evaluate the threshold. So the short answer is no :(

Then, you have another level of abstraction, since the metric already is set from a particular value of the filter.

The filter is the only place where you can find the extracted values, but it will only translate a match into a value or increment for a metric (ie: number), it will not work as a parsed log storage.

AWS recently released Cloudwatch Log Insights which may help you find the error messages.

Otherwise you can try the AWS CLI filter for logs command, which allows you to browse logs.

Example in Ubuntu

export YOUR_LOG_GROUP_NAME=SomeLogGroup

# The 1000 multiplication is to convert from seconds to milliseconds
# If you already have a specific timestamp, just replace it on the start-time argument
aws logs filter-log-events --log-group-name $YOUR_LOG_GROUP_NAME \
--start-time $(($(date +%s --date="1 minute ago") * 1000)) \
--interleaved --filter-pattern ".ERROR" \
--output=text --query events[*].[message]

Example in OSx

export YOUR_LOG_GROUP_NAME=SomeLogGroup

# The 1000 multiplication is to convert from seconds to milliseconds
# If you already have a specific timestamp, just replace it on the start-time argument
aws logs filter-log-events --log-group-name $YOUR_LOG_GROUP_NAME \
--start-time $(($(date -v-1M +%s) * 1000)) \
--interleaved --filter-pattern ".ERROR" \
--output=text --query events[*].[message]

If you want to automate it, you can replace the trigger action, to call a lambda (using some AWS SDK instead of the CLI) that would be able to generate the desired message for SNS with this information.

ie:

From: Metric -> Alarm -> SNS
To: Metric -> Alarm -> Lambda -> SNS

AWS Cloudwatch Log filtered params available on Alarm notification

1 Answers

Example in Ubuntu

Example in OSx