AWS Cloudwatch Log Insights - Filter Records by JSON filters on JSON log events

Question

I have a log group which accumulates JSON logs to each of its streams. These JSON logs look like this.

I want to filter logs where "user" = "keet". AWS documentation explains on Using Metric Filters to Extract Values from JSON Log Events. I tried this using the AWS SDK, and it worked fine for the following code in NodeJS.

let params = {
    logGroupName: 'log-goupe-name', /* required */
    filterPattern: '{$.user=keet}',

};
cloudwatchlogs.filterLogEvents(params, function(err, data) {
    if (err) console.log(err, err.stack); // an error occurred
    else     console.log(data);           // successful response
});

Question:

Similarly, I want to know whether the same is possible on AWS Cloudwatch Insights Dashboard, on AWS Console? I know string pattern matching is possible. But I wanna know whether JSON field matching is possible on the Insights dashboard using @filter. The default query that comes is as follows.

fields @timestamp, @message
| sort @timestamp desc
| limit 20

I tried following this answer on Stackoverflow, and it still did not help. This is only for parsing data. My requirement is to filter logs based on value in JSON logs.

Thanks in advance.

Dejan Peretin Dejan Peretin · Accepted Answer · 2019-05-26T22:39:06

You can parse out the user from json like this:

parse @message '"user":"*"' as user

Depending on what you want to see on dashboard, you can filter out only particular users with this:

fields @message 
| parse @message '"user":"*"' as user
| filter user == "keet"

Result will be:

#    @message                            user
----------------------------------------------                            
1    info - {"user":"keet","age":30 }    keet
2    info - {"user":"keet","age":30 }    keet  
3    info - {"user":"keet","age":30 }    keet
4    info - {"user":"keet","age":30 }    keet

Also try just:

filter user == 'keet'

maybe Insights will auto-discover the fields. You can see the list of auto-discovered fields on the right, in Discovered fields section.

AWS Cloudwatch Log Insights - Filter Records by JSON filters on JSON log events

1 Answers