How to query distinct from AWS log insights

Question

I need to query data from lambda using AWS Cloudwatch log insights. The query syntax provide by aws doesn't have distinct.

Only support (count_distinct(fieldname))

ref. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html

Example data

Column # @timestamp @ message

1 2020-02-17T13:33:29.049+07:00 [INFO] 2020 Partition key: ABC12345_A_

2 2020-02-17T11:32:29.049+07:00 [INFO] 2020 Partition key: ABC12345_B_

3 2020-02-17T11:31:29.049+07:00 [INFO] 2020 Partition key: ABC12345_B_

4 2020-02-17T11:30:29.049+07:00 [INFO] 2020 Partition key: ABC12345_C_

5 2020-02-17T11:29:29.049+07:00 [INFO] 2020 Partition key: ABC12345_A_

Expected result

1 2020-02-17T13:33:29.049+07:00 [INFO] 2020 Partition key: ABC12345_A_

2 2020-02-17T11:32:29.049+07:00 [INFO] 2020 Partition key: ABC12345_B_

4 2020-02-17T11:30:29.049+07:00 [INFO] 2020 Partition key: ABC12345_C_

If usage normal SQL syntax look like below.

select distinct(uuid) as uuid, max(time) as time from table_name group by uuid order by time desc

I was able to get the distinct rows using count_distinct. You could try something similar to this: stats count_distinct(@logStream) as IngestionTime by @ingestionTime, @logStream as LogStream | sort @ingestionTime desc | limit 10 — Rupesh

sunvenka sunvenka · Accepted Answer · 2020-10-03T04:06:04

8

votes

can use the | stats count(*) by fieldname

this works to list the distinct fieldnames

How to query distinct from AWS log insights

2 Answers