Add instanceid dimension to cloudwatch metric-filter

Question

I set up a very basic metric filter for a log-stream to count the number of lines matching a certain condition. My usecase is very similiar to the one described at the tutorial for counting apache HTTP status codes.

E.g. the log-events look like

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 404 2326

and the filter is very similiar to

[IP, UserInfo, User, Timestamp, RequestInfo, StatusCode=404, Bytes]

I set up the metric-filter via the Cloudwatch console. The filter works perfectly except that it accumulates the values for all instanceids. However the metric is only valuable to me if it it takes the instanceid into account.

Is there a way to get the metric per instanceid, e.g. to write the instanceid as dimension to the metric?

I searched for a while but could not find anything appropriate. It is very important to me, that i don't have to hardcode the instanceid since instances might get terminated at any point and new instances will be created. Further on I want to avoid evaluating the metric on the EC2-machine.

Is there any way to achieve my goal? I would really appreciate any suggestions!

Answer 1

I don't see how metric filters can help you here. First, they don't support dimensions, you can only set a metric name and namespace. Second, you don't have instance id on the logline and metric filters work per logline.

You could use metric filters if you had exactly 1 instance at a time. Then you could configure autoscaling hook to create new metric filters when new instance launches and put the instance id into the metric name, and have another autoscaling hook to delete the metric filter when the instance terminates. If you have more than 1 instance, this approach would double count the values.

You also say you want to avoid publishing custom metrics from the application itself (this would be my prefered approach).

This leaves us with subscription filters.

You can have your log events delivered to a lambda function. These events will have the information about the log group and log stream where the log event originated. You should have instance id as part of the log stream name.

You would then need to write a lambda function that takes the instance id from the log steam, parses the log event and sends this info back into CloudWatch logs using EMF. But you could just do this directly from the application using CloudWatch Agent and EMF, that's why I would prefer that approach.