My pipeline is:
cloudwatchlogs > lambda > Elastic Search
.
The problems is that those logs are way too verbose (and there is nothing i can do about it) and like 70% of it needs to be filtered, thus they won't fill uselessly my elastic cluster.
I thought i could apply metrix filtering logs on cloudwatch log group, but that's does not filter anything in the meaning that it remove them from cloudwatch, just graph some stats about it..., thus those undesired logs still appears.
All i found was this little place when you create the subscription filter: but its very primitive and i would need at least 30-40 differents filter pattern, not just one
So my question is:
Is my only way to not be bothered by unwanted logs is to filter them manually (regex etc..) inside my lambda function? There must be a much simpler way, isn't it?
Thanks