EKS Tenant Specific Cloudwatch log stream in a multi tenant environment

Question

I have multiple microservices hosted on AWS EKS. Each Service is Multi tenant i.e. it caters to multiple tenants. I have integrated Cloudwatch Container Insights using Fluentd. Fluentd is creating one Log stream for each active Service pod. Now, I want to create separate Log stream for each Tenant with only their logs.

So, Ideally what I want is let's say I have a micro service called 'ServiceA' and tenants as TenantA and TenantB. So, in CLoudWatch I should have 2 Log streams for this service something like ServiceA_TenantA and ServiceA_TenantB.

Kindly let me know the approach using fluentd configurations

priyavart priyavart · Accepted Answer · 2021-05-20T18:44:57

How are your tenant being deployed are they in same name space or different. why I am asking this is a I am running fluentbit as a sidecar and have different namesapces for each tenanat. And then my input filter pattern and out filter pattern takes care of finding right log files and sending it to seperate tenant1 cloduwatch logs groups .I am segregating on Log groups.

    [INPUT]
        Name                tail
    Tag                 tenant.*
    Path                /var/log/containers/*_tenant2_webapp-*.log
    Docker_Mode         On
    Docker_Mode_Flush   5
    Docker_Mode_Parser  container_firstline
    Parser              docker
    DB                  /var/fluent-bit/state/flb_container.db
    Mem_Buf_Limit       50MB
    Skip_Long_Lines     On
    Refresh_Interval    10
    Rotate_Wait         30
    storage.type        filesystem
    Read_from_Head      ${READ_FROM_HEAD}


[OUTPUT]
    Name                cloudwatch_logs
    Match               tenant.*
    region              ${AWS_REGION}
    log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application/tenant2
    log_stream_prefix   webapp-
    auto_create_group   true
    extra_user_agent    container-insights

EKS Tenant Specific Cloudwatch log stream in a multi tenant environment

1 Answers