Elastic Beanstalk output custom logs to

Question

I have an Elastic Beanstalk app running a .Net app through IIS in a Windows environment. I want to get my custom logs to show up in AWS CloudWatch.

The app uses a Serilog logger in one of its static classes.
The logger outputs a message when I go to an endpoint path (ex. "/api/log-a-message").
Logs are written to a file "C:\LogsFolder\LogFile.log".

Following some online searching and reading through other questions and the AWS Documents. I ended up writing a .ebextensions/log_configuration.conf with the following content:

### BEGIN .ebextensions/CloudWatch.config
files:
  "C:/Program Files/Amazon/ElasticBeanstalk/config/taillogs.d/":
    content: |
      [ZeW logs]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "Serilog"]]}`
      log_stream_name = {instance_id}
      file = C:/LogsFolder/LogFile.log
### END .ebextensions/CloudWatch.config

But the logs are still not appearing in CloudWatch.

Did you setup permissions in IAM for Elastic Beanstalk to talk to Cloudwatch? I have the CloudWatchAgentServerPolicy and AmazonEC2RoleforSSM policies added to the aws-elasticbeanstalk-ec2-role, though it looks like the latter is deprecated in favor of AmazonSSMManagedInstanceCore. — littleforest
My EC2 role has the following policy, which should allow for the needed privileges ... "logs:CreateLogGroup", "logs:CreateLogStream", "logs:GetLogEvents", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:PutRetentionPolicy" — ZeW
I also set up two commands to run using the commands directive in that same file: chkconfig awslogs on, and then service awslogs restart. — littleforest
Those are linux commands. The Windows version (I think) needs a conf file when restarting, but I don't know where my .ebextensions/CloudWatch.config file gets flung to, otherwise I could just use the appendConf option — ZeW

ZeW ZeW · Accepted Answer · 2019-10-14T15:08:41

I've managed to almost do it. ... So it turns out that for AWS CloudWatch you need to append a configuration in JSON format.

Below is my .ebextensions/custom_logs.config file that Elastic Beanstalk uses for extensions. This just creates a custom_logs.json file for CloudWatch to use.

files:
  "C:/Users/Administrator/Desktop/custom_logs.json":
    content: |
      {
        "agent": {
          "metrics_collection_interval": 5
        },
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [{
                "file_path": "C:\\MyCustomLogsFolder\\MyCustomLogFile.log",
                "log_group_name": "/aws/elasticbeanstalk/UsuallyThisIsTheEnvironmentName/MyCustomLogGroup-Log",
                "timezone": "UTC",
                "timestamp_format": "%Y-%m-%d %H:%M:%S",
                "multi_line_start_pattern": "{timestamp_format}"
              }]
            }
          }
        }
      }

After that file is created on my Desktop by Elastic Beanstalk, I can connect to the instance and run the following command (including the & at the start):

& C:\\'Program Files'\\Amazon\\AmazonCloudWatchAgent\\amazon-cloudwatch-agent-ctl.ps1 -a append-config -m ec2 -c file:C:\\Users\\Administrator\\Desktop\\cu
stom_logs.json -s

The only thing I need to figure out now is how to do this automatically once the instance starts.

Managed to do it with the following code (in addition to the files: statement above):

services:
  windows:
    AmazonCloudWatchAgent:
      enabled: 'true'
      ensureRunning: 'true'
      files:
        - "C:/MyCustomLogsFolder/MyCustomLogFile.log"
container_commands:
  01_cloudwatch_append:
    command: powershell.exe -ExecutionPolicy Bypass -Command "$cwa_ctl='C:\\Program Files\\Amazon\\AmazonCloudWatchAgent\\amazon-cloudwatch-agent-ctl.ps1'; $custom_logs_config='C:\\Users\\Administrator\\Desktop\\custom_logs.json'; & $cwa_ctl -a append-config -m ec2 -c file:$custom_logs_config -s;"
    ignoreErrors: true
    waitAfterCompletion: 10

Elastic Beanstalk output custom logs to

1 Answers