Logs to AWS Cloudwatch from Docker Containers

Question

I have a few docker containers running with docker-compose on an AWS EC2 instance. I am looking to get the logs sent to AWS CloudWatch. I was also having issues getting the logs from docker containers to AWS CloudWatch from my Mac running Sierra so I've moved over to EC2 instances running Amazon AMI.

My docker-compose file:

version: '2'
services:
  scraper:
  build: ./Scraper/
  logging:
    driver: "awslogs"
    options:
       awslogs-region: "eu-west-1"
       awslogs-group: "permission-logs"
       awslogs-stream: "stream"
  volumes:
    - ./Scraper/spiders:/spiders

When I run docker-compose up I get the following error:

scraper_1 | WARNING: no logs are available with the 'awslogs' log driver

but the container is running. No logs appear on the AWS CloudWatch stream. I have assigned an IAM role to the EC2 container that the docker-containers run on.

I am at a complete loss now as to what I should be doing and would apprecaite any advice.

Answer 1

The awslogs works without using ECS.

you need to configure the AWS credentials (the user should have IAM roles appropriate [cloudwatch logs]).

I used this tutorial, it worked for me: https://wdullaer.com/blog/2016/02/28/pass-credentials-to-the-awslogs-docker-logging-driver-on-ubuntu/

Answer 2

I was getting the same error but when I checked the cloudwatch logs, I was able to see the logs in cloudwatch. Did you check that if you have the logs group created in cloudwatch. Docker doesn't support console logging when we use the custom logging drivers.

The section on limitations here says that docker logs command is only available for json-file and journald drivers, and that's true for built-in drivers.

When trying to get logs from a driver that doesn't support reading, nothing hangs for me, docker logs prints this:

Error response from daemon: configured logging driver does not support reading

Answer 3

There are 3 main steps involved it to it.

1. Create an IAM role/User
2. Install CloudAgent
3. Modify docker-compose file or docker run command

I have referred an article here with steps to send the docker logs to aws cloudwatch.

Answer 4

The AWS logs driver you are using awslogs is for use with EC2 Container Service (ECS). It will not work on plain EC2. See documentation.

I would recommend creating a single node ECS cluster. Be sure the EC2 instance(s) in that cluster have a role, and the role provides permissions to write to Cloudwatch logs.

From there anything in your container that logs to stdout will be captured by the awslogs driver and streamed to Cloudwatch logs.