ECS Fargate - No Space left on Device

1
votes

I had deployed my asp.net core application on AWS Fargate and all was working fine. I am using awslogs driver and logs were correctly sent to the cloudwatch. But after few days of correctly working, I am now seeing only one kind of log as shown below:

enter image description here

So no application logs are showing up due to no space. If I update the ECS service, logging starts working again, suggesting that the disk has been cleaned up.

This link suggests that awslogs driver does not take up space and sends log to cloudwatch instead. https://docs.aws.amazon.com/AmazonECS/latest/userguide/task_cannot_pull_image.html

Did anyone also faced this issue and knows how to resolve the same?

2
aws-fargate

2 Answers

2
votes

It depends on how you have logging configured in your application. The AWSlogs driver is just grabbing all the output sent to the console and saving it to CloudWatch, .NET doesn't necessarily know about this and is going to keep writing logs like it would have otherwise.

Likely .NET is still writing logs to whatever location it otherwise would be.

Advice for how to troubleshoot and resolve:

  • First, run the application locally and check if log files are being saved anywhere
  • Second, optinally run a container test to see if log files are being saved there too
    • Make sure you have docker installed on your machine
    • Download the container image from ECR which fargate is running.
      • docker pull {Image URI from ECR}
    • Run this locally
      • Do some task you know will genereate some logs
    • Use docker exec -it to connect up to your container
    • Check if log files are being written to the location you identified when you did the first test

Finally, once you have identified that logs are being written to files somewhere pick one of these options

  1. Add some flag which can be optionally specified to disable logging to a file. Use this when running your application inside of the container.
  2. Implement some logic to clean up log files periodically or once they reach a certain size. (Keep in mind ECS containers have up to 20GB local storage)
  3. Disable all file logging(not a good option in my opinion)

Best of luck!

2
votes

You need to set the "LibraryLogFileName" parameter in your AWS Logging configuration to null.

So in the appsettings.json file of a .Net Core application, it would look like this:

"AWS.Logging": {
  "Region": "eu-west-1",
  "LogGroup": "my-log-group",
  "LogLevel": {
    "Default": "Information",
    "Microsoft": "Warning",
    "Microsoft.Hosting.Lifetime": "Information"
  },
  "LibraryLogFileName": null
}