3
votes

I am trying to migrate a set of microservices from Docker Swarm, to AWS ECS using Fargate.

I have created an ECS cluster. Moreover, I have initialized repositories using the ECR, each of which contains an image of a microservice.

I have successfully came up with a way to create new images, and push them into the ECR. In fact, with each change in the code, a new docker image is built, tagged, and pushed.

Moreover, I have created a task definition that is linked to a service. This task definition contains one container, and all the necessary information. Moreover, its service defines that the task will run in a VPC, and is linked to a load balancer, and has a target group. I am assuming that every new deployment uses the image with the "latest" tag.

So far with what I have explained, everything is clear and is working well.

Below is the part that is confusing me. After every new build, I would like to update the service in order for new tasks with the update image get deployed. I am using the cli to do so with the following command:

aws ecs update-service --cluster <cluster-name> --service <service-name>

Typically, after performing the command, I am monitoring the deployment logs, under the event tab, and checking the state of the service using the following command:

aws ecs describe-services --cluster <cluster-name> --service <service-name>

Finally, I tried to simulate a case where the newly created image contains a bad code. Thus, the new tasks will not be able to get deployed. What I have witnessed is that Fargate will keep trying (without stopping) to deploy the new tasks. Moreover, aside the event logs, the describe-services command does not contain relevant information, other than what Fargate is doing (e.g., registering/deregistering tasks). I am surprised that I could not find any mechanism that instructs Fargate, or the service to stop the deployment and rollback to the already existing one.

I found this article (https://aws.amazon.com/blogs/compute/automating-rollback-of-failed-amazon-ecs-deployments/ ), which provides a solution. However, it is a fairly complicated one, and assumes that each new deployment is triggered by a new task definition, which is not what I want.

Therefore, considering what I have described above, I hope you can answer the following questions:

1) Using CLI commands (For automation purposes) Is there a way to instruct Fargate to automatically stop the current deployment, after failing to deploy new tasks after a few tries?

2) Using the CLI commands, is there a way to monitor the current status of the deployment? For instance, when performing a service update on a service on Docker swarm, the terminal generates live logs on the update process

3) After a failed deployment, is there a way for Fargate to signal an error code, or flag, or message?

1
1) Don't know. 2) The way I monitor Fargate is the CloudWatch logs 3) At a failed deployment, AWS HealthCheck would fail wouldn´t it? And that would show in the CloudWatch logs.Niklas R.

1 Answers

1
votes

At the moment, ECS does not offer deployment status directly. Once you issue a deployment, there is no way to determine its status other than to continually poll for updates until you have enough information to infer from them. Plus unexpected container exits are not logged anywhere. You have to search through failed tasks. The way I get them is by cloudwatch rule that triggers a lambda upon task state change.

I recommend you read: https://medium.com/@aaron.kaz.music/monitoring-the-health-of-ecs-service-deployments-baeea41ae737