How does “latest” tag work in an ECS task definition and container instances pulling from ECR?

Question

I'm having problems using latest tag in an ECR task definition, where image parameter has value like XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/reponame/web:latest.

I'm expecting this task definition to pull an image with latest tag from ECR once a new service instance (task) is run on the container instance (an EC2 instance registered to the cluster).

However in my case when I connect to the container instance remotely and list docker images, I can see that it has not pulled the latest release image from ECR.

The latest tag there is two release versions behind the current one, from since I updated the task definition to use latest tag instance of explicitly defining the version tag i.e. :v1.05.

I have just one container instance on this cluster.

It's possible there is some quirk in my process, but this question is mainly about how this latest should behave in this kind scenario?

My docker image build and tagging, ECR push, ECS task definition update, and ECS service update process:

# Build the image with multiple tags
docker build -t reponame/web:latest -t reponame/web:v1.05 .

# Tag the image with the ECR repo URI
docker tag ${imageId} XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/reponame/web

# Push both tags separately
docker push XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/reponame/web:v1.05
docker push XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/reponame/web:latest

# Run only if the definition file's contents has been updated
aws ecs register-task-definition --cli-input-json file://web-task-definition.json

# Update the service with force-new-deployment
aws ecs update-service \
  --cluster my-cluster-name \
  --service web \
  --task-definition web \
  --force-new-deployment

With a task definition file:

{
  "family": "web",
  "containerDefinitions": [
    {
      "name": "web",
      "image": "XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/reponame/web:latest",
      "essential": true,
      "memory": 768,
      "memoryReservation": 512,
      "cpu": 768,
      "portMappings": [
        {
          "containerPort": 5000,
          "hostPort": 80
        }
      ],
      "entryPoint": [
        "yarn", "start"
      ],
      "environment": [
        {
          "name": "HOST",
          "value": "0.0.0.0"
        },
        {
          "name": "NUXT_HOST",
          "value": "0.0.0.0"
        },
        {
          "name": "NUXT_PORT",
          "value": "5000"
        },
        {
          "name": "NODE_ENV",
          "value": "production"
        },
        {
          "name": "API_URL",
          "value": "/api"
        }
      ]
    }
  ]
}

Can you show your web-task-definition.json file? Could it be that your docker build resulted in the same image (repo sha256) as two versions earlier? The latest tag moves from one image to another but can go back if your code also returns to the same point. — jogold
@jogold I've edited my question to include the task definition file as well. — ux.engineer
@jogold it seems you were correct, the sha256 digests are the same with these versions. This is odd, becase I've used COPY statement in the Dockerfile, and the contents of the folder have changed, at least in one file (package.json) and that file is not in .dockerignore — ux.engineer

ux.engineer ux.engineer · Accepted Answer · 2019-05-10T19:26:47

Turned out the problem was with my scripts. Was using a different variable that had an old value still stored with my terminal session.

I've validated that by using latest tag in the task definition's image source url does have a newly started service instance to pull in the image with latest tag from ECR.

Without needing to register a new revision of the task definition.

As a sidenote, one needs to be careful with handling the latest tag. In this scenario it seems to work out, but in many other cases it would be error prone: Ref1, Ref2

How does “latest” tag work in an ECS task definition and container instances pulling from ECR?

2 Answers