docker-compose.yml file and always pulling the last built image (without having a :latest tag)

Question

Some background:

I have a private docker registry running and working property. I have a azure pipeline which builds and pushes the images I build from my repo via Dockerfiles. The images are correctly tagged(according to the pipeline I have built) and it pushes the images to the docker registry.

This is good and I'm happy with this.

However, Whenever I want to provide my colleagues with a 'customer' version of a docker-compose.yml which pulls the images and setups the environment to run locally on their computer I have to specifically point-out which tag for each image to pull through the docker-compose.yml file. Thus always helping them out to update it whenever I have a new version of each image in the registry. This is inconvenient, but doable, but I would like to have a simpler solution(if possible)

My questions is: Is it possible to write a generic docker-compose file which for example always pulls the last pushed image from the registry(not to be confused with a :latest tag which I currently don't use)

Example(what I currently do with my colleagues docker-compose.yml file):

version: '3.3'

services:
  testimage:
    hostname: testimage
    image: my.private.registry.com/testimage:566
    volumes:
    ports:
      - '3000:3000'    

This example works, since I have an image in my registry that have this tag, but would it possible to express somehow inside the docker-compose.yml to always pull the last built image, which will not always be :566?

If that is the case, then how would it look like?

Br Magnus

Answer 1

I haven't used azure pipelines (yet) but I would add a task/job/step - whatever it is called - at the end of the pipeline to generate a docker-compose.yaml file with the proper tag with some rudimentary text manipulator command (like sed or envsubst on linux) and save this compose file as an artifact so you can download from the pipeline artifact repository or even better make the task to publish it somewhere.

Answer 2

This particular task is a little tricky to do. The Docker registry API doesn't have any notion of "most recent" and doesn't give any particular meaning to tags. Furthermore, tools like docker run and Docker Compose connect to the Docker engine API, and the only registry-related operations it has are pushing and pulling specific images.

You can at least avoid updating the docker-compose.yml file on every build by using variable substitution to fill in the tag:

image: my.private.registry.com/testimage:${TESTIMAGE_TAG:-latest}

TESTIMAGE_TAG=566 docker-compose up -d

If you had this basic approach, you could split up the launch sequence into three files. One is the docker-compose.yml file, using variables for all of the image tags. A second is a shell script that launches everything. The third file just has a listing of the image tags, and your CI system can generate that file when it builds a release.

# manifest.sh
export TESTIMAGE_TAG=566

#!/bin/sh
# launch.sh
. ./manifest.sh
if [ "$#" = 0 ]; then
  exec docker-compose up -d
else
  exec docker-compose "$@"
fi

In principle you could combine this script with one of the approaches in Listing the tags of a Docker image on a Docker hub through the HTTP API to try to find the most recent, but this is basically all work outside of Docker.

(This doesn't really get better as you move to higher-level clustering systems. In Kubernetes, for example, you typically need a templating system like Kustomize or Helm that can fill in these tag values at deployment time. Helm has the notion of a YAML "values file" so your CD system could produce a fixed values file and then invoke Helm, but the high-level approach is pretty similar to what I've described here.)

Answer 3

I managed to get it work as I intended. My images which were tagged with :655 etc now also gets tagged with a :latest tag.

This is my updated azure-pipeline.yml for one of my repos, which is used to build, tag and push my compiled docker image.

The only update I made to this pipeline is the last 'Task' as mentioned as a suggestion in the thread which was missing in my original pipeline. It does the exact same thing as the 'Task' above it, but tags it with a hardcoded name:tag instead of using the dynamically :tag value from $(Build.BuildId).

This results in 1 image being build, but with 2 tags. This has the effect that I will always have a :latest image available on the compiled image, as well as still having access to all the other images through the build number tag.
Yay! =)

Thanks for all the help and suggestions! Much appreciated!

name : $(date:yyyyMMdd)_$(Build.BuildId)
trigger:
  branches:
    include:
      - master
 
pool:
  vmImage: 'ubuntu-latest'
 
variables:
  ImageName: '<image-name>:$(Build.BuildId)'
  
stages:
- stage: Build
  displayName: Build image
  jobs:  
  - job: Build
    displayName: Build and push Docker image
    steps:
    - task: Docker@1
      displayName: 'Build the Docker image'
      inputs:
        containerregistrytype: 'Container Registry'
        dockerRegistryEndpoint: 'MyRegistered_DockerRegistry'
        command: 'Build an image'
        dockerFile: '**/Dockerfile'
        ImageName: '$(ImageName)'
        includeLatestTag: true
        useDefaultContext: false
        buildContext: '.'
     
    - task: Docker@1
      displayName: 'Push the Docker image to my.private.registry.org'
      inputs:
        containerregistrytype: 'Container Registry'
        dockerRegistryEndpoint: 'MyRegistered_DockerRegistry'
        command: 'Push an image'
        ImageName: '$(ImageName)'
      condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
    - task: Docker@1
      displayName: 'Push the :latest image to my.private.registry.org'
      inputs:
        containerregistrytype: 'Container Registry'
        dockerRegistryEndpoint: 'MyRegistered_DockerRegistry'
        command: 'Push an image'
        ImageName: '<image-name>:latest'
      condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))