Gitlab CI runner configuration with cache on docker

Question

I can't seem to get the cache or the artifacts to carry over between jobs in gitlab CI. I suspect it's something to do with my configuration, but I'm not sure what. I'm running gitlab and gitlab-ci-multirunner, both in docker, using the following docker-compose config. I've left out the database config and some environment variables for brevity:

version: '2'

services:
  gitlab:
    image: sameersbn/gitlab:8.5.1
    links:
      - redis:redisio
      - postgresql:postgresql
    ports:
      - "10080:80"
      - "10022:22"
    environment:
      ...
    volumes:
      - gitlab_data:/home/git/data

  gitlab-ci-runner:
    restart: always
    image: gitlab/gitlab-runner
    volumes:
      - gitlab_runner_config_data:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/nginx/ssl/gitlab.crt:/etc/gitlab-runner/certs/ca.crt
      - /etc/ssh:/ssh
    links:
      - gitlab:gitlab

  redis:
    ...
  postgresql:
    ...


volumes:
  postgresql_data:
  redis_data:
  gitlab_data:
  gitlab_runner_config_data:

The runner configuration (config.toml) is:

concurrent = 1

[[runners]]
  name = "docker"
  url = <public gitlab url>/ci 
  token = <gitlab token>
  tls-ca-file = "/etc/gitlab-runner/certs/ca.crt"
  executor = "docker"
  [runners.docker]
    image = "docker-bash"
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

The docker-bash image referred to is just the official docker:1.10 image with bash installed.

My build process consists of 3 steps:

1. Run npm install and the tests in the official node:5 image. For now, I've left this step out in order to test the deploy.
2. Build a docker image containing the code
3. Use ansible, via a custome built ansible docker image to deploy the built image to the production server.

The .gitlab-ci.yml file looks like this:

variables:
  FULL_IMAGE_TAG: deploy-$CI_BUILD_REF_NAME:$CI_BUILD_ID-$CI_BUILD_REF
  IMAGE_FILE: deploy-$CI_BUILD_REF_NAME.tar.gz

cache:
  paths:
    - $IMAGE_FILE

build:
  stage: build
  script:
    - docker build -t $FULL_IMAGE_TAG .
    - docker save $FULL_IMAGE_TAG | gzip -cf - > $IMAGE_FILE
  artifacts:
    paths:
      - $IMAGE_FILE

deploy:
  stage: deploy
  image: ansible-ssh
  script:
    - ls
    - ansible-playbook -e image_file=$IMAGE_FILE -e branch=$CI_BUILD_REF_NAME -e full_image_name=$FULL_IMAGE_TAG deploy-playbook.yml
  only:
    - develop
    - master

As you can see, the compressed docker image is referred to here both in the cache and artifacts sections, but is not actually available in the deploy step, where ansible is supposed to copy it to the remote machine. I have tried including an ls command so check the folder contents and the file is clearly not there, but it is definitely built and I can download it from the gitlab UI. Here is the log from the deploy job:

gitlab-ci-multi-runner 1.0.4 (014aa8c)
Using Docker executor with image ansible-ssh ...
Pulling docker image ansible-ssh ...
WARNING: Cannot pull the latest version of image ansible-ssh : Error: image library/ansible-ssh not found
WARNING: Locally found image will be used instead.

Running on runner-59d43cf3-project-8-concurrent-0 via 381c2ea97744...
Fetching changes...
Removing artifacts.zip
Removing deploy-develop.tar.gz
HEAD is now at 6009bd0 test
Checking out 6009bd0f as develop...
HEAD is now at 6009bd0... test

$ ls
Dockerfile
deploy-playbook.yml
server
$ ansible-playbook -e image_file=$IMAGE_FILE -e branch=$CI_BUILD_REF_NAME -e full_image_name=$FULL_IMAGE_TAG deploy-playbook.yml
Using /etc/ansible/ansible.cfg as config file
1 plays in deploy-playbook.yml

PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [deploy-host]

TASK [copy docker image] *******************************************************
task path: /builds/test/test/deploy-playbook.yml:44
fatal: [deploy-host]: FAILED! => {"changed": false, "failed": true, "msg": "could not find src=/builds/test/test/deploy-develop.tar.gz"}

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @deploy-playbook.retry

PLAY RECAP *********************************************************************
deploy-host            : ok=1    changed=0    unreachable=0    failed=1   


ERROR: Build failed with: exit code 1

I suspect that I am not setting up or using the runner correctly, but I can't find much in the documentation for anything beyond really simple cases and I don't know the tool well enough to know how it all fits together under the hood.

Answer 1

Caching is not designed to pass files between stages of a build.

From the doc

cache : Define list of files that should be cached between subsequent runs

I think what you need is in fact in progress : WIP: Download build artifacts from previous stages and restore them in context of the build (Technology Preview)

Answer 2

Did you enable artifacts in you gitlab.rb

gitlab_rails['artifacts_enabled'] = false

as described in the Build Artifacts documentation?

Answer 3

First of all, update gitlab and gitlab runner, especially the 1.0.4 runner was quiet experimental.

Secondly in the cache definition, you should add a key see https://docs.gitlab.com/ce/ci/yaml/README.html#cache-key

cache:
  key: "$CI_BUILD_REF_NAME"
  paths:
  - ..

From the https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#the-runners-section, you should modify the config.toml and add a cache directory

cache_dir:

directory where build caches will be stored in context of selected executor (Locally, Docker, SSH). If the docker executor is used, this directory needs to be included in its volumes parameter.

Answer 4

Caching is a bit strange, but essentially:

<dir path> under cache are available between build jobs while <dir path> artifacts will allow you to use it inside the same job.

So:

cache:
  untracked: true
  key: "$CI_BUILD_REF_NAME"
  paths:
    - cache-dir/

setup:
  stage: setup
  [snip]
  artifacts:
    paths:
     - cache-dir/ #notice that the path above is the same

This will allow you to cache your files between each build job while allowing you to use that same cache inside the same job.

Don't forget to add the files you need to artifacts on each build step.