Why e2e database tests failing within CI but not locally?

Question

I've got pipelines for dev, staging and production.

The staging pipeline is where I've got the issue. The pipeline builds just fine on dev (on and off the CI runner) but staging code builds only locally and on live server but will fail in the CI runner. I indicated suspecting code with <--.

I've checked whether the database container is running at the time of testing and it is up and running. Logs show nothing unusual.

Cypress tests fail on tests where interaction with the database is being tested:

test-ci.sh:

#!/bin/bash

env=$1
fails=""

inspect() {
  if [ $1 -ne 0 ]; then
    fails="${fails} $2"
  fi
}

# run server-side tests

dev() {
  docker-compose up -d --build
  docker-compose exec -T users python manage.py recreate_db
  docker-compose exec -T users python manage.py test
  inspect $? users
  docker-compose exec -T client npm test -- --coverage --watchAll --watchAll=false
  inspect $? client
  docker-compose down
}

# run e2e tests

e2e() {
  if [ "${env}" = "staging" ]; then
    docker-compose -f docker-compose-stage.yml up -d --build
    docker-compose -f docker-compose-stage.yml exec -T users python manage.py recreate_db  # <--
    docker run -e REACT_APP_USERS_SERVICE_URL=$REACT_APP_USERS_SERVICE_URL -v $PWD:/e2e -w /e2e -e CYPRESS_VIDEO=$CYPRESS_VIDEO --network flaskondocker_default cypress/included:6.0.0 --config baseUrl=http://nginx
    inspect $? e2e
    docker-compose -f docker-compose-stage.yml down
  else
    docker-compose -f docker-compose-prod.yml up -d --build
    docker-compose -f docker-compose-prod.yml exec -T users python manage.py recreate_db
    docker run -e REACT_APP_USERS_SERVICE_URL=$REACT_APP_USERS_SERVICE_URL -v $PWD:/e2e -w /e2e -e CYPRESS_VIDEO=$CYPRESS_VIDEO --network flaskondocker_default cypress/included:6.0.0 --config baseUrl=http://nginx
    inspect $? e2e
    docker-compose -f docker-compose-prod.yml down
  fi
  }

# run specific tests

if [ "${env}" = "staging" ]; then
  echo "****************************************"
  echo "Running e2e tests ..."
  echo "****************************************"
  e2e
elif [ "${env}" = "production" ]; then
  echo "****************************************"
  echo "Running e2e tests ..."
  echo "****************************************"
  e2e
else
  echo "****************************************"
  echo "Running client and server-side tests ..."
  echo "****************************************"
  dev
fi

if [ -n "${fails}" ]; then
  echo "Test failed: ${fails}"
  exit 1
else
  echo "Tests passed!"
  exit 0
fi

The tests are behaving like docker-compose -f docker-compose-stage.yml exec -T users python manage.py recreate_db failed or hasn't been executed but logs show no errors.

gitlab-ci.yml file:

image: docker:stable

services:
  - docker:19.03.12-dind

variables:
  COMMIT: ${CI_COMMIT_SHORT_SHA}
  MAIN_REPO: https://gitlab.com/coding_hedgehog/flaskondocker.git
  USERS: training-users
  USERS_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/users
  USERS_DB: training-users-db
  USERS_DB_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/users-db
  CLIENT: training-client
  CLIENT_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/client
  SWAGGER: training-swagger
  SWAGGER_REPO: ${MAIN_REPO}#${CI_COMMIT_BRANCH}:services/swagger

stages:
  - build
  - push

before_script:
  - export REACT_APP_USERS_SERVICE_URL=http://127.0.0.1
  - export CYPRESS_VIDEO=false
  - export SECRET_KEY=pythonrocks
  - export AWS_ACCOUNT_ID=nada
  - export AWS_ACCESS_KEY_ID=nada
  - export AWS_SECRET_ACCESS_KEY=nada
  - apk add --no-cache py-pip python2-dev python3-dev libffi-dev openssl-dev gcc libc-dev make npm
  - pip install docker-compose
  - npm install

compile:
  stage: build
  script:
    - docker pull cypress/included:6.0.0
    - sh test-ci.sh $CI_COMMIT_BRANCH

deployment:
  stage: push
  script:
    - sh ./docker-push.sh
  when: on_success

Let me just emphasize that the tests are passing locally on my computer as well as on live server. The database-related e2e tests fail when ran headlessly within CI.

What debugging steps I can take knowing that no containers are crashing, logs show no errors, same code builds locally and runs OK live but fails in the CI ?

Boyd K. Boyd K. · Accepted Answer · 2020-12-22T09:01:03

We have had some issues where database checks worked locally, but not in headless CI. We found out that it was because of datetime fields. The markup response in CI was different than locally. Thus, all assertions that checked dates failed. We fixed this by writing MySQL queries that format the datetime result. Then adjust the assertions in Cypress accordingly. Maybe your problem has to do with this issue.

SELECT DATE_FORMAT(columnname, "%d-%c-%Y") as columnname FROM table

So for further debugging, do you have any simple tests that run correctly in CI? Or does nothing work?

Why e2e database tests failing within CI but not locally?

1 Answers