Cucumber tests fail but travis build still passes

Question

I am using Travis for CI. For some reason, the builds pass even when some tests fail. See the full log here

https://travis-ci.org/msm1089/hobnob/jobs/534173396

The way I am running the tests is via a bash script, e2e.test.sh, that is run by yarn.

Searching for this specific issue has not turned up anything that helps. It is something to do with exit codes I believe. I think I need to somehow get the build to exit with non-zero, but as you can see at bottom of the log, yarn exits with 0.

e2e.test.sh

#!/usr/bin/env bash

RETRY_INTERVAL=${RETRY_INTERVAL:-0.2}

# Run our API server as a background process
if [[ "$OSTYPE" == "msys" ]]; then
        if ! netstat -aon | grep "0.0.0.0:$SERVER_PORT" | grep "LISTENING"; then
                pm2 start --no-autorestart --name test:serve "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" -- run test:serve
                until netstat -aon | grep "0.0.0.0:$SERVER_PORT" | grep "LISTENING"; do
                sleep $RETRY_INTERVAL
                done
        fi
else
        if ! ss -lnt | grep -q :$SERVER_PORT; then
                yarn run test:serve &
        fi
        until ss -lnt | grep -q :$SERVER_PORT; do
          sleep $RETRY_INTERVAL
        done
fi
npx cucumber-js spec/cucumber/features --require-module @babel/register --require spec/cucumber/steps

if [[ "$OSTYPE" == "msys" ]]; then
        pm2 delete test:serve
fi

travis.yml

language: node_js
node_js:
  - 'node'
  - 'lts/*'
  - '10'
  - '10.15.3'
services:
  - elasticsearch
before_install:
  - curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.deb
  - sudo dpkg -i --force-confnew elasticsearch-6.6.1.deb
  - sudo service elasticsearch restart
before_script:
  - sleep 10
env:
  global:
    - NODE_ENV=test
    - SERVER_PROTOCOL=http
    - SERVER_HOSTNAME=localhost
    - SERVER_PORT=8888
    - ELASTICSEARCH_PROTOCOL=http
    - ELASTICSEARCH_HOSTNAME=localhost
    - ELASTICSEARCH_PORT=9200
    - ELASTICSEARCH_INDEX=test

package.json

...
scripts:{
    "test": "yarn run test:unit && yarn run test:integration && yarn run test:e2e"
}
...

So, how can I ensure that the cucumber exit code is the one that is returned, so that the build fails as it should when the tests don't pass?

DannyB DannyB · Accepted Answer · 2019-05-20T05:49:47

There are a few possible ways to solve this. Here are two of my favorite.

Option 1:

Add set -e at the top of your bash script, so that it exits on first error, preserving the exit code, and subsequently, failing Travis if its a non zero.

Option 2:

Capture whatever exit code you want, and exit with it wherever it makes sense.

run whatever command here
exitcode=$?
[[ $exitcode == 0 ]] || exit $exitcode

As a side note - it seems like your bash script has too many responsibilities. I would consider separating them if possible, and then you give travis a list of commands to run, and possibly one or two before_script commands.

Something along these lines:

# .travis.yml
before_script:
- ./start_server.sh

script:
- npx cucumber-js spec/cucumber/features ...

Cucumber tests fail but travis build still passes

1 Answers