Gitlab pipeline test stage to fail AND create artifacts anyway

9
votes

I have a gitlab pipeline running on a windows machine with Windows 7 and powershell 4.0.

The .yaml has the typical 3 stages: build, test and deploy.

For the second stage I want to perform some simple tests that generate a log file which should be available after the test stage finishes.

Here the script section from the test:

script:
  - '$exitCode = (start-process C:\app_versions\app_20181211\bin\app.exe -PassThru -Wait).ExitCode'
  - 'cat .\TestLogs\BasicFunctionsTestPlan.log'
  - 'exit $exitCode'
artifacts:
  paths:
    - .\TestLogs
  expire_in: 1 year

Here I had one problem, after the test run has finished the stage finishes always successfully even if the test themselves failed. Then I had to force the script exit with an error code in case the application tells me that the tests failed.

This caused the second problem: the artifacts link do not get created even they are available (my test produce it anyway).

Probably if I knew how to tell gitlab that the test failed in a more clean way, the artifacts would be available anyway.

I agree that the log file is not an artifact but I would like to keep that file in order to check how the tests have performed, maybe there is a better way to save this file.

Thanks in advance for your help!

EDIT:

Looks like there were more people having the same issue here, maybe it helps understanding better the problem.

1
continuous-integrationautomated-testsgitlabpowershell-4.0
Doesn't Gitlab already store the pipeline output? You should be able to review each pipeline step and its output via the pipeline history (CI/CD > Pipelines). – Ansgar Wiechers
Yes, but I would like to have as well the test output in a log file... (if possible!) – juagicre
I am not sure I understand. Your test is failing now and you want to have the logs as artifacts? – do3cc
Yes. Whether the test fail or not the logs should be available. If the last statement form the script, so 'exit $exitCode' is 0->test finished OK, so artifacts get created. If $exitCode !=0 -> test NOT OK, gitlab does not execute the artifacts section (could make sense if output are binaries which if test fails nobody will use). So, is there any way to even the test fails getting the artifacts? – juagicre

1 Answers

13
votes

I had the same question, but it's easily solved:

You can use artifacts:when to upload artifacts on job failure or despite the failure.

artifacts:when

source: Gitlab CI yaml reference: artifacts:when

Introduced in GitLab 8.9 and GitLab Runner v1.3.0.

artifacts:when is used to upload artifacts on job failure or despite the failure.

artifacts:when can be set to one of the following values:

  1. on_success - upload artifacts only when the job succeeds. This is the default.
  2. on_failure - upload artifacts only when the job fails.
  3. always - upload artifacts regardless of the job status.

Example:

To upload artifacts only when job fails:

job:
  artifacts:
    when: on_failure

allow_failure

BTW: you can tell Gitlab CI to continue to the next job after a job failure with allow_failure: true

source: Gitlab CI yaml Reference: allow_failure

job1:
  stage: test
  script:
    - execute_script_that_will_fail
  allow_failure: true

So combined it could look something like:

job1:
  stage: test
  script:
    - execute_script_that_will_fail
  allow_failure: true
  artifacts:
    when: always # or 'on_failure'
    paths:
    - resulting_artifacts