How to make ctest run test executables in a transient/temporary directory

Question

How can I make ctest run each of my tests in a separate transient/temporary directory each time I run $ make test (or $ctest).

Let's say I have a test executable, mytest.cpp that does two things: 1) It asserts that a file called "foo.txt" does not exist in the current working directory and then 2) creates a file called "foo.txt". Now I want to be able to run make test multiple times without mytest.cpp to fail.

I want to achieve this by asking cmake/ctest to run every test (in this example, one test) in its own temporary directory.

I've searched for solutions online and I've read through the ctest documentation. In particular the add_test docs. I can provide a "WORKING_DIRECTORY" to add_test. This will run my test in that "WORKING_DIRECTORY". However any changes made to this folder persist across multiple make test runs. So the second time I run make test the test fails.

Here's a minimal, reproducible way of triggering the failure. One source file mytest.cpp that defines the test executable and a CMakeLists.txt file to build the code.

# CMakeLists.txt
cmake_minimum_required (VERSION 2.8)
project (CMakeHelloWorld)
enable_testing()

add_executable (mytest mytest.cpp)
add_test( testname mytest)

and

// mytest.cpp
#include <sys/stat.h>
#include <unistd.h>
#include <string>
#include <fstream>

inline bool exists (const std::string& name) {
    std::ifstream f(name.c_str());
    return f.good();
}

int main() {
    assert(exists("foo.txt") == false);
    std::ofstream outfile ("foo.txt");
    outfile.close();
}

Series of command that generate the failure

$ mkdir build
$ cd build
$ cmake ..
$ make
$ make test
$ make test

This will give

Running tests...
Test project /path/to/project
    Start 1: testname
1/1 Test #1: testname .........................***Exception: Other  0.25 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.26 sec

The following tests FAILED:
          1 - testname (OTHER_FAULT)
Errors while running CTest
make: *** [test] Error 8

Stanley F. Stanley F. · Accepted Answer · 2019-04-11T05:32:39

Typically, a testing framework provides some kind of pre-test (setup) and post-test (cleanup) tasks. And so does CTest.

Adding the following CTestCustom.ctest file to the build directory of your example makes the test succeed every time:

# CTestCustom.ctest
set(CTEST_CUSTOM_POST_TEST "rm foo.txt")

For more complex tasks, you may want to create a custom script, but that's the way to call it.

How to make ctest run test executables in a transient/temporary directory

1 Answers