We have several regression tests in our piece of software taking advantage of the autotools (autoconf&automake) infrastructure. These regression tests are splitted among different subdirectories according to the functionalities/units they test. We have found that once one of the subdirectories fails in any test, the subsequent tests in forthcoming subdirectories are not executed and therefore we do not get the whole picture of which functionalities keep working as expected. Is there a way to change this behavior and force the execution of all tests even if there are some tests that fail?
Consider the following minimal example in which subdir1 fails and then the make check does not progresses to subdir2. The output (after generating the configure through autoreconf -fiv and a simple ./configure call) using make check is:
...
make[2]: Entering directory `x/subdir1'
make[3]: Entering directory `x/subdir1'
FAIL: fail
PASS: pass
make[4]: Entering directory `x/subdir1'
make[4]: Nothing to be done for `all'.
make[4]: Leaving directory `x/subdir1'
============================================================================
Testsuite summary for test 1.0
============================================================================
# TOTAL: 2
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See subdir1/test-suite.log
============================================================================
make[3]: *** [test-suite.log] Error 1
make[3]: Leaving directory `x/subdir1'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory `x/subdir1'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory `x/subdir1'
make: *** [check-recursive] Error 1
The files for this smal test are:
configure.ac
AC_INIT([test], [1.0])
AM_INIT_AUTOMAKE()
AC_PROG_CC
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([
Makefile
subdir1/Makefile
subdir2/Makefile
])
AC_OUTPUT
subdir1/Makefile.am
check_PROGRAMS = fail pass
TESTS = fail pass
fail_SOURCES = fail.c
pass_SOURCES = pass.c
subdir2/Makefile.am
check_PROGRAMS = pass
TESTS = pass
pass_SOURCES = pass.c
and the shared sources for fail.c and pass.c
pass.c
int main (void)
{ return 0; }
fail.c
int main (void)
{ return 1; }
