Parallelize functional tests while using a SQLite database

Question

I have added many tests in my Symfony2 project, but now with 53 tests and 176 assertions it takes about a minute. I'm trying to reduce this time because if I enable code coverage report it takes 15 minutes.

$ phpunit -c app/phpunit.xml.dist PHPUnit 4.3.5 by Sebastian Bergmann.
Configuration read from app/phpunit.xml.dist
.....................................................
Time: 59.1 seconds, Memory: 361.00Mb
OK (53 tests, 176 assertions)

I've configured LiipFunctionalTestBundle correctly to use a SQLite database in the test environment (this is recommended by LiipFunctionalTestBundle):

app/config/config_test.yml

doctrine:
    dbal:
        default_connection: default
        connections:
            default:
                driver:   pdo_sqlite
                path:     %kernel.cache_dir%/test.db

And I've added fixtures with DoctrineFixturesBundle.

Now I have to create tests suites or groups to launch several instances of PHPUnit. But I'm seeing a future problem: how can tests can be parallelized (for example with paratest) if two or more instances of phpunit write and read data in the same SQLite file?

I can Pass a variable to PhpUnit to change the kernel.cache_dir value and create one cache directory per phpunit instance. But it can't be done from the command line, so if I choose this solution I'll have to create several phpunit.xml.dist. I'm looking for a more convenient solution.

53 functional tests in one minute? I don't think there is any performance problem. Besides, IMO code coverage should be used only for unit tests. — gontrollez
@gontrollez can you please explain briefly what do you mean by unit tests? Are they targeted tests which focus on a few pages, functions, etc. Do you suggest to run code coverage analysis only once a day for example? — A.L

A.L A.L · Accepted Answer · 2015-09-15T13:21:54

I found a workaround by creating as many cache directories as test files.

I defined 8 test suites to fit with the 8 cores of my CPU:

<!-- app/phpunit.xml.dist -->
<?xml version="1.0" encoding="UTF-8"?>

<phpunit
    backupGlobals = "false"
    …
>
    …
    <testsuites>
        <testsuite name="Group1">
            <file>../src/AcmeBundle/Tests/Command/CommandTest.php</file>
            <file>…</file>
        </testsuite>
        …
        <testsuite name="Group8">
            <file>../src/AcmeBundle/Tests/Controller/AdminControllerTest.php</file>
        </testsuite>
    </testsuites>
</phpunit>

I created a phpunit.sh script which use export (as suggested by David Harkness in order to define a specific path for each test suite:

#!/bin/bash

function test() {
    testsuite=$1
    export CACHE_PATH="$testsuite"
    OUTPUT=$(phpunit -c app/phpunit.xml.dist --testsuite $testsuite)
    RETURN=$?
    if [ $RETURN -eq 0 ]
    then
        echo -e -n "\033[01;32m●  OK\t\033[00m \033[01;35m$testsuite\033[00m"
        tail=$(echo "$OUTPUT" | tail -n 3)
        echo -e "\t\"$tail\"" | tr '\n' '\t' | tr -s '\t'
        echo ""
    else
        echo -e "\033[01;31m❌  ERROR\033[00m \033[01;35m$testsuite\033[00m (\033[01;34m$RETURN\033[00m)\033[00m"
        echo "-----"
        echo "$OUTPUT"
        echo "-----"
    fi
}

for testsuite in $(seq 1 8)
do
    tester "Group$testsuite" &
done

# http://unix.stackexchange.com/questions/231602/how-to-detect-that-all-the-child-processes-launched-in-a-script-ended/231608#231608
wait

echo "Done"

Successful test suites are displayed with with green marks if everything is OK, and the output of phpunit will be truncated to keep a small output. If there is an error it will be displayed.

Then I added this to app/AppKernel.php:

/**
 * @see http://symfony.com/fr/doc/2.3/cookbook/configuration/override_dir_structure.html#surcharger-le-repertoire-cache
 */
public function getCacheDir()
{
    $cachePath = getenv('CACHE_PATH');

    if (
        ($this->environment == 'test')
        &&
        (! empty($cachePath))
    ) {
        return(parent::getCacheDir().'/'.$cachePath.'/');
    }

    # else

    return parent::getCacheDir();
}

This code will tell Symfony to create a sub-directory in the cache folder, meaning than multiple instances of PHPUnit won't use the same SQLite file.

While it works, this solution is not perfect and have some drawbacks:

it recreates the entire cache even if I only wanted to change the path of the SQLite database
it will use space and take time to create one cache per rest file, I suggest you to avoid this if you don't use the RAM as a cache (my solution is to delete cache in app/ and create a symlink: ln -s /run/shm/project/cache cache, tests are faster because caching is done in RAM and not on the hard drive)

Parallelize functional tests while using a SQLite database

2 Answers