How to Run Multiple Test Files with Haskell Stack Project

Question

I want to setup an existing Haskell project with Stack. The existing project uses multiple files under a test directory; these separate test files by default, Stack (or cabal?) appears to utilize a single test/Spec.hs for testing. How can I continue to use multiple files with this project?

NOTE: I'm learning Haskell, and this project approaches my learning from a "kata" approach. So tests are isolate to focus on one aspect of the language at a time.

Answer 1

Here is a setup for a directory structure like this

> tree                                                                                       
.
├── example.cabal
├── app
│   └── Main.hs
├── ChangeLog.md
├── LICENSE
├── Setup.hs
├── src
│   ├── A
│   │   └── C.hs
│   ├── A.hs
│   └── B.hs
├── stack.yaml
└── tst
    ├── integration
    │   └── Spec.hs
    └── unit
        ├── A
        │   └── CSpec.hs
        ├── ASpec.hs
        ├── BSpec.hs
        └── Spec.hs

you want to have integration tests that are separate from the usual unit tests and several sub-modules that correspond to each module in your src-folder

first of all you need to add the test suites to your

example.cabal file

name:                example
...
-- copyright:
-- category:
build-type:          Simple
extra-source-files:  ChangeLog.md
cabal-version:       >=1.10

executable testmain
  main-is:       Main.hs
  hs-source-dirs: app
  build-depends: base
               , example

library
  exposed-modules:     A.C,A,B
  -- other-modules:
  -- other-extensions:
  build-depends:       base >=4.9 && <4.10
  hs-source-dirs:      src
  default-language:    Haskell2010

test-suite unit-tests
  type:          exitcode-stdio-1.0
  main-is:       Spec.hs
  hs-source-dirs: tst/unit
  build-depends: base
               , example
               , hspec
               , hspec-discover
               , ...

test-suite integration-tests
  type:          exitcode-stdio-1.0
  main-is:       Spec.hs
  hs-source-dirs: tst/integration
  build-depends: base
               , example
               , hspec
               , ...

put the following in your tst/unit/Spec.hs it is from hspec-discover and it discovers (hence the name) all modules of the form ...Spec.hs and executes the spec function from each of those modules.

tst/unit/Spec.hs

{-# OPTIONS_GHC -F -pgmF hspec-discover #-}

just this single line

Other test files

then add your unit tests in your ASpec.hs, and others in BSpec.hs,CSpec.hs and your Spec.hs in the tst/integration folder

module ASpec where

import Test.Hspec
import A

spec :: Spec
spec = do
  describe "Prelude.head" $ do
    it "returns the first element of a list" $ do
      head [23 ..] `shouldBe` (23 :: Int)

    it "returns the first element of an *arbitrary* list" $
      property $ \x xs -> head (x:xs) == (x :: Int)

    it "throws an exception if used with an empty list" $ do
      evaluate (head []) `shouldThrow` anyException

you can then compile and run your tests with

$> stack test
# now all your tests are executed
$> stack test :unit-tests
# now only the unit tests run
$> stack test :integration-tests
# now only the integration tests run

Sources

You can find all the examples at https://hspec.github.io, if you want to know more about hspec-style testing I guess it would be best to start there. For the stack - go to https://haskellstack.org - there is some information about testing/benchmarking there - I mean about running tests and benchmarks.

For different testing style in haskell see HUnit, QuickCheck, Smallcheck, doctests (If I forgot one, my dearest apologies - those are the ones that I use regularly as well).

Answer 2

Here is a solution with just stack and HUnit. Nothing against hspec, htf, tasty, etc etc, but likewise there is not much glue needed even without those, if you're already using HUnit. It doesn't require editing the cabal file. The original question implies use of hspec, so the @epsilonhalbe is still closer on that criteria.

.
├─stack.yaml
├─package.yaml
├─src/
|   ├─A.hs
|   ├─B.hs
|   ├─Main.hs
├─tst/
|   ├─ATest.hs
|   ├─BTest.hs
|   ├─Main.hs

Example package.yaml file:

name:                example
version:             0.1.0.0

dependencies:
- HUnit >= 1.6.1.0 && < 2

library:
  source-dirs: src

executables:
  example-exe:
    main:                Main.hs
    source-dirs:         src
    dependencies:
    - example

tests:
  example-test:
    main:                Main.hs
    source-dirs:         tst
    dependencies:
    - example
    - HUnit

In ATest.hs and BTest.hs declare list of tests called huTests in the usual HUnit way, eg,

huTests = ["egTest" ~: "a" ~=? "a"].

Then tst/Main.hs has the glue in a common HUnit idiom (see eg this answer):

import ATest (huTests)
import BTest (huTests)

import System.Exit
import Test.HUnit

main :: IO ()
main = do
    results <- runTestTT $
                    test (ATest.huTests ++ BTest.huTests)
    if errors results + failures results == 0 then
        putStrLn "Tests passed."
    else
        die "Tests failed."