tasty-discover with stack run 0 tests

Question

I've just moved from using cabal directly to stack, but I'm having some problems with the test-suite. The library and executable works fine but stack test gives me

hml-0.1.0.0: test (suite: HML-test) 
All 0 tests passed (0.00s)

I.e. no tests were run. When I run the tests with cabal directly they work fine. I'm using tasty with tasty-discover. In hml.cabal the suite looks like this:

test-suite HML-test
  main-is:             Driver.hs
  hs-source-dirs:      test
  default-language:    Haskell2010
  ghc-options:         -Wall
  other-modules:       HML.Types.Test.PosInt
                       (...etc...)
  build-depends:       base >= 4.9
                       (... etc...)

The driver test/Driver.hs for tasty-discover looks like this:

{-# OPTIONS_GHC -F -pgmF tasty-discover -optF --tree-display #-}

Any ideas?

EDIT:

When compiling with cabal I find the executable HML-test in dist/build/HML-test, and running it directly runs all the tests.

When compiling with stack I find it at .stack-work/x86_64-linux/Cabal-1.24.2.0/build/HML-test, and running it directly still runs 0 tests.

The full output of stack - v test is

Version 1.5.1, Git revision 600c1f01435a10d127938709556c1682ecfd694e (4861 commits) x86_64 hpack-0.17.1 2017-11-19 12:25:05.759671: [debug] Checking for project config at: /home/cabox/workspace/hml/stack.yaml @(Stack/Config.hs:974:9) 2017-11-19 12:25:05.759980: [debug] Loading project config file stack.yaml @(Stack/Config.hs:999:13) 2017-11-19 12:25:05.762372: [debug] Trying to decode /home/cabox/.stack/build-plan-cache/x86_64-linux/lts-9.13.cache @(Data/Store/VersionTagged.hs:72:5) 2017-11-19 12:25:05.788669: [debug] Success decoding /home/cabox/.stack/build-plan-cache/x86_64-linux/lts-9.13.cache @(Data/Store/VersionTagged.hs:76:13) 2017-11-19 12:25:05.789952: [debug] Run process: /sbin/ldconfig -p @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.796009: [debug] Process finished in [92m5ms[0m: /sbin/ldconfig -p @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.796327: [debug] Run process: /usr/bin/gcc -v @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.799566: [debug] Process finished in [92m3ms[0m: /usr/bin/gcc -v @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.799710: [debug] PIE enabled @(Stack/Setup.hs:583:17) 2017-11-19 12:25:05.799912: [debug] Found shared library libtinfo.so.5 in 'ldconfig -p' output @(Stack/Setup.hs:559:29) 2017-11-19 12:25:05.800147: [debug] Did not find shared library libtinfo.so.6 @(Stack/Setup.hs:573:38) 2017-11-19 12:25:05.800285: [debug] Did not find shared library libncursesw.so.6 @(Stack/Setup.hs:573:38) 2017-11-19 12:25:05.800375: [debug] Found shared library libgmp.so.10 in 'ldconfig -p' output @(Stack/Setup.hs:559:29) 2017-11-19 12:25:05.800478: [debug] Did not find shared library libgmp.so.3 @(Stack/Setup.hs:573:38) 2017-11-19 12:25:05.800560: [debug] Using standard GHC build @(Stack/Setup.hs:606:9) 2017-11-19 12:25:05.801346: [debug] Asking GHC for its version @(Stack/Setup/Installed.hs:103:13) 2017-11-19 12:25:05.801592: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc --numeric-version @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.802203: [debug] Getting Cabal package version @(Stack/GhcPkg.hs:189:5) 2017-11-19 12:25:05.802507: [debug] Getting global package database location @(Stack/GhcPkg.hs:55:5) 2017-11-19 12:25:05.808193: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.808687: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --no-user-package-db list --global @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.855073: [debug] Process finished in [92m46ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --no-user-package-db list --global @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.857161: [debug] Process finished in [92m48ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --no-user-package-db field --simple-output Cabal version @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.869048: [debug] Process finished in [92m67ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc --numeric-version @(System/Process/Read.hs:306:3) 2017-11-19 12:25:05.871256: [debug] Resolving package entries @(Stack/Setup.hs:260:5) 2017-11-19 12:25:05.897711: [debug] Starting to execute command inside EnvConfig @(Stack/Runners.hs:175:18) 2017-11-19 12:25:05.897916: [debug] Parsing the cabal files of the local packages @(Stack/Build/Source.hs:328:5) 2017-11-19 12:25:05.905436: [debug] Parsing the targets @(Stack/Build/Source.hs:265:5) 2017-11-19 12:25:05.917919: [debug] Start: getPackageFiles /home/cabox/workspace/hml/hml.cabal @(Stack/Package.hs:259:21) 2017-11-19 12:25:06.009229: [debug] Finished in 91ms: getPackageFiles /home/cabox/workspace/hml/hml.cabal @(Stack/Package.hs:259:21) 2017-11-19 12:25:06.010508: [debug] Finding out which packages are already installed @(Stack/Build/Installed.hs:69:5) 2017-11-19 12:25:06.010877: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.062451: [debug] Process finished in [92m51ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.086280: [debug] Ignoring package haskeline due to wanting version 0.7.4.0 instead of 0.7.3.0 @(Stack/Build/Installed.hs:199:5) 2017-11-19 12:25:06.086478: [debug] Ignoring package terminfo due to wanting version 0.4.1.0 instead of 0.4.0.2 @(Stack/Build/Installed.hs:199:5) 2017-11-19 12:25:06.086581: [debug] Ignoring package xhtml due to wanting version 3000.2.2 instead of 3000.2.1 @(Stack/Build/Installed.hs:199:5) 2017-11-19 12:25:06.086945: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --user --no-user-package-db --package-db /home/cabox/.stack/snapshots/x86_64-linux/lts-9.13/8.0.2/pkgdb dump --expand-pkgroot @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.209627: [debug] Process finished in [92m122ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --user --no-user-package-db --package-db /home/cabox/.stack/snapshots/x86_64-linux/lts-9.13/8.0.2/pkgdb dump --expand-pkgroot @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.210994: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --user --no-user-package-db --package-db /home/cabox/workspace/hml/.stack-work/install/x86_64-linux/lts-9.13/8.0.2/pkgdb dump --expand-pkgroot @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.255376: [debug] Process finished in [92m43ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --user --no-user-package-db --package-db /home/cabox/workspace/hml/.stack-work/install/x86_64-linux/lts-9.13/8.0.2/pkgdb dump --expand-pkgroot @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.256147: [debug] Constructing the build plan @(Stack/Build/ConstructPlan.hs:188:5) 2017-11-19 12:25:06.270612: [debug] Checking if we are going to build multiple executables with the same name @(Stack/Build.hs:210:5) 2017-11-19 12:25:06.271198: [debug] Executing the build plan @(Stack/Build/Execute.hs:478:5) 2017-11-19 12:25:06.282278: [debug] Getting global package database location @(Stack/GhcPkg.hs:55:5) 2017-11-19 12:25:06.282498: [debug] Run process: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --no-user-package-db list --global @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.319127: [debug] Process finished in [92m36ms[0m: /home/cabox/.stack/programs/x86_64-linux/ghc-8.0.2/bin/ghc-pkg --no-user-package-db list --global @(System/Process/Read.hs:306:3) 2017-11-19 12:25:06.320228: [info] hml-0.1.0.0: test (suite: HML-test) @(Stack/Build/Execute.hs:830:23) 2017-11-19 12:25:06.321702: [debug] Creating process: /home/cabox/workspace/hml/.stack-work/dist/x86_64-linux/Cabal-1.24.2.0/build/HML-test/HML-test @(System/Process/Run.hs:139:5) 2017-11-19 12:25:06.344988: [info] @(Stack/Build/Execute.hs:1587:52) All 0 tests passed (0.00s)

EDIT2: If I organize tasty manually (i.e. refrain from using, tasty-discover) it works. This, however would lead to a lot of work and a lot of boilerplate..

I never used tasty. But stack test works fine with Hspec. Maybe running test with --verbose flag could help in someway. — palik
@palik As far as I can tell there are no hints in the -v output; I included it in the OP — jorgen
I could reproduce the issue. But there is no issue by running the tests without {-# OPTIONS_GHC -F -pgmF tasty-discover #-} and with defaultMain instead. In stack test output appears the line with "Compiling XYZ". XYZ is the module, that couldn't be discovered. — palik
@palik By using defaultMain , do you mean listing the tests manually (i.e. with testGroup, testProperty etc), or are they still discovered automatically? — jorgen

jorgen jorgen · Accepted Answer · 2017-11-20T05:19:03

The closest I've gotten to a solution is to move from tasty-discover to tasty-th.

The full automatic discovery still doesn't work, but using the TemplateHaskell through $(testGroupGenerator) the boilerplate at least scales with the number of test files rather than the number of tests.

tasty-discover with stack run 0 tests

1 Answers