Pass arguments to a specific test framework when using test-only with SBT

Question

I've got an SBT project using a couple of test frameworks (ScalaTest and ScalaCheck). For the most part this just works as expected. Eg. running test runs both variations of tests.

libraryDependencies ++= Seq(
  ...
  "org.scalatest"    %% "scalatest"    % "2.2.4"  % "test",
  "org.scalacheck"   %% "scalacheck"   % "1.12.5" % "test"
)

However when running test-only, when I try to supply an argument to ScalaCheck, ScalaTest steps in and complains:

test-only *SomeProp -- -s 10

[error] (test:testOnly) java.lang.IllegalArgumentException: Specifying a suite (-s ) or nested suite (-i ) is not supported when running ScalaTest from sbt; Please use sbt's test-only instead.

In the above example ScalaTest has been invoked with the -s argument instead of ScalaCheck.

By looking at the test fingerprints for ScalaTest:

superclass is "org.scalatest.Suite" or annotated with: "org.scalatest.WrapWith"

and ScalaCheck:

superclass is either "org.scalacheck.Properties" or "org.scalacheck.Prop"

I assume there should be no confusion on the part of SBT as to which framework to run for a given type of test given the above fingerprints. (*SomeProps variants extend org.scalacheck.Properties.)

Is there some way for me to specify which test framework to invoke with test-only? If not do I need to configure SBT to run ScalaTest and ScalaCheck in different configurations?

ssanj ssanj · Accepted Answer · 2016-09-07T14:53:41

I managed to get this working by creating a separate config for property-based tests (ScalaCheck). I've explained the changes to my build.sbt below.

I kept ScalaTest as the "default" test framework in the test config:

libraryDependencies ++= Seq(
  ...
  "org.scalatest"    %% "scalatest"    % "2.2.4"  % "test"
)

I created a separate prop config to run ScalaCheck properties. I had to move all my property-based tests from src/test/scala to src/prop/scala. I also scoped my ScalaCheck dependency to the prop config. This made it available only when used in the prop config. Similarly, ScalaTest would only be available in the test config.

lazy val root =
  Project("root", file(".")).
    configs(Prop).
    settings(inConfig(Prop)(Defaults.testSettings)).
    settings(libraryDependencies += scalacheck)

lazy val Prop = config("prop") extend (Runtime)

lazy val scalacheck = "org.scalacheck"   %% "scalacheck"   % "1.12.5" % "prop"

And now I can run both ScalaTest and ScalaCheck with arguments without any issues. So basically test and test-only run ScalaTest while prop:test and prop:test-only run ScalaCheck.

test-only *Spec -- -z "some content" //runs only ScalaTest

prop:test-only *SomeProp -- -s 10 //runs only ScalaCheck

More information on how to setup test configurations can be found in the SBT documentation.

The only downside is that I can't run test to run all the tests anymore. I have to resort to:

;test;prop:test

Creating an alias for the above would make things easier.

Pass arguments to a specific test framework when using test-only with SBT

1 Answers