sbt integration tests returning exit code 0 when tests fail

Question

I am using sbt 0.13.9 (and previously 0.13.8) and Scala 2.11.6.

When running integration tests for my project, sbt always returns "success" with exit code 0 even when it acknowledges test failures. The integration test configuration and settings are the stock ones from sbt (see build.sbt below).

Notice how the failure does not print out a "Failed tests:" section unlike the unit test failure.

This is odd compared to the unit tests, which do correctly trigger sbt to return an exit code of 1.

With both my unit and integration tests, I had to enable forking to avoid some classloader issues due to dynamically loading a Java library. If forking was the problem, I would assume that I would see the issue with unit tests as well.

My integration tests do involve starting external processes, which I don't know if that affects the results from sbt or not.

The project in question can be found here: https://github.com/chipsenkbeil/scala-debugger

The project's build.sbt is as follows:

//
// DEBUGGER API PROJECT CONFIGURATION
//
lazy val scalaDebuggerApi = project
  .in(file("scala-debugger-api"))
  .configs(IntegrationTest)
  .settings(Common.settings: _*)
  .settings(Defaults.itSettings: _*)
  .settings(Seq(
    name := "scala-debugger-api",

    // NOTE: Fork needed to avoid mixing in sbt classloader, which is causing
    //       LinkageError to be thrown for JDI-based classes
    fork in Test := true,
    fork in IntegrationTest := true,

    libraryDependencies ++= Seq(
      "org.slf4j" % "slf4j-api" % "1.7.5",
      "org.slf4j" % "slf4j-log4j12" % "1.7.5" % "test,it",
      "log4j" % "log4j" % "1.2.17" % "test,it",
      "org.scalatest" %% "scalatest" % "2.2.1" % "test,it",
      "org.scalamock" %% "scalamock-scalatest-support" % "3.2.1" % "test,it"
    ),
    // JDK Dependency (just for sbt, must exist on classpath for execution,
    // cannot be redistributed)
    internalDependencyClasspath in Compile +=
      { Attributed.blank(Build.JavaTools) },
    internalDependencyClasspath in Runtime +=
      { Attributed.blank(Build.JavaTools) },
    internalDependencyClasspath in Test +=
      { Attributed.blank(Build.JavaTools) },
    internalDependencyClasspath in IntegrationTest +=
      { Attributed.blank(Build.JavaTools) }
  ): _*)
  .dependsOn(scalaDebuggerTest % "test->compile;it->compile")

//
// DEBUGGER TEST CODE PROJECT CONFIGURATION
//
lazy val scalaDebuggerTest = project
  .in(file("scala-debugger-test"))
  .settings(Common.settings: _*)
  .settings(
    // Do not publish the test project
    publishArtifact := false,
    publishLocal := {}
  )

//
// MAIN PROJECT CONFIGURATION
//
lazy val root = project
  .in(file("."))
  .settings(Common.settings: _*)
  .settings(
    name := "scala-debugger",
    // Do not publish the aggregation project
    publishArtifact := false,
    publishLocal := {}
  ).aggregate(scalaDebuggerApi, scalaDebuggerTest)

yǝsʞǝla yǝsʞǝla · Accepted Answer · 2015-10-08T04:20:11

Looks like you hit an open issue: https://github.com/sbt/sbt/issues/1890. It's related to: https://github.com/scalatest/scalatest/issues/432. Try to get the latest version and see if it's at least partially fixed, i.e. when not running in parallel.

sbt integration tests returning exit code 0 when tests fail

1 Answers