Check for unexpected exceptions using ScalaTest + ScalaCheck

Question

I'm trying to write a property that basically states "it should either not throw an exception, or throw one of a list of possible exceptions" using ScalaTest and it's GeneratorDrivenPropertyChecks which in turn uses scalatest. The issue is that I wasn't able to combine the noException with an logical or, so the best I could come with is this ugly test:

it should "not throw unexpected exceptions" in {
  forAll { (s: String) =>
    try { parse(s) }
    catch { case e:Throwable => e shouldBe a [ParsingFailedException] }
    true shouldBe true // prevent compile error
}}

What I would like to see instead would read more like

it should "not throw unexpected exceptions" in {
  forAll { (s: String) => {
    (noException should Be thrownBy) or (a [ParsingFailedException] shouldBe thrownBy) { parse(s)  }
}}

Is this difference non-deterministic? Because otherwise I'd suggest splitting your testing in two scenarios: one for when no exception should be thrown and one for one of the list should be thrown. In this way it's also clearer for other readers to know what should happen when — Diego Martinoia

Kulu Limpa Kulu Limpa · Accepted Answer · 2014-12-01T23:42:19

Since we want to use the exception as a value and not to control an exceptional flow, we could use scala.util.Try and make assertions on the value of Try. So instead of calling parse(s) we could call Try(parse(s)).

Unfortunately, because the values are wrapped now, I can't think of a clean way to assert predicates on them other than writing a custom matcher. For your specific example, a custom matcher could look as follows:

class FailAsMatcher[E <: Throwable](t: Class[E]) extends Matcher[Try[Any]] {
  def apply(theTry: Try[Any]) =
    MatchResult(
      theTry.isFailure && (theTry.failed.get.getClass == t),
      s"did not fail as a $t",
      s"did fail as a $t"
    )
}

Since Try is covariant, we can define the generic type of our custom matcher to be Try[Any]. The matcher matches only instances of Failure[Any] that fail with an exception of the provided type. Now, we could call this as:

it should "not throw unexpected exceptions" in {
  forAll { (s: String) =>
    Try(parse(s)) should ( be ('success) or failAs(classOf[ParsingFailedException]))
  }
}

def failAs[E <: Throwable](t: Class[E]) = new FailAsMatcher[E](t)

Now, if a non-expected exception happens, the error could look as follows:

TestFailedException was thrown during property evaluation.
  Message: Failure(java.lang.NullPointerException) was not success, and did not fail as a class ParsingFailedException

Check for unexpected exceptions using ScalaTest + ScalaCheck

1 Answers