Reproducing a ScalaCheck test run

6

votes

This was asked as a "bonus question" in https://stackguides.com/questions/12639454/make-scalacheck-tests-deterministic, but not answered:

Is there a way to print out the random seed used by ScalaCheck, so that you can reproduce a specific test run?

There is a hacky way: wrap a random generator to print its seed on initialization and pass it to Test.Parameters. Is there a better option?

scalacheck

3

votes

As of today, this is possible (see scalacheck#263). There are some nice examples here: Simple example of using seeds with ScalaCheck for deterministic property-based testing.

In short, you can do:

propertyWithSeed("your property", Some("seed")) =
  forAll { ??? }

and the seed will be printed when this property fails.

2

votes

There is no way to do this today. However, it will be implemented in the future, see https://github.com/rickynils/scalacheck/issues/67

1

votes

This is my answer there:

Bonus question: Is there an official way to print out the random seed used by ScalaCheck, so that you can reproduce even a non-deterministic test run?

From specs2-scalacheck version 4.6.0 this is now a default behaviour:

Given the test file HelloSpec:

package example

import org.specs2.mutable.Specification
import org.specs2.ScalaCheck

class HelloSpec extends Specification  with ScalaCheck {
package example

import org.specs2.mutable.Specification
import org.specs2.ScalaCheck

class HelloSpec extends Specification  with ScalaCheck {
  s2"""
    a simple property       $ex1
  """

  def ex1 = prop((s: String) => s.reverse.reverse must_== "")
}

build.sbt config:


ThisBuild / scalaVersion     := "2.13.0"
ThisBuild / version          := "0.1.0-SNAPSHOT"
ThisBuild / organization     := "com.example"
ThisBuild / organizationName := "example"

lazy val root = (project in file("."))
  .settings(
    name := "specs2-scalacheck",
    libraryDependencies ++= Seq(
      "org.specs2" %% "specs2-core" % "4.6.0",
      "org.specs2" %% "specs2-matcher-extra" % "4.6.0",
      "org.specs2" %% "specs2-scalacheck" % "4.6.0"
    ).map(_ % "test")
  )

When you run the test from the sbt console:

sbt:specs2-scalacheck> testOnly example.HelloSpec

You get the following output:

[info] HelloSpec
[error]     x a simple property
[error]  Falsified after 2 passed tests.
[error]  > ARG_0: "\u0000"
[error]  > ARG_0_ORIGINAL: "猹"
[error]  The seed is X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=
[error]  
[error]  > '' != '' (HelloSpec.scala:11)
[info] Total for specification HelloSpec

To reproduce that specific run (i.e with the same seed), you can take the seed from the output and pass it using the command line scalacheck.seed:

sbt:specs2-scalacheck>testOnly example.HelloSpec -- scalacheck.seed X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=

And this produces the same output as before.

You can also set the seed programmatically using setSeed:

def ex1 = prop((s: String) => s.reverse.reverse must_== "").setSeed("X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=")

Yet another way to provide the Seed is pass an implicit Parameters where the seed is set:

package example

import org.specs2.mutable.Specification
import org.specs2.ScalaCheck
import org.scalacheck.rng.Seed
import org.specs2.scalacheck.Parameters

class HelloSpec extends Specification  with ScalaCheck {

  s2"""
    a simple property       $ex1
  """

  implicit val params = Parameters(minTestsOk = 1000, seed = Seed.fromBase64("X5CS2sVlnffezQs-bN84NFokhAfmWS4kAg8_gJ6VFIP=").toOption)

  def ex1 = prop((s: String) => s.reverse.reverse must_== "")
}

Here is the documentation about all those various ways. This blog also talks about this.

Reproducing a ScalaCheck test run

3 Answers