How to use FsCheck to generate random numbers as input for property-based testing

Question

I thought it's time to try out FsCheck but it proves tougher than I thought. There's a lot of documentation on Arb, generators and so on, but there doesn't seem to be any guidance in how to apply that knowledge. Or I'm just not getting it.

What may make it harder to grasp is that the relation between tests, properties, generators, arbitraries, shrinking and, in my case, randomness (some tests automatically generate random data, others don't) is not clear to me. I don't have a Haskell background so that doesn't help much either.

Now for the question: how do I generate random integers?

My test scenario can be explained on the properties of multiplication, let's say distributivity:

static member  ``Multiplication is distributive`` (x: int64) y z =
    let res1 = x * (y + z)
    let res2 = x * y + x * z

    res1 = res2

// run it:
[<Test>]
static member FsCheckAsUnitTest() =
    Check.One({ Config.VerboseThrowOnFailure with MaxTest = 1000 }, ``Multiplication is distributive``)

When I run this with Check.Verbose or the NUnit integration, I get test sequences like:

0:
(-1L, -1L, -1L)
1:
(-1L, -1L, 0L)
2:
(-1L, -1L, -1L)
3:
(-1L, -1L, -1L)
4:
(-1L, 0L, -1L)
5:
(1L, 0L, 2L)
6:
(-2L, 0L, -1L)
7:
(-2L, -1L, -1L)
8:
(1L, 1L, -2L)
9:
(-2L, 2L, -2L)

After 1000 tests it hasn't gotten over 100L. Somehow I imagined this would "automatically" choose random numbers evenly distributed over the whole range of int64, at least that's how I interpreted the documentation.

Since it doesn't, I started experimenting and came up with silly solutions like the following to get higher numbers:

type Generators = 
    static member arbMyRecord =
        Arb.generate<int64>
        |> Gen.where ((<) 1000L)
        |> Gen.three
        |> Arb.fromGen

But this becomes incredibly slow and is clearly not the right approach. I'm sure there must be a simple solution that I'm missing. I tried with Gen.choose(Int64.MinValue, Int64.MaxValue), but this only supports ints, not longs (but even with just ints I couldn't get it working).

In the end I need a solution that works for all the primitive numeric data types, that includes their maxes and mins, their zeroes and ones, and some random selection from whatever is within.

@s952163, yes, that's why I tried with MaxTest = 1000, see code above. But that doesn't help. Perhaps you mean the StartTest and EndTest values, but setting these to Int32.MinValue/MaxValue has the effect that all permutations use Int32.MinValue as a constant value. — Abel

Mark Seemann Mark Seemann · Accepted Answer · 2016-12-02T06:53:49

As explained in this other FsCheck question, the default configurations for most of the Check functions has EndSize = 100. You can increase that number, but you can also, as you suggest, use Gen.choose.

Even so, though, the int generator is intentionally well-behaved. It doesn't, for example, include Int32.MinValue and Int32.MaxValue, since this could lead to overflows.

FsCheck does, however, also come with generators that give you uniform distributions over their entire range: Arb.Default.DoNotSizeInt16, Arb.Default.DoNotSizeUInt64, and so on.

For floating point values, there's Arb.Default.Float32, which , according to its documentation, generates "arbitrary floats, NaN, NegativeInfinity, PositiveInfinity, Maxvalue, MinValue, Epsilon included fairly frequently".

There's no uniform API for 'just' any number, since F# doesn't have typeclasses (this is something you'd be able to express in Haskell).

Also, I'm not sure your typical unit testing framework will be able run generic tests, but at least with xUnit.net, you can use this trick to run generically typed tests.

Specifically, though, you can write the above test like this, using FsCheck.Xunit:

open FsCheck
open FsCheck.Xunit

[<Property>]
let ``Multiplication is distributive`` () =
    Arb.generate<DoNotSize<int64>>
    |> Gen.map (fun (DoNotSize x) -> x)
    |> Gen.three
    |> Arb.fromGen
    |> Prop.forAll <| fun (x, y, z) ->

        let res1 = x * (y + z)
        let res2 = x * y + x * z

        res1 = res2

This could hypothetically fail from overflowing, but after having run some 1,000,000 cases, I haven't seen it fail yet.

The generator, however, does indeed look like it's picking values from the full range of 64-bit integers:

> Arb.generate<DoNotSize<int64>> |> Gen.sample 1 10;;
val it : DoNotSize<int64> list =
  [DoNotSize -28197L; DoNotSize -123346460471168L; DoNotSize -28719L;
   DoNotSize -125588489564554L; DoNotSize -29241L;
   DoNotSize 7736726437182770284L; DoNotSize -2382327248148602956L;
   DoNotSize -554678787L; DoNotSize -1317194353L; DoNotSize -29668L]

Notice that even though I bind the size argument of Gen.sample to 1, it picks 'arbitrarily' large positive and negative values.

How to use FsCheck to generate random numbers as input for property-based testing

1 Answers