How to generate meaningful examples to test n-ary properties with property-based testing?

Question

In a property-based test setting like Haskell's quickcheck for custom data structures, how do you generate test data for n-ary properties of relations, e.g., transitivity or symmetry? The implementation language does not matter, I think.

Here is a naive C++ example using rapidcheck (just because I have this tool at hand, right now):

  rc::check("Double equality is symmetric.", [](double a, double b) {
     RC_ASSERT(!(a == b) || (b == a)); // a == b ==> b == a
  });

In such a naive case it is quite unlikely that the tool will generate many examples where the premise (a == b) actually holds, so you end up wasting a lot of effort on meaningless tests. It gets even worse for 3-ary relations like transitivity.

Is there a general technique to tackle these issues? Do I need to generate equal pairs (for some constructive definition of "equals")? What about stuff like orderings?

johanneslink johanneslink · Accepted Answer · 2020-01-11T06:11:50

What I do to raise the probability of value clashes is to restrict value generation to a smaller range and sometimes combine that with a more general generator.

Consider the following generator adapted from https://johanneslink.net/how-to-specify-it/#46-a-note-on-generation:

@Provide
Arbitrary<Integer> keys() {
    return Arbitraries.oneOf(
            Arbitraries.integers().between(-25, 25),
            Arbitraries.integers()
    );
}

Generation will first choose with equal probability between any integer and an integer between -25 and +25. Thus about every 100th value will be a duplicate.

In more difficult cases I might even have a generator that picks from a small set of predefined values.

UPDATE: The latest version of jqwik allows to explicitly generate duplicates with a given probability: https://jqwik.net/docs/snapshot/user-guide.html#inject-duplicate-values I don't know, though, if QuickCheck or any other PBT library has a similar feature.

How to generate meaningful examples to test n-ary properties with property-based testing?

1 Answers