Force specific input with JSVerify

Question

I'm starting to use JSVerify for property based testing.

Given a function (just an example) that accepts a string as parameter, I use JSVerify to pass in a large number of arbitrary strings and see if the function behaves as intended for all of them. It turns out that there are some strings that make the test fail: so far I've found that if the string contains \0000, \0001 or \n, the test fails.

I want to correct my code so that this cases are handled accordingly, but in order to prevent regression, I want to make sure that each one of this cases is included in every test run. I also want to avoid hardcoding the number generator's seed (rngState) since that would prevent the discovery of additional corner cases in the future.

To clarify: suppose that I'm testing my function foo():

jsc.assert(jsc.forall("string", (str) => {
  const result = foo(str)
  return (/* some expression that evaluates whether the result is correct */)
}))

This test is feeding 100 random strings into foo() every time a run the test suite. Initially it's passing. After some runs it suddenly fails because, precisely this time, the random string generator has generated a string containing the character \0000, which my function doesn't handle as expected. From now on, I would like that my test uses this string as input every time, plus the usual 100 random inputs.

Is there a built-in way to do this using JSVerify? Or should I just treat these inputs as separate test cases?

benathon benathon · Accepted Answer · 2019-01-22T10:20:09

This is how i'm addressing the problem you mentioned. Instead of testing specific values, my code can specify as many "bad" seeds as needed, in addition to running a random one with each test invocation.

Notes: 1) I am using Mocha 2) I am using async code, so I return a promise 3) the JSON.stringify allows me to easily see the seed given the above

You can still use this method without 1,2,3 above with some refactoring:

const _ = require('lodash');
var jsc = require('jsverify');

// mocha's describe
describe("StackOverflow", function () {
  it('example1', function(done) {

    let t = jsc.forall('array nat', function (in1) {
      let asComplex = foo(in1);
      let backAgain = bar(asComplex);

      return new Promise(function(resolve, reject) {
        setTimeout(() => {
          resolve(_.isEqual(backAgain,in1));
        },500); // setTimeout
      }); // promise
    }); // forall

    let props = {size:0xffffffff, tests: 1000};
    jsc.check(t, props).then( r => r === true ? done() : done(new Error(JSON.stringify(r))));
    props.tests = 3;
    jsc.check(t, props).then( r => r === true ? done() : done(new Error(JSON.stringify(r))));
    props.rngState = "8e1da6702e395bc84f";
    jsc.check(t, props).then( r => r === true ? done() : done(new Error(JSON.stringify(r))));
    // add more seeds here
  }); // it
}); // describe

Force specific input with JSVerify

1 Answers