Generate events/commands using a property based testing tool?

1
votes

As I understand it, most property testing tools operate at the level of functions. Given a set of arguments, such tools will generate random input and test output against some invariant.

I have read that ScalaCheck is now starting to include generation of events to test a statefull system. However, I can't find great deal of information on it. Is this becoming popular in rest of the *check ecosystem as well (fscheck, quickcheck, other variations)?

1
quickcheckscalacheckfscheckproperty-based-testing
I don't understand the question you are asking. Property based testing is based on assumptions that the code being tested does not have mutation or side effects. If your code has mutation or side effects that are central to the functionality, you need a different approach. This is what the stateful testing framework in scala check is for. As for your question about popularity in other frameworks, I think that only has subjective answers and therefore is not a great question for SO – Asa
By "Is this becoming popular" I meant I just want to know if similar frameworks in other languages have a counterpart to scalacheck's stateful testing. Not a ranking of which is most popular. Since event based architectures are becoming more popular, I'm curious to know if automated-randomized testing ecosystem is coming up with corresponding solutions. – Shahbaz
If you take a Reactive Functional Programming approach, events are simply data structures. Define them as record types, and e.g. FsCheck will have no problem creating them. – Mark Seemann
@MarkSeemann the way I see it, an 'event' base system can be simply modeled as a stateful class with a number of methods. I understand that fscheck/etc. can call a single function number of times with random inputs, but is it designed to call the whole public interface of a class with random inputs, in random order and then check the state of the class? – Shahbaz
FsCheck and QuickCheck don't call any methods. They generate data, and you write the code that uses that data. – Mark Seemann

1 Answers

1
votes

What you call "generation of events" to my knowledge originates in "Testing Monadic Code with QuickCheck", by Koen Claessen and John Hughes. The example they give is testing a queue. The approach that is used is always similar - as comments say, since "basic" quickcheck (I'll use lowercase quickcheck to describe the family of QuickCheck ports on various platforms) assumes it generates immutable data, at first sight it's not easy to use quickcheck to test a side-effecting, stateful system.

Until you realize that a stateful system gets to a certain state by executing a sequence of state transitions (these are variously called commands, actions, events etc). And this sequence can be represented perfectly as an immutable list of immutable transitions! Typically then each transition is executed on the real system under test, and a model of its state. Then after each transition the model state is compared with the real state.

To see how this plays out in Quvik QuickCheck (for Erlang) for example you can read "Testing Telecoms Software with Quviq QuickCheck" by Thomas Arts, John Hughes, Joakim Johansson and Ulf Wiger.

I do believe most quickchecks, including QuickCheck itself, have a layer on top of the basic quickcheck functionality that allows you to generate a sequence of state transitions, typically using a state machine like approach with pre-and postconditions etc.

I don't think this is particularly new, but probably a bit under-emphasized.

For example, FsCheck has had model based testing for years (dislosure: I am FsCheck's main contirbutor). I think the same is true for ScalaCheck. Quvik QuickCheck's is likely the most advanced implementation (certainly with the most advanced applications).