Testing not equal in Quickcheck?

Question

I'm new to QuickCheck and can't quite wrap my head around how to use it.

Let's say I accidentally implemented a data-type with a Set (instead of a List):

data Profile = Profile (Set Strategy)
--for completeness:
data Strategy = Strategy Int

and then ran into this bug later, where two objects are equal even if they shouldn't:

Profile (Set.fromList [1,2,3]) == Profile (Set.fromList [2,1,3])
-- D'OH! Order doesn't matter in sets!

How can I write a QuickCheck test case to test for this case? In pseudo-code this would look something like this:

assertNotEqual(Profile (Set.fromList [1,2,3]), Profile (Set.fromList [2,1,3]))
assertEqual(Profile (Set.empty), Profile (Set.empty ))

I've tried looking at the examples on the project's github, but it seems they don't cover such trivial cases.

Any hints welcome!

Answer 1

You can do it using existential quantification, supported by SmallCheck:

> depthCheck 5 $ exists $ \xs -> (xs :: [Integer]) /= sort xs
    Depth 5:
      Completed 1 test(s) without failure.
> depthCheck 5 $ exists $ \xs -> fromList (xs :: [Integer]) /= fromList (sort xs)
    Depth 5:
      Failed test no. 1. Test values follow.
      non-existence

Another option, using universal quantification (also works in QuickCheck):

> smallCheck 5 $ \xs ys -> xs /= ys ==> fromList (xs :: [Integer]) /= fromList ys
Depth 0:
  Completed 1 test(s) without failure.
  But 1 did not meet ==> condition.
Depth 1:
  Completed 4 test(s) without failure.
  But 2 did not meet ==> condition.
Depth 2:
  Failed test no. 26. Test values follow.
  [0]
  [0,0]

Answer 2

How can I write a QuickCheck test case to test for this case?

You shouldn't! QuickCheck is a tool for property based testing. In property based testing, you give a property of your data structure (or whatever) and the testing tool will automatically generate test cases to see if that property holds for the generated test cases. So, let's see how you can give a property instead of giving concrete test cases like [1,2,3] and why properties are advantageous!

So. I started off with

import Test.QuickCheck
import qualified Data.Set as Set 
import Data.Set (Set)

data Profile = Profile (Set Int)
  deriving (Eq, Show)

mkProfile :: [Int] -> Profile
mkProfile = Profile . Set.fromList

-- | We will test if the order of the arguments matter.
test_mkProfile :: [Int] -> Bool
test_mkProfile xs = (mkProfile xs `comp` mkProfile (reverse xs))
  where comp | length xs <= 1 = (==)
             | otherwise      = (/=)

This is how I reasoned for my property: Well, for empty and singleton list case, then reverse is just the identity, so we expect mkProfile xs be the same as mkProfile (reverse xs). Right? I mean mkProfile gets exactly the same argument. In the case that length xs >= 2 then reverse xs clearly isn't xs. Like reverse [1, 2] /= [2, 1]. And we know that a Profile do care about the order.

Now lets try this out in ghci

*Main> quickCheck test_mkProfile 
*** Failed! Falsifiable (after 3 tests and 1 shrink):     
[0,0]

Note now that there actually are two mistakes in our code. One, First, Profile should be using a list and not a set. Second, our property is wrong! Because even if length xs >= 2, xs == reverse (xs) can be true. Let's try to fix the first error and see how quickcheck will still point out the second flaw.

data Profile2 = Profile2 [Int]
  deriving (Eq, Show)

mkProfile2 :: [Int] -> Profile2
mkProfile2 = Profile2 

-- | We will test if the order of the arguments matter.
test_mkProfile2 :: [Int] -> Bool
test_mkProfile2 xs = (mkProfile2 xs `comp` mkProfile2 (reverse xs))
  where comp | length xs <= 1 = (==)
             | otherwise      = (/=)

Remember, our code is correct now but our property flawed!

*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
+++ OK, passed 100 tests.
*Main> quickCheck test_mkProfile2
*** Failed! Falsifiable (after 8 tests):                   
[-8,-8]

Yes. You still need to think! Or you might get the false impression that everything is ok just because your code literally passed 700 test cases! Ok now lets fix our property too!

test_mkProfile2_again :: [Int] -> Bool
test_mkProfile2_again xs = (mkProfile2 xs `comp` mkProfile2 ys)
  where ys   = reverse xs
        comp | xs == ys  = (==)
             | otherwise = (/=)

Now lets see that it works multiple times!

*Main> import Control.Monad
*Main Control.Monad> forever $ quickCheck test_mkProfile2_again
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
+++ OK, passed 100 tests.
... (a lot of times)

Hooray. We've now not only squashed the bug in our Profile implementation, but also got a much better understanding of our code and the properties it adheres too!

Answer 3

As I commented, my main issue answering this question is the lack of structure around your Profile type. If you define Profile, a set of operations, and invariants then it becomes easy to make quickcheck tests.

For example, lets say you have a Profile, a way to build profiles, and one way to modify profiles. The properties will all be uniqueness

 module Profile (Profile, mkProfile, addItem) where
 import Data.Set

 newtype Profile = Profile { unProfile :: Set Int }
   deriving (Eq, Ord, Show)

 mkProfile :: [Int] -> Profile
 mkProfile = Profile . fromList

 addItem :: Int -> Profile -> Profile
 addItem x = Profile . insert x . unProfile

you could test such an ADT with quickcheck by stating properties before and after each operation:

 import Test.QuickCheck
 import Profile as P

 prop_unique_list_unique_profile :: [Int] -> [Int] -> Bool
 prop_unique_list_unique_profile xs ys =
    xs /= ys ==> mkProfile xs /= mkProfile ys

 prop_addItem_nonequal :: Int -> [Int] -> Bool
 prop_addItem_nonequal x xs = P.addItem x xs /= xs