haskell - Average floating point error using QuickCheck

Question

I am using QuickCheck-2.5.1.1 to do QA. I am testing two pure functions gold :: a -> Float and f :: a -> Float, where a instances Arbitrary.

Here gold is a reference calculation and f is a variation I am optimizing.

To date, most of my tests using quickcheck have been using tests like \a -> abs (gold a - f a) < 0.0001.

However, I would like to gather statistics along with checking the threshold, since knowing the average error and standard deviation are useful in guiding my design.

Is there any way to use QuickCheck to gather statistics like this?

Concrete example

To give a concrete example of the sort of thing I'm looking for, suppose I have the following two functions for approximating square roots:

-- Heron's method
heron :: Float -> Float
heron x = heron' 5 1
    where
      heron' n est
          | n > 0 = heron' (n-1) $ (est + (x/est)) / 2
          | otherwise = est

-- Fifth order Maclaurin series expansion
maclaurin :: Float -> Float
maclaurin x = 1 + (1/2) * (x - 1) - (1/8)*(x - 1)^2
                + (1/16)*(x - 1)^3 - (5/128)*(x - 1)^4
                + (7/256)*(x - 1)^5

A test for this might be:

test = quickCheck
       $ forAll (choose (1,2))
       $ \x -> abs (heron x - maclaurin x) < 0.02

So what I'd like to know, as a side-effect of the test, is the statistics on abs (heron x - maclaurin x) (such as the mean and standard deviation).

First, I expect that you need to control the distribution of the 'a' input? How do you do this now? — Chris Kuklewicz
@ChrisKuklewicz : I generate a random, treelike structure. It's pretty complicated and it's not really the problem I'm worried about. I will modify my original question to give a concrete example of the sort of thing I'm looking for. — Matt W-D
Could you explain why the standard quickcheck mechanisms (i.e. label, etc.) do not work for you? — Ingo
As @Ingo says, you can bin-up the results, convert to strings, and use 'label'. — Chris Kuklewicz

Matt W-D Matt W-D · Accepted Answer · 2013-03-14T16:33:50

Thanks to the comments from Chris Kuklewicz and Ingo, I came up with the following that collects the statistics I want in my example:

resultToWeightList :: Result -> [(Double,Int)]
resultToWeightList r = [ (read s, n) | (s,n) <- labels r]

weightListMuSigma :: [(Double,Int)] -> (Double,Double)
weightListMuSigma wlst = (mu,sigma)  
  where 
    (weightSum,weightSqrSum,entryCount) = foldl addEntry (0,0,0) wlst
    addEntry (s,s2,c) (v,w) = (s + (v * w'), s2 + (v**2 * w'), c + w)
      where w' = fromIntegral w
    entryCount' = fromIntegral entryCount
    mu = weightSum / entryCount'
    var = weightSqrSum / entryCount' - mu**2
    sigma = sqrt var

quietCheckResult :: Testable prop => prop -> IO Result
quietCheckResult p = quickCheckWithResult args p
  where args = stdArgs { chatty = False }

test :: IO ()
test = do { r <- quietCheckResult $ forAll (choose (1,2)) test'
          ; let wlst = resultToWeightList r
          ; let (mu,sigma) = weightListMuSigma wlst 
          ; putStrLn $ "Average: " ++ show mu
          ; putStrLn $ "Standard Deviation: " ++ show sigma
          }
   where
     test' x = collect err (err < 0.1)
       where err = abs $ heron x - maclaurin x

haskell - Average floating point error using QuickCheck

Concrete example

1 Answers