I'm building an API with Servant, and it seems to work pretty well. And in line with best practices I am writing some tests for it, following the official guide here. But I'm struggling with the part using Servant-Quickcheck.
I'm trying to use it to test that my API doesn't give any 500 errors, like this:
quickCheckSpec :: Spec
quickCheckSpec = describe "QuickCheck global tests for public API -" $ do
it "API never gives 500 error" $
withServantServer publicAPI (return server) $ \burl ->
serverSatisfies publicAPI burl args (not500 <%> mempty)
but unfortunately the test is failing. This is a surprise to me, given that I've not encountered such an error when testing manually - but hey, this is why we have automated tests, right? Especially with tools like QuickCheck which, as far as I understand, is supposed to zero in on "edge cases" that you might not think to test manually. (This is actually my first time using QuickCheck in any form.)
The problem is though that a test failure is useless unless you know what it failed on. And this is what Servant-QuickCheck does not seem to want to tell me. The only output I get regarding the test failure is this:
API never gives 500 error FAILED [1]
Failures:
src\\Servant\\QuickCheck\\Internal\\QuickCheck.hs:146:11:
1) QuickCheck global tests for public API - API never gives 500 error
Nothing
To rerun use: --match "/QuickCheck global tests for public API -/API never gives 500 error/"
Randomized with seed 1712537046
Finished in 10.8513 seconds
4 examples, 1 failure
(There are some unit tests run before, which all pass - the 10 seconds isn't all on the above test!)
This is rather bemusing, because as you can see there's nothing which indicates how the failure occurred. Since it's testing an API and, as far as I understand it, choosing valid routes at random to test with, it ought to tell me at which route it encountered the 500 error. Yet, as you can see, there is precisely nothing there.
(I'm developing the project with Stack btw, and a little bit below the above it says Logs printed to console, so I don't believe I'm missing any log files buried somewhere which could enlighten me. I haven't been able to find any in my .stack-work folder either.)
I've even changed my args so that it has the chatty field set to True, but I still only get the above.
The only clue I get, which isn't related to QuickCheck, is that my API uses Persistent and logs the database queries to the terminal, so I can see the queries that have been run - which I can in turn relate back to the route. However I don't particularly trust this, because the route that runs the query shown is definitely working in manual tests. (And it's not always the same query when it fails.) There are also a couple of simple routes that don't query the database, so the failure isn't necessarily related to the last database query printed (although needless to say, I've manually tested those routes too and they're fine).
I've even wondered if the problem might not be that QuickCheck hits my localhost server so many times in a row that it simply can't cope and errors out, but in that case I would expect this to be a common problem, remarked upon in the tutorial. (And the database logging output suggests that it in fact always fails on the very first "hit".)
Anyway, that's just speculation, and I realise that it's up to me to figure out why my API might be failing the tests, given that I didn't think it relevant to share any details of my actual API.
What I'm really asking is: why am I not told which routes were tested, and which ones passed/failed, so that I at least know exactly which route(s) to investigate? And is there any way to make Servant-Quickcheck show me that information?
Any answer to that would be much appreciated.
FURTHER INFORMATION: I've tried with @MarkusBarthlen's suggestion below, but it didn't give me any more information. However, it did point up that the Nothing which you can see in my console output is probably somehow key to this, because in Markus's example he gets a Just value holding the request information. Looking at line 146 of the source file referenced in the failure message, it's clearly the x there that apparently is a Maybe value. But I can't figure out what type it actually is, and what the significance is of it being Nothing in my case. Unfortunately the code is beyond my own Haskell experience/ability to make much sense of - I can see the x is read out of an MVar, which is populated somehow with the result of the test, but as I said I'm not able to make sense of the details. But I really need to know, not only why my test is failing, but why this particular x value is Nothing for me, and what that means in terms of the test. I would really appreciate an answer that explains this in a way I can understand.
