How do I expect failure in a unit test?

10
votes

We're writing unit tests for our code in Clojure using clojure.test.

Some of our tests ignore the API and purposely break the code, in order to serve as documentation for latent deficiencies in the code.

However, we want to distinguish between failures of these tests, and failures of normal tests.

We haven't seen any suggestions in the clojure.test documentation -- only (is (thrown? ...)), which of course doesn't do what we need.

Any suggestions? Basically, we're looking for something like (is (not <condition>)), except that the test framework should record an expected failure -- something like this.

4
unit-testingclojure
When you say "the test framework should record an expected failure", do you mean the test framework should recognize that the test passed because it threw an exception that you expected, or do you mean it should recognize three kinds out outcomes: passes, failures, and expected failures, the latter of which are explicitly recorded? – user100464
@user100464: something like this: docs.python.org/library/…. So we'd like to explicitly record expected failures. – Matt Fenwick

4 Answers

3
votes

I have made tests throw an exception when they 'fail' like this, and then used thrown? to test if the exception arrives as expected. There very well may exist a more elegant way, but this gets the job done.

3
votes

As @andy said you can rebind report function.

(defmacro should-fail [body]
  `(let [report-type# (atom nil)]
     (binding [clojure.test/report #(reset! report-type# (:type %))]
       ~body)
     (testing "should fail"
       (is (= @report-type# :fail )))))

And use this macro like this:

(should-fail (is (= 1 2)))

That will be successful passing test.

0
votes

Rebind the report function as documented in clojure.test. From there you can change how the 'failure' is handled and reported on.

0
votes

A little late to the party, I know, but if you're using Leiningen you could use test selectors to exclude expected failures from the "normal" build. In your project.clj:

:test-selectors {:default #(not :expected-failure %)
                 :expected-failure :expected-failure}

Then write your test as:

(deftest ^:expected-failure test-with-expected-failure ...)

Calling lein test will run only those tests without the :expected-failure metadata. Calling lein test :expected-failure will run your expected failure tests. Maybe not quite what you were hoping for, but IMHO better than having to use thrown? all over the place. At least this way expected failures are explicitly documented in the code and won't interfere with the build.