0
votes

I'm reasonably new to Clojure, and am trying to parallelize some calls to a function.

Let's say I have a map as follows:

{:a 1 :b 2 :c 3}

and I want to use the keys and values as arguments of a function called my-function, with the function called for each key/value pair. I could use doseq as below:

(doseq [entry my-map] (my-function (key entry) (val entry)))

I then decide I want the calls to my-function to be in parallel (in my actual situation my-function will be making rest calls which are independent from each other, and so I want to do them in parallel). I came up with the following:

(apply pcalls (map #(partial my-function (key %) (val %)) my-map))

This seems to work as I expect (please correct me if I'm wrong or if there's a better way). The trouble I'm having now is getting my Midje tests to pass. They pass correctly if I use the doseq version above. In my tests, I am using (provided .... ) to ensure that the correct calls are made, and I wish to use this to check the calls to my-function. Is it possible to do this? I'm finding it hard to find much information on pcalls.

Thanks

--------- Edit ---------

I have contrived an example that demonstrates the failue.

The implementation can be found here, and the tests here. I have found that a single test runs fine, but multiple tests cause the following:

FAIL at (pcalls_unit.clj:29) These calls were not made the right number of times: (my-func 2 2) [expected at least once, actually never called]

FAIL "Test some pcalls 3" at (pcalls_unit.clj:26) Expected: nil Actual: java.util.concurrent.ExecutionException: java.lang.Error: You seem to have created a prerequisite for pcalls-test.pcalls/my-func that interferes with that function's use in Midje's own code. To fix, define a function of your own that uses my-func, then describe that function in a provided clause.

1
What does your test look like? The problem might be that Midje is expecting the results to appear in a certain order. - bdesham
Hi. My tests look something like: (fact "test something" (method-that-calls-pcalls-stuff-and-more arg1 arg2 arg3) => {:status 200} (provided (my-function key1 val1) => some-result1 (my-function key2 val2) => some-result2)) - dmcgillen
Can you link to a gist of a failing test? Please add the failure output. - Brian Marick
Midje is supposed to require all lazy sequences to be fully evaluated (as with doall), but perhaps there's a bug. You could try wrapping the computation of the actual value with doall: (fact (doall (computation)) => expected (provided ...) - Brian Marick
Thanks for your reply Brian. I've wrapped the computation in doall but I get the same failures - (my-func arg1 arg2) [expected :times 1, actually called zero times] It is not always the same ones that appear, sometimes only 2 fail, sometimes 3, sometimes 4 etc. If I use doall and pmap as in Arthur's answer below, I get an error about my function interfering with that function's use in Midje's own code - dmcgillen

1 Answers

1
votes

To answer only the first part of the question pmap can also be used to write this:

(doall (pmap my-function (keys my-map) (vals my-map)))