How to specify the exact execution order of asynchronous calls in Scala unit tests?

Question

I have written many different unit tests for futures in Scala. All asynchronous calls use an execution context. To make sure that the asynchronous calls are always executed in the same order, I need to delay some tasks which is rather difficult and slows the tests down. The executor might still (depending on its implementation) complete some tasks before others.

What is the best way to test concurrent code with a specific execution order? For example, I have the following test case:

"firstSucc" should "complete the final future with the first one" in {
    val executor = getExecutor
    val util = getUtil
    val f0 = util.async(executor, () => 10)
    f0.sync
    val f1 = util.async(executor, () => { delay(); 11 })
    val f = f0.firstSucc(f1)
    f.get should be(10)
  }

where delay is def delay() = Thread.sleep(4000) and sync synchronizes the future (calls Await.ready(future, Duration.Inf)). That's how I want to make sure that f0 is already completed and f1 completes AFTER f0. It is not enough that f0 is completed since firstSucc could be shuffling the futures. Therefore, f1 should be delayed until after the check of f.get.

Another idea is to create futures from promises and complete them at a certain point in time:

  "firstSucc" should "complete the final future with the first one" in {
    val executor = getExecutor
    val util = getUtil
    val f0 = util.async(executor, () => 10)
    val p = getPromise
    val f1 = p.future
    val f = f0.firstSucc(f1)
    f.get should be(10)
    p.trySuccess(11)
  }

Is there any easier/better approach to define the execution order? Maybe another execution service where one can configure the order of submitted tasks? For this specific case it might be enough to delay the second future until after the result has been checked but in some cases ALL futures have to be completed but in a certain order.

The complete code can be found here: https://github.com/tdauth/scala-futures-promises

The test case is part of this class: https://github.com/tdauth/scala-futures-promises/blob/master/src/test/scala/tdauth/futuresandpromises/AbstractFutureTest.scala

This question might be related since Scala can use Java Executor Services: Controlling Task execution order with ExecutorService

Why do you think that testing concurrent code with delays and specific execution order is a good idea in the first place? The bugs in concurrently running code usually occur exactly because of the unexpected order of events and failures to synchronize the memory between the threads. Stuff like Thread.sleep(4000) would make the tests mostly useless, because the JVM might decide to synchronize the memory after a while, even without any explicit synchronization in the code, which would mean that even broken code can easily pass your tests. — Andrey Tyukin
@AndreyTyukin "the bugs occur because of the unexpected order" - yes, but it may still be useful to write a test, that ensures there are no bugs in case events happen in a specific order — Dima
@Dima Then one should separate the logic that one wants to test from the concurrent evaluation strategy, implement a separate sequential deterministic interpreter, and test the logic with that. Only then should one switch to a concurrent implementation, and test that it also works as expected (regardless of execution order, and without artificial delays). Implementing everything with concurrency and futures first, and then trying to impose delays and a specific order looks more like an attempt to make a river flow against the gravity. — Andrey Tyukin
@AndreyTyukin yes, that's all right. The question is how to test that the original implementation didn't break after you switch it to concurrent. — Dima
I don't think that it is a good idea but I need to test cases where there is a specific order of events. Otherwise, I cannot expect a certain result. Maybe mocking the futures would help, I don't know. — Baradé

Dima Dima · Accepted Answer · 2018-09-23T13:23:08

For most simple cases, I'd say a single threaded executor should be enough - if you start your futures one-by-one, they'll be executed serially, and complete in the same order.

But it looks like your problem is actually more complex than what you are describing: you are not only looking for a way to ensure one future completes later than the other, but in general, to make a sequence of arbitrary events happen in a particular order. Fr example, the snippet in your question, verifies that the second future starts after the first one completes (I have not idea what the delay is for in that case btw).

You can use eventually to wait for a particular event to occur before continuing:

   val f = Future(doSomething) 
   eventually { 
      someFlag shouldBe true
   }
   val f1 = Future(doSomethingElse) 
   eventually { 
      f.isCompleted shouldBe true
   }
   someFlag = false   
   eventually { 
      someFlag shouldBe true
   }

   f1.futureValue shoudlBe false

How to specify the exact execution order of asynchronous calls in Scala unit tests?

1 Answers