When are the Clojure core.async go-blocks useful?

Question

Since the thread pool running go-blocks is fairly small, it seems that some operations inside go-blocks aren't recommended. For example doing blocking I/O is bad, and doing compute-heavy operations is bad. A few such operations running in go-blocks will hog the entire thread pool, and no other go-blocks would be able to run at all.

What most common scenarios do Clojure programmers use go-blocks for? Is it mainly useful for working with already asynchronous libraries? Transform callback-using libraries/code into a more palatable channel variant? I'll be very interested to see practical examples where go-blocks with parking channel ops are preferable to threads.

Answer 1

Each thread requires a minimum amount of memory (default 1MB on 64-bit linux) for its stack, etc. While several hundred threads may not be a problem on an average computer, several thousand or more may consume a lot of memory (i.e. 1000 threads => 1GB memory). And, these resources are consumed even when the thread is blocked (not running).

Suppose you have a task that is only running 1-10% of the time. Then using regular threads consumes 10x-100x more resources than required. Since go blocks share threads via "parking", you can reduce the thread resources required by 10x-100x. This is the benefit.

A second benefit is that core.async can be used in ClojureScript and run in the browser. Since JavaScript (which CLJS compiles into) has only a single thread, core.async allows one to have the illusion of multithreading in the browser even though regular Java threads are not available.

Answer 2

In short, use go blocks when you are using core.async channels in an asynchronous clojure application.

core.aysnc gives you built-in backpressure, that is, a mechanism which says "no, I will not create new workers because we're busy already" (i.e., creating 100000 go blocks will not wreck your system, but don't try this at home with threads, or a thread pool with an unbounded queue). This is a big win, and eliminates the need to synchronize the threads to do this yourself. Problems which require backpressure (such as producer-consumer problems) often fit this model well, and core.async is a great choice.

However, the go macro is not part of the clojure core library, and it is not intended to be used as a general purpose tool to achieve concurrency. Its purpose is to provide an abstraction on top of asynchronous code to make it flow in a more synchronous way. Consider your use case and if the model fits, go for it!