Basic goroutine and channel pattern: multiple goroutines for one channel

Question

i'm new to Go and am wondering about some pretty basic problem that i can't figure out clearly.

Just for the exercize (an abstraction of a real need), i need to:

• initialize a slice of string with a number of elements fixed by the constant ITERATIONS
• iterate over this slice and run a goroutine for each element
• each goroutine will take a certain amount of time to process the element (a random second duration)
• once the job is finished, i want the goroutine to push the result to a channel
• then i need to catch all the results from this channel (in a function called from the main goroutine), append them to the final slice and then, once its finished
• print the length of the final slice and add some basic time tracking

Below is a code that works.

No surprise, the total amount of program time is always more or less equals to const MAX_SEC_SLEEP's value, as all the processing goroutine do they work in parallel.

But what about:

1. the receive part:

do i really need to wrap my select statement in a for loop, iterating the exact amount of ITERATIONS , to have the exact same number of receivers than the number of goroutines that will end to the channel ? Is it the only way to avoid deadlock here ? And what if for some reason, one of the goroutine fails ?

I can't find a way to have a simple for (ever) loop wrapping the select, with two cases (the one receiving from the results channel and another one like case <-done that wouldreturn from the function). Would it be a better pattern ?

Or would it be better to iterate over the channel and detect if it is closes from somewhere ?

2. the send part

Should i close somewhere the channel, after all the iterations ? but i would surely close it before at least one of the gouroutine finishes, ending in a panic error (trying to send to a closed channel)

If i were to plug a done <- true pattern, would it be here ?

3. Wait groups

i did not really try waitgroups, ad i need a way to catch all 'return' values from the goroutines and append them to the final slice; and i did not find a proper way to return from a goroutine except by using channels.

4. Misc

Should i pass channels in func arguments or let them global to the program as it is ?

5. The (bad) code

package main

import (
    "fmt"
    "log"
    "math/rand"
    "time"
)

const ITERATIONS = 200

var (
    results   chan string
    initial   []string
    formatted []string
)

func main() {
    defer timeTrack(time.Now(), "program")

    format()  //run format goroutines
    receive() //receive formatted strings

    log.Printf("final slice contains %d/%d elements", len(formatted), len(initial))
}

//gets all results from channel and appends them to formatted slice
func receive() {
    for i := 0; i < ITERATIONS; i++ {
        select {
        case result := <-results:
            formatted = append(formatted, result)
        }
    }
}

//loops over initial slice and runs a goroutine per element
//that does some formatting operation and then pushes result to channel
func format() {
    for i := 0; i < ITERATIONS; i++ {
        go func(i int) {
            //simulate some formatting code that can take a while
            sleep := time.Duration(rand.Intn(10)) * time.Second
            time.Sleep(sleep)
            //append formatted string to result chan
            results <- fmt.Sprintf("%s formatted", initial[i])
        }(i)
    }

}

//initialize chans and inital slice
func init() {
    results = make(chan string, ITERATIONS)
    for i := 0; i < ITERATIONS; i++ {
        initial = append(initial, fmt.Sprintf("string #%d", i))
    }
}

func timeTrack(start time.Time, name string) {
    elapsed := time.Since(start)
    log.Printf("%s took %s", name, elapsed)
}

Answer 1

I can't find a way to have a simple for (ever) loop wrapping the select, with two cases (the one receiving from the results channel and another one like case <-done that wouldreturn from the function). Would it be a better pattern ?

If the channel is closed once all writers are done, you can use a simple for ... range loop:

for result := range ch {
    ... do something with the result ...
}

In order for this simple variant to work, the channel must become closed, otherwise the for loop will not terminate.

Should i close somewhere the channel, after all the iterations ?

If at all possible, yes.

i did not really try waitgroups ...

A sync.WaitGroup, or something very similar, is almost certainly the way to go here. Each goroutine that can write to the channel should be accounted for initially, e.g.:

var wg Sync.WaitGroup
wg.Add(ITERATIONS)

Then you can just spawn all your goroutines that write and let them run. As each runs, it calls wg.Done() to indicate that it is finished.

You then—somewhere; the where part is slightly tricky—call wg.Wait() to wait for all of the writers to be done. When all the writers indicate that they are done, you can close() the channel.

Note that if you call wg.Wait() from the same goroutine that is reading the channel—i.e., the goroutine that will run the for result := range ... loop—you have a problem: you cannot simultaneously read from the channel and wait for the writers to write to the channel. So you either have to call wg.Wait() after the loop finishes, which is too late; or before the loop starts, which is too early.

This makes the problem and its solution clear: you must read from the channel in one goroutine, and do the wait-and-then-close in another goroutine. At most one of these goroutines can be the main one that entered the main function originally.

It tends to be pretty simple to make the wait-and-then-close goroutine its own private one:

go func() {
    wg.Wait()
    close(results)
}()

for instance.

what if for some reason, one of the goroutine fails?

You'll need to define precisely what you mean by fails here, but if you mean: what if the called goroutine itself calls, say, panic and therefore does not get to its wg.Done() call, you can use defer to make sure the wg.Done() happens even on a panic:

func(args) {
    defer wg.Done()
    ... do the work ...
}

wg.Add(1)
go func(args) // `func` will definitely call `wg.Done`, even if `func` panics

Should i pass channels in func arguments or let them global to the program as it is ?

Stylistically, global variables are always a bit messy. This does not mean you cannot use them; it's up to you, just keep all the tradeoffs in mind. Closure variables are not as messy, but remember to be careful with for loop iteration variables:

for i := 0; i < 10; i++ {
    go func() {
        time.Sleep(50 * time.Millisecond)
        fmt.Println(i)  // BEWARE BUG: this prints 10, not 0-9
    }()
}

behaves badly. Try this on the Go playground; note that go vet now complains about the bad use of i here.

I took your original sample code over to the Go Playground and made minimal changes to it as described above. The result is here. (To make it less slow I made the sleeps wait n-hundred-milliseconds instead of n seconds.)