Golang writing to http response breaks input reading?

Question

I'm attempting to write a small webapp in Go where the user uploads a gzipped file in a multipart form. The app unzips and parses the file and writes some output to the response. However, I keep running into an error where the input stream looks corrupted when I begin writing to the response. Not writing to the response fixes the problem, as does reading from a non-gzipped input stream. Here's an example http handler:

func(w http.ResponseWriter, req *http.Request) {

//Get an input stream from the multipart reader
//and read it using a scanner
multiReader, _ := req.MultipartReader()
part, _ := multiReader.NextPart()
gzipReader, _ := gzip.NewReader(part)
scanner := bufio.NewScanner(gzipReader)

//Strings read from the input stream go to this channel     
inputChan := make(chan string, 1000)

//Signal completion on this channel 
donechan := make(chan bool, 1)

//This goroutine just reads text from the input scanner
//and sends it into the channel 
go func() {
    for scanner.Scan() {
        inputChan <- scanner.Text()
    }       
    close(inputChan)
}()

//Read lines from input channel. They all either start with #
//or have ten tab-separated columns
go func() {
    for line := range inputChan {
        toks := strings.Split(line, "\t")
        if len(toks) != 10 && line[0] != '#' {
            panic("Dang.")
        }
    }
    donechan <- true 
}()

//periodically write some random text to the response
go func() {
    for {
        time.Sleep(10*time.Millisecond)     
        w.Write([]byte("write\n some \n output\n"))
    }
}()

//wait until we're done to return
<-donechan
}

Weirdly, this code panics every time because it always encounters a line with fewer than 10 tokens, although at different spots every time. Commenting out the line that writes to the response fixes the issue, as does reading from a non-gzipped input stream. Am I missing something obvious? Why would writing to the response break if reading from a gzip file, but not a plain text formatted file? Why would it break at all?

icza icza · Accepted Answer · 2015-02-05T07:39:01

The HTTP protocol is not full-duplex: it is request-response based. You should only send output once you're done with reading the input.

In your code you use a for with range on a channel. This will try to read the channel until it is closed, but you never close the inputChan.

If you never close inputChan, the following line is never reached:

donechan <- true

And therefore receiving from donechan blocks:

<-donechan

You have to close the inputChan when EOF is reached:

go func() {
    for scanner.Scan() {
        inputChan <- scanner.Text()
    }       
    close(inputChan) // THIS IS NEEDED
}()

Golang writing to http response breaks input reading?

1 Answers