Go - HTTP Requests, JSON, reusing connections

Question

I'm using Go to make many requests over HTTPS, and I'm having issues with not reusing connections, and running out of ports. The requests I'm making are to an API that returns data in the JSON format, which I then json.Decode into a Go value.

According to questions I've come across on this site (#1, #2), in order for Go to reuse the connection for another request, I must read the entirety of the response's body before closing (note this wasn't always the behavior, as stated here).

Previously the HTTP client's (*Response).Body.Close would try to keep
reading until EOF, hoping to reuse the keep-alive HTTP connection...

In a typical instance, I would use the example shown in the earlier links, like so:

ioutil.ReadAll(resp.Body)

but since I'm pulling the data out of the JSON through code like this:

...

req, _ := http.NewRequest("GET", urlString, nil)
req.Header.Add("Connection", "keep-alive")
resp, err = client.Do(req)
defer resp.Body.Close()

...

decoder := json.NewDecoder(resp.Body)
decoder.Decode(data)

I'm not sure how the two methods would interact.

So the question is, how do I make sure the entire response has been read, so that the connection can later be reused for another request?

Hmm. Maybe simplest is to implement the old behavior yourself: defer func() { io.Copy(ioutil.Discard, resp.Body); resp.Body.Close() }(). — twotwotwo
Maybe after decoder.Decode(data), returned then you have read all your data. — nvcnvn
If you're only expecting 1 json response of a reasonable size in the body, don't bother with the Decoder. Use ReadAll and json.Unmarshal. — JimB
I'm expecting the JSON to be of a reasonable size, but there may be many concurrent/parallel goroutines each handling a response. Currently I'm on a system that has plenty of RAM to handle this, but I don't want to rely on that. — bawjensen

JimB JimB · Accepted Answer · 2015-10-20T21:37:47

If you want to only decode a single object using the Decoder, then you can use the More() method to see if there is more in the stream that needs to be read.

decoder := json.NewDecoder(resp.Body)
err := decoder.Decode(data)
if err != nil {
    // handle err
}
if decoder.More() {
    // there's more data in the stream, so discard whatever is left
    io.Copy(ioutil.Discard, resp.Body)
}

You can also just defer the Copy on every call, but this way you could more easily handle or log if there was unexpected data or an error.

Go - HTTP Requests, JSON, reusing connections

1 Answers