Reusing http connections in Go

Question

I'm currently struggling to find a way to reuse connections when making HTTP posts in Go.

I've created a transport and client like so:

// Create a new transport and HTTP client
tr := &http.Transport{}
client := &http.Client{Transport: tr}

I'm then passing this client pointer into a goroutine which is making multiple posts to the same endpoint like so:

r, err := client.Post(url, "application/json", post)

Looking at netstat this appears to be resulting in a new connection for every post resulting in a large number of concurrent connections being open.

What is the correct way to reuse connections in this case?

Answer 1

Ensure that you read until the response is complete AND call Close().

e.g.

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()

Again... To ensure http.Client connection reuse be sure to:

• Read until Response is complete (i.e. ioutil.ReadAll(resp.Body))
• Call Body.Close()

Answer 2

If anyone is still finding answers on how to do it, this is how I am doing it.

package main

import (
  "bytes"
  "io/ioutil"
  "log"
  "net/http"
  "time"
)

func httpClient() *http.Client {
    client := &http.Client{
        Transport: &http.Transport{
            MaxIdleConnsPerHost: 20,
        },
        Timeout: 10 * time.Second,
    }

    return client
}

func sendRequest(client *http.Client, method string) []byte {
    endpoint := "https://httpbin.org/post"
    req, err := http.NewRequest(method, endpoint, bytes.NewBuffer([]byte("Post this data")))
    if err != nil {
        log.Fatalf("Error Occured. %+v", err)
    }

    response, err := client.Do(req)
    if err != nil {
        log.Fatalf("Error sending request to API endpoint. %+v", err)
    }

    // Close the connection to reuse it
    defer response.Body.Close()

    body, err := ioutil.ReadAll(response.Body)
    if err != nil {
        log.Fatalf("Couldn't parse response body. %+v", err)
    }

    return body
}

func main() {
    c := httpClient()
    response := sendRequest(c, http.MethodPost)
    log.Println("Response Body:", string(response))
}

Go Playground: https://play.golang.org/p/cYWdFu0r62e

In summary, I am creating a different method to create an HTTP client and assigning it to a variable, and then using it to make requests. Note the

defer response.Body.Close() 

This will close the connection after the request is complete at the end of the function execution and you can reuse the client as many times.

If you want to send a request in a loop call the function that sends the request in a loop.

If you want to change anything in the client transport configuration, like add proxy config, make a change in the client config.

Hope this will help someone.

Answer 3

Edit: This is more of a note for people that construct a Transport and Client for every request.

Edit2: Changed link to godoc.

Transport is the struct that holds connections for re-use; see https://godoc.org/net/http#Transport ("By default, Transport caches connections for future re-use.")

So if you create a new Transport for each request, it will create new connections each time. In this case the solution is to share the one Transport instance between clients.

Answer 4

IIRC, the default client does reuse connections. Are you closing the response?

Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.

Answer 5

about Body

// It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.

So if you want to reuse TCP connections, you have to close Body every time after read to completion. Also, with defer, you can make sure Body.Close() is called after all. An function ReadBody(io.ReadCloser) is suggested like this.

package main

import (
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

func main() {
    req, err := http.NewRequest(http.MethodGet, "https://github.com", nil)
    if err != nil {
        fmt.Println(err.Error())
        return
    }
    client := &http.Client{}
    i := 0
    for {
        resp, err := client.Do(req)
        if err != nil {
            fmt.Println(err.Error())
            return
        }
        _, _ = readBody(resp.Body)
        fmt.Println("done ", i)
        time.Sleep(5 * time.Second)
    }
}

func readBody(readCloser io.ReadCloser) ([]byte, error) {
    defer readCloser.Close()
    body, err := ioutil.ReadAll(readCloser)
    if err != nil {
        return nil, err
    }
    return body, nil
}

And don't call Close like below:

res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body) // what if io.Copy panics, res.Body.Close() will not called.
res.Body.Close()

Answer 6

Another approach to init() is to use a singleton method to get the http client. By using sync.Once you can be sure that only one instance will be used on all your requests.

var (
    once              sync.Once
    netClient         *http.Client
)

func newNetClient() *http.Client {
    once.Do(func() {
        var netTransport = &http.Transport{
            Dial: (&net.Dialer{
                Timeout: 2 * time.Second,
            }).Dial,
            TLSHandshakeTimeout: 2 * time.Second,
        }
        netClient = &http.Client{
            Timeout:   time.Second * 2,
            Transport: netTransport,
        }
    })

    return netClient
}

func yourFunc(){
    URL := "local.dev"
    req, err := http.NewRequest("POST", URL, nil)
    response, err := newNetClient().Do(req)
    // ...
}

Answer 7

The missing point here is the "goroutine" thing. Transport has its own connection pool, by default each connection in that pool is reused (if body is fully read and closed) but if several goroutines are sending requests, new connections will be created (the pool has all connections busy and will create new ones). To solve that you will need to limit the maximum number of connections per host: Transport.MaxConnsPerHost (https://golang.org/src/net/http/transport.go#L205).

Probably you also want to setup IdleConnTimeout and/or ResponseHeaderTimeout.

Answer 8

https://golang.org/src/net/http/transport.go#L196

you should set MaxConnsPerHost explicitly to your http.Client. Transport does reuse the TCP connection, but you should limit the MaxConnsPerHost (default 0 means no limit).

func init() {
    // singleton http.Client
    httpClient = createHTTPClient()
}

// createHTTPClient for connection re-use
func createHTTPClient() *http.Client {
    client := &http.Client{
        Transport: &http.Transport{
            MaxConnsPerHost:     1,
            // other option field
        },
        Timeout: time.Duration(RequestTimeout) * time.Second,
    }

    return client
}

Answer 9

It is very useful function for GO http call, you can keep connection alive and resue this connection.

    var (
        respReadLimit       = int64(4096)
    )
    
    // Try to read the response body so we can reuse this connection.
    func (c *Client) drainBody(body io.ReadCloser) error {
        defer body.Close()
        _, err := io.Copy(ioutil.Discard, io.LimitReader(body, respReadLimit))
        if err != nil {
            return err
        }
        return nil
    }

Answer 10

client.Do() would handle closing the response body automatically. You just build the request and call the client.Do(req)

    func (c clientImpl) BackupSite(siteId)
        client := &http.Client{}
        req, err := http.NewRequest("POST", 
        fmt.Sprintf("https://%s/backup/site/%s/", 
                c.APIBaseURL, siteId), nil)
        if err != nil {
        return err
        }
        req.Header.Add("Api-Key", c.APIKey)
        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        if resp.StatusCode != http.StatusOK {
            return fmt.Errorf("failed to backup site," + 
                "site: %s, status code %d", siteId, resp.StatusCode)
        }
    }

Read more about client.Do() here

Answer 11

There are two possible ways:

1. Use a library that internally reuses and manages the file descriptors, associated with each requests. Http Client does the same thing internally, but then you would have the control over how many concurrent connections to open, and how to manage your resources. If you are interested, look at the netpoll implementation, which internally uses epoll/kqueue to manage them.

2. The easy one would be, instead of pooling network connections, create a worker pool, for your goroutines. This would be easy, and better solution, that would not hinder with your current codebase, and would require minor changes.

Let's assume you need to make n POST request, after you recieve a request.

You could use channels, to implement this.

Or, simply you could use third party libraries.
Like: https://github.com/ivpusic/grpool