GOLANG, HTTP having "use of closed network connection" error

Question

I am getting a lot of error like below mentioned,

read tcp xx.xx.xx.xx:80: use of closed network connection

read tcp xx.xx.xx.xx:80: connection reset by peer

//function for HTTP connection

func GetResponseBytesByURL_raw(restUrl, connectionTimeOutStr, readTimeOutStr string) ([]byte, error) {
    connectionTimeOut, _ /*err*/ := time.ParseDuration(connectionTimeOutStr)
    readTimeOut, _ /*err*/ := time.ParseDuration(readTimeOutStr)
    timeout := connectionTimeOut + readTimeOut // time.Duration((strconv.Atoi(connectionTimeOutStr) + strconv.Atoi(readTimeOutStr)))
    //timeout = 200 * time.Millisecond
    client := http.Client{
        Timeout: timeout,
    }
    resp, err := client.Get(restUrl)
    if nil != err {
        logger.SetLog("Error GetResponseBytesByURL_raw |err: ", logs.LevelError, err)
        return make([]byte, 0), err
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    return body, err
}

Update (July 14):

Server : NumCPU=8, RAM=24GB, GO=go1.4.2.linux-amd64

I am getting such error during some high traffic. 20000-30000 request per minutes, and I have a time frame of 500ms to fetch response from third party api.

netstat status from my server (using : netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n) to get frequency

      1 established)
      1 Foreign
      9 LISTEN
     33 FIN_WAIT1
    338 ESTABLISHED
   5530 SYN_SENT
  32202 TIME_WAIT

sysctl -p

**sysctl -p**
fs.file-max = 2097152
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.ip_local_port_range = 2000 65535
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.core.rmem_default = 31457280
net.core.rmem_max = 12582912
net.core.wmem_default = 31457280
net.core.wmem_max = 12582912
net.core.somaxconn = 65536
net.core.netdev_max_backlog = 65536
net.core.optmem_max = 25165824
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.udp_mem = 65536 131072 262144
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384
net.ipv4.tcp_wmem = 8192 65536 16777216
net.ipv4.udp_wmem_min = 16384
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv6.bindv6only = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
error: "net.ipv4.icmp_ignore_bogus_error_messages" is an unknown key
kernel.exec-shield = 1
kernel.randomize_va_space = 1
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

Looks like the server is closing the connections without proper content length headers. Did you check the headers and content returned? — Not_a_Golfer
What is your rate of request/roughly how much data are you trying to pull per minute? I've only ever received responses like that as a matter of hodge podge rate limiting... — evanmcdonnal
As the others have said, this is because the sever is closing the connection. Others possible reasons I've encountered: The server has a timeout that consistently closes the connection at about the same time you're making the next request, or it's an poor reverse-proxy/server combination that is incorrectly closing requests without a Connection: close header. — JimB
@JimB I think same, but tcp_keepalive_time = 300 & tcp_keepalive_time = 300 — niraj.nijju

JimB JimB · Accepted Answer · 2015-07-14T14:17:29

When making connections at a high rate over the internet, it's very likely you're going to encounter some connection problems. You can't mitigate them completely, so you may want to add retry logic around the request. The actual error type at this point probably doesn't matter, but matching the error string for use of closed network connection or connection reset by peer is about the best you can do if you want to be specific. Make sure to limit the retries with a backoff, as some systems will drop or reset connections as a way to limit request rates, and you may get more errors the faster you reconnect.

Depending on the number of remote hosts you're communicating with, you will want to increase Transport.MaxIdleConnsPerHost (the default is only 2). The fewer hosts you talk to, the higher you can set this. This will decrease the number of new connections made, and speed up the requests overall.

If you can, try the go1.5 beta. There have been a couple changes around keep-alive connections that may help reduce the number of errors you see.

GOLANG, HTTP having "use of closed network connection" error

2 Answers