client.Timeout exceeded while awaiting headers

Question

I have a lambda written in Go that communicates with a lightweight http app behind an application load balancer in AWS:

package main

import (
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "time"
)

func main() {

    for {
        req, err := http.NewRequest("GET", "http://my-app-12345.us-east-1.elb.amazonaws.com:8080", nil)
        if err != nil {
            log.Fatal("Error reading request. ", err)
        }

        req.Header.Set("Load-Rate", os.Getenv("LOAD"))

        client := &http.Client{Timeout: time.Second * 10}

        resp, err := client.Do(req)
        if err != nil {
            log.Fatal("Error reading response. ", err)
        }
        if resp.StatusCode == http.StatusOK {
            bodyBytes, err := ioutil.ReadAll(resp.Body)
            if err != nil {
                log.Fatal(err)
            }
            bodyString := string(bodyBytes)
            log.Println(bodyString)
        }

        time.Sleep(time.Duration(10) * time.Millisecond)
    }
}

I've opened worldwide access on all ports in the security groups being used by both the lambda and the ALB and I can curl the endpoint fine from my home machine. However, when I attempt to run this in Lambda, I get the following in my CloudWatch logs:

2020-06-04T07:06:31.028-05:00 Process exited before completing request
2020-06-04T07:06:41.100-05:00 2020/06/04 12:06:41 Error reading response. Get http://my-app-12345.us-east-1.elb.amazonaws.com:8080: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-06-04T07:06:41.101-05:00 2020/06/04 12:06:41 exit status 1

I'm having a hard time figuring out if this is a Go issue or some configuration I have wrong in AWS. I'm not very familiar at all with the error Go is throwing here.

Just like the error says, the request timed out. There's not much we can guess from that, other than it took more than the 10 seconds you gave it, probably because it still can't connect. — JimB

Marcin Marcin · Accepted Answer · 2020-06-04T12:30:24

You lambda seems to be in VPC since you write about its security groups. If this is the case then the most likely reason for the timeout is that lambda in VPC does not have internet access nor public IP, even if its in public subnet.

From docs:

Connecting a function to a public subnet does not give it internet access or a public IP address.

Since your ALB is public (you curl it from home) your lambda can't access it, even if they are both in the same subnet.

To rectify this you would have to setup NAT gateway or instance with correct routes to be able to communicate from lambda in VPC to your public ALB, and place your lambda in a private subnet.

client.Timeout exceeded while awaiting headers

1 Answers