Why is API Gateway sometimes stuck returning "502 Bad Gateway" when accessed via a custom domain

Question

I have a CloudFormation stack that uses API Gateway, Lambda and a custom domain. When I fully tear down the stack and redeploy it I often find that API Gateway is in a state where it returns "502 Bad Gateway" for every API request made through the custom domain.

Here are some other observations:

I can still successfully call the API using the standard execution URL (e.g.<api-id>.execute-api.us-east-1.amazonaws.com)
All the domain settings in the AWS Console (API Gateway and Route 53) appear correct
In the AWS Console I can manually create an alternate custom domain with the same settings and it works
I tried deleting the custom domain manually and recreating it with the same settings but it fails the same way
The only fix seems to be to delete the stack and wait a day or two to redeploy

This "feels like" an API Gateway caching bug where it is still trying to route my custom domain to the old (since deleted) API instance (even though the AWS Console shows the correct values).

Any idea what might cause this problem or how to debug it? The logs I've looked at seem unhelpful because the error occurs before my API is actually reached.

The relevant parts of my CloudFormation looks like this:

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31

...

 ApiDomainName:
    Type: AWS::ApiGatewayV2::DomainName
    Properties:
      DomainName: !Ref DomainName
      DomainNameConfigurations:
        - CertificateArn: !Ref CertificateArn

  ApiMapping:
    Type: AWS::ApiGatewayV2::ApiMapping
    Properties:
      ApiId: !Ref ServerlessHttpApi
      DomainName: !Ref ApiDomainName
      Stage: !Ref ServerlessHttpApiApiGatewayDefaultStage

  DNSRecord:
    Type: AWS::Route53::RecordSet
    Properties:
      Name: !Ref DomainName
      Type: A
      AliasTarget:
        DNSName: !GetAtt ApiDomainName.RegionalDomainName
        HostedZoneId: !GetAtt ApiDomainName.RegionalHostedZoneId
      HostedZoneId: !Ref HostedZoneId

> curl https://api.example.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>
                                    
> curl https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com
{"message":"Hello World"}%

Crossposted to aws dev forums here: https://forums.aws.amazon.com/thread.jspa?messageID=960826

Ihor Shylo Ihor Shylo · Accepted Answer · 2020-10-21T10:11:33

It seems like you are building a serverless application. For this I would be using Serverless Application Model (SAM): https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started.html

Once you switch, a declaration of the custom domain name would look as follows:

    MyApi:
        Type: AWS::Serverless::Api
        Properties:
            BinaryMediaTypes: [image~1jpg]
            StageName: !Ref Stage
            EndpointConfiguration: REGIONAL
            Domain:
                DomainName: api.example.com
                CertificateArn: <ARN of the Certificate from ACM>
                EndpointConfiguration: REGIONAL
                Route53:
                  HostedZoneId: <Hosted zone id from route 53>

Why is API Gateway sometimes stuck returning "502 Bad Gateway" when accessed via a custom domain

2 Answers