Cloudfront custom-origin distribution returns 502 “ERROR The request could not be satisfied.” for some URLs

71
votes

We have a Cloudfront distribution with custom origin that has been working just fine for quite a long time, serving static assets for one of our sites. Just this morning, we noticed that our logo was displaying as a broken link.

Upon further investigation, Cloudfront is returning a strange error message that I've never seen before for the URL in question:

ERROR

The request could not be satisfied.

Generated by cloudfront (CloudFront)

Several other Cloudfront URLs from this distribution return the same error, but then others (again, from the same distribution) are working just fine. I don't see a pattern to what works and what doesn't.

Some other data points:

  • The origin URLs work just fine. There's been no recent interruption in service, to my knowledge.
  • I've invalidated the logo URL specifically, to no effect.
  • I've invalidated the root URL of the distribution, to no effect.

Any idea what's going on here? I've never seen Cloudfront do this before.

UPDATE:

Here's the verbatim HTTP response from Cloudfront:

$ http GET https://d2yu7foswg1yra.cloudfront.net/static/img/crossway_logo.png
HTTP/1.1 502 Bad Gateway
Age: 213
Connection: keep-alive
Content-Length: 472
Content-Type: text/html
Date: Wed, 18 Dec 2013 17:57:46 GMT
Server: CloudFront
Via: 1.1 f319e8962c0268d31d3828d4b9d41f98.cloudfront.net (CloudFront)
X-Amz-Cf-Id: H_HGBG3sTOqEomHzHubi8ruLbGXe2MRyVhGBn4apM0y_LjQa_9W2Jg==
X-Cache: Error from cloudfront

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
</BODY></HTML>

<BR clear="all">
<HR noshade size="1px">
<ADDRESS>
Generated by cloudfront (CloudFront)
</ADDRESS>
</BODY></HTML>
14
sslcdnamazon-cloudfront
Interesting.... I just created my first distribution (no custom CNAME) and am getting the same thing. Started with everything basic but no luck yet. - Shawn Dube
Yes, I created a new distribution to test with, and same thing. :\ - David Eyk
I had a similar issue, although I got a 504 gateway time out for some static files from a CloudFront distribution. I realised that I had enabled pglcmd which was blocking IP ranges through iptables. I still don't know why CloudFront was checking for these files, which have expiration headers set for one year. - paradroid

14 Answers

35
votes

I had a similar issue recently which turned out to be due to ssl_ciphers that I was using.

From http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html,

"CloudFront forwards HTTPS requests to the origin server using the SSLv3 or TLSv1 protocols and the AES128-SHA1 or RC4-MD5 ciphers. If your origin server does not support either the AES128-SHA1 or RC4-MD5 ciphers, CloudFront cannot establish an SSL connection to your origin. "

I had to change my nginx confg to add AES128-SHA ( deprecated RC4:HIGH ) to ssl_ciphers to fix the 302 error. I hope this helps. I have pasted the line from my ssl.conf 

ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:RSA+3DES:AES128-SHA:!ADH:!AECDH:!MD5;
128
votes

I had this error today with Amazon Cloudfront. It was because the cname I used (e.g cdn.example.com) was not added to the distribution settings under "alternate cnames", I only had cdn.example.com forwarded to the cloudfront domain in my site/hosting control panel, but you need to add it to Amazon CloudFront panel too.

13
votes

Found my answer and adding it here in case it helps David (and others).

Turns out my origin server (say www.example.com) had a 301 redirect setup on it to change HTTP to HTTPS:

HTTP/1.1 301 Moved Permanently
Location: https://www.example.com/images/Foo_01.jpg

However, my Origin Protocol Policy was set to HTTP only. This caused CloudFront to not find my file and throw a 502 error. Additionally, I think it cached the 502 error for 5 min or so as it didn't work immediately after removing that 301 redirect.

Hope that helps!

9
votes

In our case, everything LOOKED ok, but it took most of the day to figure this out:

TLDR: Check your certificate paths to make sure the root certificate is correct. In the case of COMODO certificates, it should say "USERTrust" and be issued by "AddTrust External CA Root". NOT "COMODO" issued by "COMODO RSA Certification Authority".

From the CloudFront docs: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/SecureConnections.html

If the origin server returns an invalid certificate or a self-signed certificate, or if the origin server returns the certificate chain in the wrong order, CloudFront drops the TCP connection, returns HTTP error code 502, and sets the X-Cache header to Error from cloudfront.

We had the right ciphers enabled as per: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#RequestCustomEncryption

Our certificate was valid according to Google, Firefox and ssl-checker: https://www.sslshopper.com/ssl-checker.html

SSL Checker result without all required certificates

However the last certificate in the ssl checker chain was "COMODO RSA Domain Validation Secure Server CA", issued by "COMODO RSA Certification Authority"

It seems that CloudFront does not hold the certificate for "COMODO RSA Certification Authority" and as such thinks the certificate provided by the origin server is self signed.

This was working for a long time before apparently suddenly stopping. What happened was I had just updated our certificates for the year, but during the import, something was changed in the certificate path for all the previous certificates. They all started referencing "COMODO RSA Certification Authority" whereas before the chain was longer and the root was "AddTrust External CA Root".

Bad certificate path

Because of this, switching back to the older cert did not fix the cloudfront issue.

I had to delete the extra certificate named "COMODO RSA Certification Authority", the one that did not reference AddTrust. After doing this, all my website certificates' paths updated to point back to AddTrust/USERTrust again. Note can also open up the bad root certificate from the path, click "Details" -> "Edit Properties" and then disable it that way. This updated the path immediately. You may also need to delete multiple copies of the certificate, found under "Personal" and "Trusted Root Certificate Authorities"

Good certificate path

Finally I had to re select the certificate in IIS to get it to serve the new certificate chain.

After all this, ssl-checker started displaying a third certificate in the chain, which pointed back to "AddTrust External CA Root"

SSL Checker with all certificates

Finally, CloudFront accepted the origin server's certificate and the provided chain as being trusted. Our CDN started working correctly again!

To prevent this happening in the future, we will need to export our newly generated certificates from a machine with the correct certificate chain, i.e. distrust or delete the certificate "COMODO RSA Certification Authroity" issued by "COMODO RSA Certification Authroity" (expiring in 2038). This only seems to affect windows machines, where this certificate is installed by default.

4
votes

One more possible solution: I have a staging server that serves the site and the Cloudfront assets over HTTP. I had my origin set to "Match Viewer" instead of "HTTP Only". I also use the HTTPS Everywhere extension, which redirected all the http://*.cloudfront.net URLs to the https://* version. Since the staging server isn't available over SSL and Cloudfront was matching the viewer, it couldn't find the assets at https://example.com and cached a bunch of 502s instead.

2
votes

I just went through troubleshooting this issue, and in my case it indeed was related to redirects, but not related to incorrect settings in my CloudFront Origin or Behavior. This will happen if your origin server is still redirecting to origin URLs, and not what you have set up for your cloudfront URLs. Seems this is very common if you forget to change configs. For example lets say if you have www.yoursite.com CNAME to your cloudfront distribution, with an origin of www.yoursiteorigin.com. Obviously people will come to www.yoursite.com. But if your code tries to redirect to any page on www.yoursiteorigin.com you WILL get this error.

For me, my origin was still doing the http->https redirects to my origin URLs and not my Cloudfront URLs.

2
votes

In my case, it was because we had an invalid ssl cert. The problem was on our staging box and we use our prod cert on that as well. It had worked for the past couple of years with this configuration, but all of a sudden we started getting this error. Strange.

If others are getting this error, check that the ssl certificate is valid. You can enable logging to s3 via the AWS CloudFront Distribution interface to aid debugging.

Also, you can refer to amazon's docs on the matter here: http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/SecureConnections.html

1
votes

I ran into this problem, which resolved itself after I stopped using a proxy. Maybe CloudFront is blacklisting some IPs.

1
votes

Fixed this issue by concatenating my certificates to generate a valid certificate chain (using GoDaddy Standard SSL + Nginx).

http://nginx.org/en/docs/http/configuring_https_servers.html#chains

To generate the chain:

cat 123456789.crt gd_bundle-g2-g1.crt > my.domain.com.chained.crt

Then: 

ssl_certificate /etc/nginx/ssl/my.domain.com.chained.crt;
ssl_certificate_key /etc/nginx/ssl/my.domain.com.key;

Hope it helps!

1
votes

The problem, in my case, was that I was using Amazon's Cloudflare and Cloudfront's Cloudfront in tandem, and Cloudfront did not like the settings that I had provided Cloudflare.

More specifically, in the Crypto settings on Cloudflare, I had set the "Minimum TLS Settings" to 1.2, without enabling the TLS 1.2 communication setting for the distribution in Cloudfront. This was enough to make Cloudfront declare a 502 Bad Gateway error when it tried to connect to the Cloudflare-protected server.

To fix this, I had to disable SSLv3 support in the Origin Settings for that Cloudfront distribution, and enable TLS 1.2 as a supported protocol for that origin server.

To debug this problem, I used command-line versions of curl, to see what Cloudfront was actually returning when you asked for an image from its CDN, and I also used the command-line version of openssl, to determine exactly which protocols Cloudflare was offering (it wasn't offering TLS 1.0).

tl:dr; make sure everything accepts and asks for TLS 1.2, or whatever latest and greatest TLS everyone is using by the time you read this.

1
votes

For my particular case it was due to the fact that the Origin ALB behind my CloudFront Behavior had a DEFAULT ACM certificate which was pointing to a different domain name.

To fix this I had to:

  1. Go to the ALB
  2. Under the Listeners tab, selected my Listener and then Edit
  3. Under the Default SSL Certificate, choose the correct Origin Certificate.
0
votes

In my case, I use nginx as reverse-proxy for an API Gateway URL. I got same error.

I resolved the issue when I added the following two lines to the Nginx config:

proxy_set_header Host "XXXXXX.execute-api.REGION.amazonaws.com";
proxy_ssl_server_name on;

Source is here: Setting up proxy_pass on nginx to make API calls to API Gateway

0
votes

In our case, we had dropped support for SSL3, TLS1.0, and TLS1.1 for PCI-DSS compliance on our origin servers. However, you have to manually add support for TLS 1.1+ on your CloudFront origin server config. The AWS console displays the client-to-CF SSL settings, but does not easily show you CF-to-origin settings until you drill down. To fix, in the AWS console under CloudFront:

  1. Click DISTRIBUTIONS.
  2. Select your distro.
  3. Click ORIGINS tab.
  4. Select your origin server.
  5. Click EDIT.
  6. Select all protocols that your origin supports under "Origin SSL Protocols"
0
votes

Beware the Origin Protocol Policy:

For HTTPS viewer requests that CloudFront forwards to this origin, one of the domain names in the SSL certificate on your origin server must match the domain name that you specify for Origin Domain Name. Otherwise, CloudFront responds to the viewer requests with an HTTP status code 502 (Bad Gateway) instead of returning the requested object.

In most cases, you probably want CloudFront to use "HTTP Only", since it fetches objects from a server probably hosted with Amazon too. No need for additional HTTPS complexity at this step.

Note that this is different to the Viewer Protocol Policy. You can read more about the differences between the two here.