5
votes

I'm using Google App Engine Flexible env (already migrated to env:flex) with python 3.4 runtime.

Over last month, i noticed multiple times(but less than 5% of requests) that i or automated processes, get a 502 from the server (Bad Gateway). Couldn't reproduce it locally and couldn't find any trace for it under GAE service logs. But looking for 502 error across all services, i realized that they come from Cloud HTTP Load Balancer service.

Going over the jsonPayload of these 502 errors, i see this reason:

statusDetails: "failed_to_pick_backend"
@type:"type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

I don't have a custom load-balancer service from GCP, i never had to deal with it till now. The GAE VM instance is consistently below 10% of CPU Utilization

Any suggestions on how to solve it?

This question seems to be related: stackoverflow.com/questions/41305122/… - István
It does sounds related, but again, as am using GAE, i don't have control over the load balancer - Yarh
Did you figure this out? - luke
I figured it out, but not by digging into the logs. The issue was that there were requests to the server that called a function that does time.sleep(something), and doing that too much (like for waiting for an Ajax request) caused the load balancer error. Once i removed this function, the errors disappeared. - Yarh
Did you have manual scaling set in your app.yaml? - Jason Washo