How to resolve google app engine latency while autoscaling?

Question

Our project is running on the Google App Engine standard environment with auto-scaling configured to automatic. Warm up requests are enabled in the app and we are using Google Endpoints service. However, I am facing latency issue while scaling up. For example, it's taking more than 50 seconds for the API. From the application log I can notice that loading the AppEngineWebAppContext itself is taking upto 52 seconds. How can I reduce the latency during scaling?

Environment: Java 8, Instance type: F4_1G

Answer 1

What are you probably seeing taking 50 seconds is the pending latency which is measured by how long a request stays in the queue before it is handled by an instance.

You should check your logs and see which requests takes more than usual and try to break them in smaller jobs. Usually this helps when you are autoscaling since by having more small jobs instead of one huge one or several will help with this latency issue you are experiencing.

Also, pending latency will probably go up with lots of small jobs but this will cause the app to scale it properly.