Google Cloud Run is very slow vs. local machine

12
votes

We have a small script that scrapes a webpage (~17 entries), and writes them to Firestore collection. For this, we deployed a service on Google Cloud Run.

The execution of this code takes ~5 seconds when tested locally using Docker Container image. The same image when deployed to Cloud Run takes over 1 minute.

Even simple command as "Delete all Documents in a Collection", which takes 2-3 seconds locally, takes over 10 seconds when deployed on Cloud Run.

We are aware of Cold Start, and so we tested the performance of Cloud Run on the third, fourth and fifth subsequent runs, but it's still quite slow.

We also experimented with the number of CPUs, instances, concurrency, memory, using both default values as well as extreme values at both ends, but Cloud Run's performance is slow.

Is this expected? Are individual instances of Cloud Run really this weak? Can we do something to make it faster?

The problem with this slowness is that if we run our code for large number of entries, Cloud Run would eventually time out (not to mention the cost of Cloud Run per second)

1
google-cloud-run
What is the number of CPU of you local environment? And it's velocity in Mhz? Can you try to create a VM (n1-standard1) and to run you script on it to compare if it's Cloud Run environment (and its GVisor sandbox) that slow you process, or simply if it's the nb of CPU. Another way,, do you have GVisor warning in your stackdriver logs? – guillaume blaquiere
In addition to the above, can you also clarify if you write a lot on the filesystem? Cloud Run's filesystem is slower than a local machine. – Steren
Thank you for comments. We figured out the reason, and it was our own implementation eventually moving to background calls which is not recommended by Cloud Run documentation. – sudcha

1 Answers

10
votes

Posting answer to my own question as we experimented a lot with this, and found issues in our own implementation.

In our case, the reason for super slow performance was async calls without Promises or callbacks.

What we initially missed was this section: Avoiding background activities

Our code didn't wait for the async operation to end, and responding to the request right away. The async operation then moved to background activity and took forever to finish.

Responding to comments posted, or similar questions that may arise: 1. We didn't try experiment with local by setting up a VM with same config because we figured out the cause sooner.

  1. We are not writing anything on filesystem (yet), and operations are simple calls. But this is a good question, and we'll keep it in mind when we store/write data