I have an app that is running on Google App Engine and uses the Task Queue Api to do some of the heavier lifting in the background. Some of those tasks need to connect to Cloud SQL to do their work. At scale I get too many tasks attempting to connect to Cloud SQL at once. What I need is some sort of data service layer of a shared client so that the tasks themselves aren't making individual connections to Cloud SQL. If anyone has any ideas I'd love to hear them.
1 Answers
1
votes
Yes, you can actually do this, but it will take a little bit of planning, coding, and configuration in your side.
One idea is to use a Pull Queue (instead of Push Queues). With a pull queue you can schedule your tasks and execute them in a separate module of your application. The hardware of that module can be configured separately from your main module, thus you can avoid too many instances serving requests which in turns will allow you to better use connection pooling.
Of course, depending on the traffic you are getting you might want to decide on how many concurrent backend instances you want running (and connecting to your DB) to avoid/minimize contention.
Easier said than done, but here are two resources that will help you out:
- App Engine Modules - https://developers.google.com/appengine/docs/java/modules/
- Pull Queues - https://developers.google.com/appengine/docs/python/taskqueue/overview-pull
