Alternative to Cloud Tasks / Cron / Task Queue on CGP in Python 3 that doesn't have a 10 minute timeout

Question

I've recently started using an App Engine on Google Cloud Platform and have set up some cron jobs to get some scheduled work done. However recently one of my tasks took more than 10 minutes and it timed out... obviously I could break this work into batches or find another way around the problem, however I'm keen to not always be mindful of how long a job might take and want future jobs to run until completed or failed.

I've looked into various services that Google offer but with no success; Task Queue is Python 2.x only and Cloud Tasks has the same 10 minute limit unless you manually manage scaling (which I would prefer to stay automatic as that's the point of App Engine for me).

Am I missing something? This 10 minute limit seems like a big unnecessary blocker and I have no idea where to look.

https://cloud.google.com/tasks/docs/creating-appengine-handlers

Thanks for your time.

Answer 1

Google services such as App Engine are designed to model a web server HTTP Request / Response design. You are trying to use them as task/execute engines.

Use the correct service if you require long execution times, which usually means requests that take longer than a few minutes to complete. Use Cloud Tasks and Compute Engine. Otherwise you will need to architect your application to fit with App Engine's requirements and limitations.

Cloud Tasks for Asynchronous task execution

Answer 2

If you want to use App Engine, you need to use either Basic Scaling or Manual Scaling. I understand that manual scaling isn't your favorite, I also don't like this mode. But the basic scaling is acceptable.

In addition, it's more designed to perform background task, exactly what you try to achieve.

If you accept this change, you can use Cloud Task. You have up to 24H of timeout if your App Engine service is in basic scaling (or manual)

You have this same information, on the scaling description on App Engine documentation.

When you use basic scaling, your instance type needs to be updated to BXXX.