Sync & Async Python FastAPI scripts on Google App Engine

Question

I've tried running the following py file script.py locally:

from fastapi import FastAPI
import asyncio
import time

app = FastAPI()

@app.get('/async/')
async def func(text: str):
    await asyncio.sleep(5)
    return {'text': text}


@app.get('/sync/')
async def func(text: str):
    time.sleep(5)
    return {'text': text}

using uvicorn script:app --reload

When making 10 concurrent requests to sync endpoint http://127.0.0.1:8000/sync/?text=test, each call takes 5 seconds and blocks the next, total time is 50 seconds

And when making 10 concurrent requests to async endpoint http://127.0.0.1:8000/async/?text=test, each call takes 5 seconds, total time is also 5 seconds as it's non-blocking.

I've tried deploying the script to Google App Engine using entrypoint: gunicorn -w 4 -k uvicorn.workers.UvicornWorker script:app and did the same tests instead of locally, and got the following results:

When making 10 concurrent requests to sync endpoint, the total time is 12 seconds, all of the 10 requests executed on the same 1 instance. ( 1000 concurrent requests took 60 seconds on 20+ instances )

And when making 10 concurrent requests to async endpoint, the total time is 5 seconds, all of the 10 requests executed on the same 1 instance. ( 1000 concurrent requests took 30 seconds on 20+ instances )

Why on GAE the sync code took 12 seconds for the 10 concurrent requests instead of 50 seconds?

And how can I run all 10 requests simultaneously on GAE for the sync endpoint, to get them all in 5 seconds?

using k6 stress test ( 10 Virtual users with 10 Iterations ) — Mezo

frankie567 frankie567 · Accepted Answer · 2021-02-18T11:15:10

In your gunicorn command, you specified the option -w 4, which means 4 workers processes are spawned (https://docs.gunicorn.org/en/stable/settings.html#worker-processes).

It's a common way to enable concurrency even on blocking synchronous code. Your 10 requests are split across the workers (10 requests * 5 seconds / 4 workers = 12.5 seconds).

If you run it like this:

gunicorn -w 1 -k uvicorn.workers.UvicornWorker script:app

You'll do experience 50 seconds of waiting.

But it's still strongly recommended to spawn several workers.

Now for the sync part, here is an interesting technical detail about FastAPI. If you define your route or dependency as async, it'll run in the main process, regardless of whether you are running blocking code or not inside.

This is why your /sync endpoint blocks.

However, if you define it as a standard non-async function, FastAPI will run it in a threadpool to avoid blocking the main process.

Doc: https://fastapi.tiangolo.com/async/?h=technical#path-operation-functions

If you write your /sync endpoint like this:

@app.get('/sync/')
def func(text: str):  # Just removed the async keyword
    time.sleep(5)
    return {'text': text}

You'll notice the endpoint is not blocking the main process anymore.

Sync & Async Python FastAPI scripts on Google App Engine

1 Answers