Get aiohttp request response in callback with a infinite loop of asyncio task

Question

What I am doing- I am trying to make multiple post requests(to the same URL with different parameters) in an infinite loop using asyncio and aiohttp.

def callback(response):
    print(response) 

async def make_request(session, payload, sema):
        async with sema, session.post(url=URL, headers=HEADERS, data=json.dumps(payload)) \
                as response:
            return await response.json()


async def generate_requests(payloads):
    tasks = []
    sema = asyncio.BoundedSemaphore(value=100)
    async with aiohttp.ClientSession() as session:
        for payload in payloads:
            tasks.append(make_request(session, payload, sema))
        results = await asyncio.gather(*tasks, return_exceptions=True)
        print(results)


def run_asyncio_requests():
    loop = asyncio.get_event_loop()
    while True:
        for country in countries:
            cities = country.cities
            loop.run_until_complete(generate_requests(cities))


if __name__ == '__main__':
    run_asyncio_requests()

Problem: run_asyncio_requests function generates and make multiple requests(cities against a country) and then await for all responses for one country requests then executes the next country requests.

What I want: All countries' requests should be executed at once and callback(some kind of linked function with each request) function should handle each response because post request response time for each request is different. It should not wait to gather all the responses. Requests should be generated in each cycle(infinite-loop) and callback function in the background should be linked with each response. I know I am using asyncio.gather to get all responses until requests completed. But I have no idea how to do that.

user4815162342 user4815162342 · Accepted Answer · 2020-03-03T14:42:19

You can apply the same technique when invoking generate_requests that you did when invoking make_request:

async def generate_all_countries():
    tasks = [generate_requests(country.cities) for country in countries]
    results = await asyncio.gather(*tasks)
    print(results)

if __name__ == '__main__':
    asyncio.run(generate_all_countries())

Note that you don't need run_asyncio_requests as a separate sync function, you can simply invoke an async function using asyncio.run() and proceed from there.

Get aiohttp request response in callback with a infinite loop of asyncio task

1 Answers