18
votes

Since Python 3.5 introduced async with the syntax recommended in the docs for aiohttp has changed. Now to get a single url they suggest:

import aiohttp
import asyncio

async def fetch(session, url):
    with aiohttp.Timeout(10):
        async with session.get(url) as response:
            return await response.text()

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    with aiohttp.ClientSession(loop=loop) as session:
        html = loop.run_until_complete(
            fetch(session, 'http://python.org'))
        print(html)

How can I modify this to fetch a collection of urls instead of just one url?

In the old asyncio examples you would set up a list of tasks such as

    tasks = [
            fetch(session, 'http://cnn.com'),
            fetch(session, 'http://google.com'),
            fetch(session, 'http://twitter.com')
            ]

I tried to combine a list like this with the approach above but failed.

1
Could you explain what is your fail? - Andrew Svetlov
@AndrewSvetlov Wonderful to hear from you. What I mean is I could not understand how to do it. When I define a list of tasks then use results = loop.run_until_complete(tasks) I get a runtime error. async with is such a new feature with so little literature that it would be super convenient for people learning to use it if the aiohttp doc showed an example of grabbing more than one url. The library looks terrific, just needing a bit of hand-holding to get started. Thank you! - Hans Schindler

1 Answers

38
votes

For parallel execution you need an asyncio.Task

I've converted your example to concurrent data fetching from several sources:

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        if response.status != 200:
            response.raise_for_status()
        return await response.text()

async def fetch_all(session, urls):
    tasks = []
    for url in urls:
        task = asyncio.create_task(fetch(session, url))
        tasks.append(task)
    results = await asyncio.gather(*tasks)
    return results

async def main():    
    urls = ['http://cnn.com',
            'http://google.com',
            'http://twitter.com']
    async with aiohttp.ClientSession() as session:
        htmls = await fetch_all(session, urls)
        print(htmls)

if __name__ == '__main__':
    asyncio.run(main())