Maximize number of parallel requests (aiohttp)

Question

tl;dr: how do I maximize number of http requests I can send in parallel?

I am fetching data from multiple urls with aiohttp library. I\

Answer 1

By default aiohttp limits number of simultaneous connections to 100. It achieves by setting default limit to TCPConnector object that is used by ClientSession. You can bypass it by creating and passing custom connector to session:

connector = aiohttp.TCPConnector(limit=None)
async with aiohttp.ClientSession(connector=connector) as session:
    # ...

Note however that you probably don't want to set this number too high: your network capacity, CPU, RAM and target server have their own limits and try to make enormous amount of connection can lead to increasing failures.

Optimal number can probably be found only through experiments on concrete machine.

---

Unrelated:

You don't have to create tasks without reason. Most asyncio api accept regular coroutines. For example, your last lines of code can be altered this way:

loop = asyncio.get_event_loop()
loop.run_until_complete(run(url_list))

Or even to just asyncio.run(run(url_list)) (doc) if you're using Python 3.7