aiohttp

Im in the process of moving some synchronous code to asyncio using aiohttp. the synchronous code was taking 15 minutes to run, so I'm hoping to improves this. I have some working code which gets data from some urls and returns the body of each. But this is just against 1 lab site, I have 70+ actual sites. So if I got a loop to create a list of all the urls for all sites that would make 700 urls in a list to be processed. Now processing them I don't think is a problem? But doing 'stuff' with the results, I'm not sure how to program? I have code already that will do 'stuff' to each of the

I'm using aiohttp to build an API server that sends TCP requests off to a seperate server. The module that sends the TCP requests is synchronous and a black box for my purposes. So my problem is that these requests are blocking the entire API. I need a way to wrap the module requests in an asynchronous coroutine that won't block the rest of the API. So, just using sleep as a simple example, is there any way to somehow wrap time-consuming synchronous code in a non-blocking coroutine, something like this: async def sleep_async(delay): # After calling sleep, loop should be released until sleep is

Simple example: I need to make two unrelated HTTP requests in parallel. What's the simplest way to do that? I expect it to be like that: async def do_the_job(): with aiohttp.ClientSession() as session: coro_1 = session.get('http://httpbin.org/get') coro_2 = session.get('http://httpbin.org/ip') return combine_responses(await coro_1, await coro_2) In other words, I want to initiate IO operations and wait for their results so they effectively run in parallel. This can be achieved with asyncio.gather : async def do_the_job(): with aiohttp.ClientSession() as session: coro_1 = session.get('http:/

tl;dr : how do I maximize number of http requests I can send in parallel? I am fetching data from multiple urls with aiohttp library. I'm testing its performance and I've observed that somewhere in the process there is a bottleneck, where running more urls at once just doesn't help. I am using this code: import asyncio import aiohttp async def fetch(url, session): headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0'} try: async with session.get( url, headers=headers, ssl = False, timeout = aiohttp.ClientTimeout( total=None, sock_connect = 10,

I am trying to understand what is the right way to use aiohttp with Sanic. From aiohttp documentation , I find the following: Don’t create a session per request . Most likely you need a session per application which performs all requests altogether. More complex cases may require a session per site, e.g. one for Github and another one for Facebook APIs. Anyway making a session for every request is a very bad idea. A session contains a connection pool inside. Connection reuse and keep-alive (both are on by default) may speed up total performance. And when I go to Sanic documentation I find an