aiohttp set number of requests per second

问题

I'm writing an API in Flask with 1000+ requests to get data and I'd like to limit the number of requests per second. I tried with:

conn = aiohttp.TCPConnector(limit_per_host=20)

and 

conn = aiohttp.TCPConnector(limit=20)

But is seems doesn't work

My code looks like this:

import logging
import asyncio
import aiohttp

logging.basicConfig(filename="logfilename.log", level=logging.INFO, format='%(asctime)s %(levelname)s:%(message)s')


async def fetch(session, url):
    async with session.get(url, headers=headers) as response:
        if response.status == 200:
            data = await response.json()
            json = data['args']
        return json

async def fetch_all(urls, loop):
    conn = aiohttp.TCPConnector(limit=20)
    async with aiohttp.ClientSession(connector=conn, loop=loop) as session:
        results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True)
        return results

async def main():
    loop = asyncio.new_event_loop()
    url_list = []
    args = ['a', 'b', 'c', +1000 others]
    urls = url_list
    for i in args:
        base_url = 'http://httpbin.org/anything?key=%s' % i
        url_list.append(base_url)

    htmls = loop.run_until_complete(fetch_all(urls, loop))
    for j in htmls:
        key = j['key']
        # save to database
        logging.info(' %s was added', key)

If I run code, within 1s I send over than 200 requests. Is there any way to limit requests?

回答1:

The code above works as expected (apart from a small error regarding headers being undefined).

Tested on my machine the httpbin URL responds in around 100ms which means that with a concurrency of 20 it will serve around 200 requests in 1 second (which is what you're seeing as well):

100 ms per request means 10 requests are completed in a second
10 requests per second with a concurrency of 20 means 200 requests in one second

The limit option (aiohttp.TCPConnector) limits the number of concurrent requests and does not have any time dimension.

To see the limit in action try with more values like 10, 20, 50:

# time to complete 1000 requests with different keys
aiohttp.TCPConnector(limit=10): 12.58 seconds 
aiohttp.TCPConnector(limit=20): 6.57 seconds
aiohttp.TCPConnector(limit=50): 3.1 seconds

If you want to use a requests per second limit send batch of requests (20 for example) and use asyncio.sleep(1.0) to pause for a second, then send the next batch and so on.

来源：https://stackoverflow.com/questions/61774125/aiohttp-set-number-of-requests-per-second

标签

python

asynchronous

flask-sqlalchemy

aiohttp