Python asyncio.semaphore in async-await function

问题:

I am trying to teach myself Python's async functionality. To do so I have built an async web scraper. I would like to limit the total number of connections I have open at once to be a good citizen on servers. I know that semaphore's are a good solution, and the asyncio library has a semaphore class built in. My issue is that Python complains when using yield from in an async function as you are combining yield and await syntax. Below is the exact syntax I am using...

import asyncio import aiohttp  sema = asyncio.BoundedSemaphore(5)  async def get_page_text(url):     with (yield from sema):         try:             resp = await aiohttp.request('GET', url)             if resp.status == 200:                 ret_val = await resp.text()         except:             raise ValueError         finally:             await resp.release()     return ret_val 

Raising this Exception:

File "<ipython-input-3-9b9bdb963407>", line 14     with (yield from sema):          ^ SyntaxError: 'yield from' inside async function 

Some possible solution I can think of...

1. Just use the @asyncio.coroutine decorator
2. Use threading.Semaphore? This seems like it may cause other issues
3. Try this in the beta of Python 3.6 for this reason.

I am very new to Python's async functionality so I could be missing something obvious.

回答1:

You can use the async with statement to get an asynchronous context manager:

#!/usr/local/bin/python3.5 import asyncio from aiohttp import ClientSession   sema = asyncio.BoundedSemaphore(5)  async def hello(url):     async with ClientSession() as session:         async with sema, session.get(url) as response:             response = await response.read()             print(response)  loop = asyncio.get_event_loop() loop.run_until_complete(hello("http://httpbin.org/headers")) 

Example taken from here. The page is also a good primer for asyncio and aiohttp in general.

回答2:

OK, so this is really silly but I just replaces yield from with await in the semaphore context manager and it is working perfectly.

sema = asyncio.BoundedSemaphore(5)  async def get_page_text(url):     with (await sema):         try:             resp = await aiohttp.request('GET', url)             if resp.status == 200:                 ret_val = await resp.text()         except:             raise ValueError         finally:             await resp.release()     return ret_val 

文章来源: Python asyncio.semaphore in async-await function