Async List Comprehensions in Python

问题

Is it possible to do a list comprehension in Python asynchronously? I am trying to make many calls to a web api, where the bulk of the wait time is waiting for data to come back, which I would like to do asynchronously. Then after the data is fetched, I want to return the data and store it in a variable. How would I go about doing this?

Below is the non asynchronous code:

api_data = [{data_name: get_data(data_name)} for data_name in data_name_list]

Below is my attempted asynchronous solution:

async def get_api_data(data_name_list):
    api_data = [{data_name: get_data(data_name)} for data_name in data_name_list]
    await asyncio.wait(api_data)
    await conn.close()
    return api_data

data = get_api_data(data_name_list)

The problem is that the api_data gets returned as a coroutine object. Shouldn't it not return as a coroutine object since I close the connection? Is there a better way to turn my non asynchronous code into an asynchronous solution?

回答1:

Yes, it is indeed possible to fire off multiple requests and asynchronously wait for them. Because Python is traditionally a synchronous language, you have to be very careful about what libraries you use with asynchronous Python. Any library that blocks the main thread (such as requests) will break your entire asynchronicity. aiohttp is a common choice for asynchronously making web API calls in Python. What you want is to create a bunch of future objects inside a Python list and await it. A future is an object that represents a value that will eventually resolve to something.

EDIT: Since the function that actually makes the API call is synchronous and blocking and you don't have control over it, you will have to run that function in a separate thread.

import asyncio

async def main():
    loop = asyncio.get_event_loop()
    futures = [asyncio.ensure_future(loop.run_in_executor(None, get_data, data)) for data in data_name_list]
    await asyncio.gather(*futures) # wait for all the future objects to resolve
    # Do something with futures
    # ...


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

Please note that this is Python 3.6 code and might vary significantly from 3.7+ async code

来源：https://stackoverflow.com/questions/57263090/async-list-comprehensions-in-python

标签

python

asynchronous

list-comprehension