How paginate through api response asynchronously with asyncio and aiohttp

问题

I'm trying to make api calls with python asynchronously. I have multiple endpoints in a list and each endpoint will return paginated results. I'm able to set up going though the multiple endpoints asynchronously, however am not able to return the paginated results of each endpoint.

From debugging, I found that the fetch_more() function runs the while loop, but doesn't actually get past the async with session.get(). So basically. The function fetch_more() is intended to get remaining results from the api call for each endpoint, however I find that the same number of results are returned with or without the fetch_more() function. I've tried looking for examples of pagination with asyncio but have not have much luck.

From my understanding, I should not be doing a request inside a while loop, however, I'm not sure a way around that in order to get paginated results.

if __name__ == 'main':

    starter_fun(url, header, endpoints):

starter_func(url, header, endpoints):

    loop = asyncio.get_event_loop() #event loop
    future = asyncio.ensure_future(fetch_all(url, header, endpoints))
    loop.run_until_complete(future) #loop until done

async def fetch_all(url, header, endpoints):

    async with ClientSession() as session:
        for endpoint in endpoints:
           task = asyncio.ensure_future(fetch(url, header, endpoint))
           tasks.append(task)
        res = await asyncio.gather(*tasks) # gather task responses
        return res

async def fetch(url, header, endpoint): 

    total_tasks = []
    async with session.get(url, headers=header, params=params, ssl=False) as response:
        response_json = await response.json()
        data = response_json['key']
       tasks = asyncio.ensure_future(fetch_more(response_json, data, params, header, url, endpoint, session)) //this is where I am getting stuck
        total_tasks.append(tasks)
    return data


//function to get paginated results of api endpoint
async def fetch_more(response_json, data, params, header, url, endpoint, session): //this is where I am getting stuck

    while len(response_json['key']) >= params['limit']:
        params['offset'] = response_json['offset'] + len(response_json['key'])
        async with session.get(url, headers=header, params=params, ssl=False) as response_continued:
            resp_continued_json = await response_continued.json()
            data.extend(resp_continued_json[kebab_to_camel(endpoint)])
   return data

Currently I am getting 1000 results with or without the fetch_more function, however it should be a lot more with the fetch_more. Any idea as to how to approach asynchronously paginating?

来源：https://stackoverflow.com/questions/56248641/how-paginate-through-api-response-asynchronously-with-asyncio-and-aiohttp