Should I utilize multiple HttpClients for bulk async GET requests?

问题

I have a scenario where I need to make a large number of GET requests in as little time as possible (think around 1000).

I know generally it's best to keep a single client and reuse it as much as possible:

// Create Single HTTP Client
HttpClient client = new HttpClient();

// Create all tasks
for (int x = 0; x < 1000; x++)
{
    tasks.Add(ProcessURLAsync($"https://someapi.com/request/{x}", client, x));
}

// wait for all tasks to complete.
Task.WaitAll(tasks.ToArray());

...

static async Task<string> ProcessURLAsync(string url, HttpClient client, int x)
{
    var response = await client.GetStringAsync(url);

    ParseResponse(response.Result, x);

    return response;
}

But doing so takes approximately 70 seconds for all requests to complete.

On the other hand, If I create multiple clients beforehand and distribute the requests across them, the it takes about 3 seconds to complete:

// Create arbitrary number of clients
while (clients.Count < maxClients)
{
    clients.Add(new HttpClient());
}

// Create all tasks
for (int x = 0; x < 1000; x++)
{
    tasks.Add(ProcessURLAsync(
        $"https://someapi.com/request/{x}", clients[x % maxClients], x));
}

// Same same code as above

Due to the nature of the data requested, I need to either keep the results sequential or pass along the index associated with the request.

Assuming the API cannot be changed to better format the requested data, and the all requests must complete before moving on, is this solution wise or am I missing a smarter alternative?

(For the sake of brevity I've used an arbitrary number of HttpClient whereas I would create a pool of HttpClient that releases a client once it receives a response and only create a new one when none are free)

回答1:

I would suggest two main changes.

Remove the await so that multiple downloads can occur at the same time.
Set DefaultConnectionLimit to a larger number (e.g. 50).

来源：https://stackoverflow.com/questions/54072800/should-i-utilize-multiple-httpclients-for-bulk-async-get-requests

标签

http

get