How to use the bluebird concurrency option for the map function

问题

I am trying to use bluebird's map function with the built-in concurrency control.

I want to retrieve a list of names, then make a number of POST requests for each name. For example, I want to make a request for each name for each day of the week. However, I need to throttle the number of concurrent POST requests because the intended server has rate limits.

function getNames() {
    //Open mongodb connection
    //Get collection and array of names
    //return array of names in a promise 
}

function createDatesArray() {
    //Create an array of rates
    //return array of dates in a promise
    //Ex. return Promise.resolve(datesArray);
}

getNames().map(function (name) {
    return createDatesArray().map(function (date) {
        return requestData(date, name);
    }, {concurrency: 5});
}).then(function () {
//do something
});

Is this the correct way to use bluebird's concurrency?

The documentation link is here bluebird documentation.

回答1:

Short answer: yes , this will limit the number of requests to 5.

Caveat: keep in mind you might still be susceptible to more limits, such as the HTTP client's, or any other pools, modules and services you might be using.

Also, a Mongo connection is meant to be used as a persistent one, so you should probably only open one and then use it rather than open and close one every time.

If createDatesArray does not do anything asynchronous, you do not have to Promise.resolve it, you can instead use the static variant of map as Promise.map(datesArray, function(date){ ... }) etc. I would also not nest. Assuming createDatesArray is indeed async:

Promise.join(getNames(), createDatesArray(), function(names, dates){
    var tasks = [];
    names.forEach(function(name ){ // create Cartesian product of names * dates
        dates.forEach(function(date){
             tasks.push(function(){ return requestData(name, date); });
        });
    });
    return Promise.map(tasks, function(job){ return job(); } , { concurrency: 5} );
}), then(function(results){
     // do whatever
});

来源：https://stackoverflow.com/questions/24745455/how-to-use-the-bluebird-concurrency-option-for-the-map-function

标签

concurrency

promise

bluebird