NodeJS: How to use async.js to process a list of items in database

问题

I have this problem: I get a list of items from an API and I have to save them in database. For each of them I have to display some feedback, like " - item name. Saving in DB.." and after saving it show something like "Sucess! Next...".

I'm using async link to iterate through the items in the array and save in DB. The problem is that the messages don't sync. It shows a bunch of the item name message and after a bunch of the success message. My code below.

var saveInDB = function saveInDB(item, callback) {

    console.log(' - ' + item.market_name + ' .......');

    // Simulating some delay
    setTimeout(function () {
        console.log('Success! Next....');
        callback(null);
    }, 3000);
}


util.getJsonFromUrl(API_URL, function (err, data) {

    if (err || !data || !data.results || data.results.length == 0)
    {
        console.log('BAD');
    }
    else
    {
        console.log('Items downloaded...');

        async.each(
            data.results,
            function (item, callback) {

                saveInDB(item, callback);

            },
            function (err) {
                if (err) {
                    console.log('ERROR');
                }
                else {
                    console.log('Success!');
                }
            }
        );
    }
});

Yes, I have tried to use .series but I just cant get it to work. I was looking for examples and I still don't get how to simply pass the item as in .each.

Can you help me? Thanks in advance for any help.

回答1:

Based on the async documentation using async.each

Applies the function iteratee to each item in arr, in parallel.

And look they even warned that this does not preserve the order since its parallel:

Note, that since this function applies iteratee to each item in parallel, there is no guarantee that the iteratee functions will complete in order.

If I understand correctly, you wish your items (assuming you have A,B,C) to be executed in the following order:

//This uses async.eachSeries
For each - A
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > A ....... Done.
Success! Next....
For each - B
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > B ....... Done.
Success! Next....
For each - C
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > C ....... Done.
Success! Next....

And not in the following order (or maybe different, its parallel, no order guaranteed):

//uses async.each 
For each - A
SaveInDB called.
Timeout set. Waiting to timeout ...
For each - B
SaveInDB called.
Timeout set. Waiting to timeout ...
For each - C
SaveInDB called.
Timeout set. Waiting to timeout ...
Processing DB transaction for > A ....... Done.
Success! Next....
Processing DB transaction for > B ....... Done.
Success! Next....
Processing DB transaction for > C ....... Done.
Success! Next....

So change your method from async.each to async.eachSeries as well as to make sure that you don't get confused see below saveInDB function.

function saveInDB(item, callback) {
   //this will not wait for timeout. 
   console.log('SaveInDB called.');

   // Simulating some delay
   setTimeout(function () {
     //everything in this block waits for timeout. 
     console.log('Processing DB transaction for > ' + item + ' ....... Done.');
     console.log('Success! Next....');
     callback(null)
   }, 5000);

   //this will not wait for timeout
   console.log('Timeout set. Waiting to timeout ...');
 }

I would give it a go with real database, rather than simulation with timeout. Be careful with timeout, see below from documentation:

It is important to note that your callback will probably not be called in exactly delay milliseconds - Node.js makes no guarantees about the exact timing of when the callback will fire, nor of the ordering things will fire in. The callback will be called as close as possible to the time specified.

回答2:

    async.each(data.results, function (result, cb_results) {
       //try your here 
       cb_results();
    }, function (err) {
        if (err) { throw err; }

    });

Example:

    var returnData = [];
    async.each(data.results, function (result, cb_results) {
       var myJson = {};
       myJson.id = result.id;
       ................
       returnData.push(myJson);
       cb_results();
    }, function (err) {
        if (err) { throw err; }
        callback(returnData);
    });

来源：https://stackoverflow.com/questions/35394416/nodejs-how-to-use-async-js-to-process-a-list-of-items-in-database

标签

javascript

node.js

asynchronous

async.js