Mongoose Cursor: http bulk request from collection

问题

I have a problem which is relevant to rxJS and bulk HTTP requests from the huge collection (1M+ docs)

I have the following code, with quite simple logic. I push all the docs from the collection to allplayers array and making bulk 20 HTTP requests to API at once (guess you understand why it's limited) So, the code works fine, but I guess it's time to refactor it from this:

        const cursor = players_db.find(query).lean().cursor();
        cursor.on('data', function(player) { allPlayers.push(player); });
        cursor.on('end', function() {
            logger.log('warng',`S,${allPlayers.length}`);
            from(allPlayers).pipe(
                mergeMap(player => getPlayer(player.name, player.realm),20),
            ).subscribe({
                next: player => console.log(`${player.name}@${player.realm}`),
                error: error => console.error(error),
                complete: () => console.timeEnd(`${updatePlayer.name}`),
            });
        });

As for now, I'm using find with cursor with (batchSize), but if I understood this right (via .length), and according to this question: {mongoose cursor batchSize} batchSize is just a way of optimization and it's not return me array of X docs.

So what should I do now and what operator should I choose for rxJS?

For example I could form arrays with necessary length (like 20) and transfer it to rxJS as I use it before. But I guess there should be another way, where I could use rxJS inside this for promise loop

    const players = await players_db.find(query).lean().cursor({batchSize: 10});
    for (let player = await players.next(); player != null; player = await players.next()) {
        //do something via RxJS inside for loop
    }

Also I found this question {Best way to query all documents from a mongodb collection in a reactive way w/out flooding RAM} which also relevant to my problem and I understand the logic, but don't the syntax of it. I also know that cursor variable isn't a doc I cann't do anything useful with it. Or actually I could?

1. rxJS's bufferCount is a quite interesting operator
2. https://gist.github.com/wellcaffeinated/f908094998edf54dc5840c8c3ad734d3 probable solution?

回答1:

So, in the end I found that rxJS isn't needed (but can be used) for this case.

The solution was quite simple and using just MongoCursor:

async function BulkRequest (bulkSize = 10) {
    try {
       let BulkRequest_Array = [];
       const cursor = collection_db.find({}).lean().cursor({batchSize: bulkSize});
       cursor.on('data', async (doc) => {
           BulkRequest_Array.push(/*any function or axios instance*/)
            if (BulkRequest_Array.length >= bulkSize) {
                cursor.pause();
                console.time(`========================`);;
                await Promise.all(BulkRequest_Array);
                BulkRequest_Array.length = 0;
                cursor.resume();
                console.timeEnd(`========================`);
            }
       }
    } catch (e) {
       console.error(e)
    }
}

BulkRequest();

来源：https://stackoverflow.com/questions/56752695/mongoose-cursor-http-bulk-request-from-collection