Best way to query all documents from a mongodb collection in a reactive way w/out flooding RAM

问题

I want to query all the documents in a collection in a reactive way. The collection.find() method of the mongodb nodejs driver returns a cursor that fires events for each document found in the collection. So I made this:

function giant_query = (db) => {
    var req = db.collection('mycollection').find({});   
    return Rx.Observable.merge(Rx.Observable.fromEvent(req, 'data'),
                               Rx.Observable.fromEvent(req, 'end'),
                               Rx.Observable.fromEvent(req, 'close'),
                               Rx.Observable.fromEvent(req, 'readable'));
}

It will do what I want: fire for each document, so I can treat then in a reactive way, like this:

Rx.Observable.of('').flatMap(giant_query).do(some_function).subscribe()

I could query the documents in packets of tens, but then I'd have to keep track of an index number for each time the observable stream is fired, and I'd have to make an observable loop which I do not know if it's possible or the right way to do it.

The problem with this cursor is that I don't think it does things in packets. It'll probably fire all the events in a short period of time, therefore flooding my RAM. Even if I buffer some events in packets using Observable's buffer, the events and events data (the documents) are going to be waiting on RAM to be manipulated.

What's the best way to deal with it n a reactive way?

回答1:

I'm not an expert on mongodb, but based on the examples I've seen, this is a pattern I would try.

I've omitted the events other than data, since throttling that one seems to be the main concern.

var cursor = db.collection('mycollection').find({});  

const cursorNext = new Rx.BehaviourSubject('next');  // signal first batch then wait
const nextBatch = () => {
  if(cursor.hasNext()) {
    cursorNext.next('next');
  }
});

cursorNext
  .switchMap(() =>                            // wait for cursorNext to signal
     Rx.Observable.fromPromise(cursor.next())  // get a single doc
       .repeat()                               // get another
       .takeWhile(() => cursor.hasNext() )     // stop taking if out of data
       .take(batchSize)                        // until full batch
       .toArray()                              // combine into a single emit
  )
  .map(docsBatch => {
    // do something with the batch
    // return docsBatch or modified doscBatch
  })
  ... // other operators?
  .subscribe(x => {
    ...
    nextBatch();
  });

I'm trying to put together a test of this Rx flow without mongodb, in the meantime this might give you some ideas.

来源：https://stackoverflow.com/questions/48362449/best-way-to-query-all-documents-from-a-mongodb-collection-in-a-reactive-way-w-ou

标签

node.js

mongodb

rxjs

reactivex