How to get a unified onFinish from separate streams (some created from within original stream)

问题

I have a stream process like this:

Incomming file via HTTP (original stream)
  -> Check if zipfile
    - Yes -> push through an unzip2-stream
    - No -> push to S3

When the unzip2-stream finds zip-entries, these are pushed through the same chain of streams, i.e.

Incomming file entry from zip file ("child" stream)
  -> Check if zipfile
    - Yes -> push through an unzip2-stream
    - No -> push to S3

Thanks to https://stackoverflow.com/users/3580261/eljefedelrodeodeljefe I managed to solve the main problem after this conversation: How to redirect a stream to other stream depending on data in first chunk?

The problem with creating new "child" streams for every zip entry is that these will have no connection to the original stream, so I cannot get a unified onFinish for all the streams.

I don't want to send a 202 of to the sender before I have processed (unzipped and sent to S3) every file. How can I accomplish this?

I'm thinking that I might need some kind of control object which awaits onFinish for all child streams and forces the process to dwell in the original onFinish event until all files are processed. Would this be overkill? Is there a simpler solution?

回答1:

I ended up making a separate counter for the streams. There is probably a better solution, but this works.

I send the counter object as an argument to the first call to my saveFile() function. The counter is passed along to the unzip stream so it can be passed to saveFile for every file entry.

Just before a stream is started (i.e. piped) I call streamCounter.streamStarted().
In the last onFinish in the pipe chain I call streamCounter.streamFinished()
In the event of a stream going bad I call streamCounter.streamFailed()

Just before I send the 202 in the form post route I wait for streamCounter.streamPromise to resolve.

I'm not very proud of the setInterval solution. It'd probably be better with some kind of event emitting.

module.exports.streamCounter = function() {
  let streamCount = 0;
  let isStarted = false;
  let errors = [];

  this.streamStarted = function(options) {
    isStarted = true;

    streamCount += 1;
    log.debug(`Stream started for ${options.filename}. New streamCount: ${streamCount}`);
  };

  this.streamFinished = function(options) {
    streamCount -= 1;
    log.debug(`Finished stream for ${options.filename}. New streamCount: ${streamCount}`);
  };

  this.streamFailed = function(err) {
    streamCount -= 1;
    errors.push(err);
    log.debug(`Failed stream because (${err.message}). New streamCount: ${streamCount}`);
  };

  this.streamPromise = new Promise(function(resolve, reject) {
    let interval = setInterval(function() {
      if(isStarted && streamCount === 0) {
        clearInterval(interval);

        if(errors.length === 0) {
          log.debug('StreamCounter back on 0. Resolving streamPromise');
          resolve();
        } else {
          log.debug('StreamCounter back on 0. Errors encountered.. Rejecting streamPromise');
          reject(errors[errors.length-1]);
        }
      }
    }, 100);
  });
};

At first I tried this concept with a promise array and waited for Promise.all() before sending status 202. But Promise.all() only works with static arrays as far as I can tell. My "streamCount" is changing during the streaming so I needed a more dynamic "Promise.all".

来源：https://stackoverflow.com/questions/33390967/how-to-get-a-unified-onfinish-from-separate-streams-some-created-from-within-or

标签

node.js

stream

unzip