implement a queueing system with a fixed stack in RxJs

问题

Say I want a queue where only 3 items are being processed asynchronously at any one time, how do I go about this?

This is what I mean: If I have a collection of items to upload to the backend i.e. upload some artefacts to cloud storage and consequently create/update a doc to reflect the url of each artefact and I don't want to:

1. Async/Await each upload operation before the next - as this would be slow
2. Send everything off at the same time - this could lead to write hot-spotting or rate limiting
3. Do a promise.race - this eventually leads to (2)
4. Do a promise.all - the process becomes slow if there's one long running upload.

And what I want to do is:

1. Have a queue of all the uploads, say with an RxJs create method e.g. from(array-of-upload-items) with a stack of 3 items being processed at any one time.
2. And when one item leaves the stack i.e. completes, we add a new item to the queue
3. Ensure that at any one point, there are always 3 items in the stack being processed until there are no more items in the queue waiting to be put in the stack.

How would I go about this using RxJs?

EDITED: 27/June/2020

Here's what I am thinking of:

 const rxQueue = from(filesArray) // this is the list of files to upload say like 25 files or so

      rxQueue
        .pipe(
          mergeMap((item) =>
            of(item).pipe(
              tap(async (item) => {
                  await Promise.race([
                      processUpload(item[0]),
                      processUpload(item[1]),
                      processUpload(item[2]),
                  ])
              }),
            ),
            3
          ),
        )
        .subscribe()

The goal is to ensure that at any point, 3 files are being processed (uploaded) so much so that if one file upload process ends, another file is added to keep the stack at 3 upload processes. Same way, if 2 file uploads end at the same time, 2 new files are added to the stack and so on until all the files in the file array are uploaded.

回答1:

I think you could try this:

from(filesArray)
  .pipe(
    mergeMap(file => service.uploadFile(file), 3)
  )

This assumes that service.uploadFile returns a promise or an observable.

Let's say you have 5 files, then 3 observables will be created from the first 3 and when one of them completes, the 4th file will be taken and a new observable will be created from it and so on.

回答2:

Use Subject as queue and with mergeMap there's a concurrency param that you can limit max number of concurrency

const queue=new Subject()
queque.asObservable().pipe(mergeMap(item=>httpCall(item),3)
queue.next(item)

来源：https://stackoverflow.com/questions/62608353/implement-a-queueing-system-with-a-fixed-stack-in-rxjs