Reactive pipeline - how to control parallelism?

Question

I\'m building a straightforward processing pipeline where an item is fetched as an input, it is being operated by multiple processors in a sequential manner and finally it i

Answer 1

You might need to rearrange the code you posted but this would be one way to do it:

var eventLoopScheduler = new EventLoopScheduler ();
(from semaphore in Observable.Return(new Semaphore(2,2))
 from input in GetInputObs()
 from getAccess in Observable.Start(() => semaphore.WaitOne(),eventLoopScheduler)
 from output in ProcessInputOnPipeline(input)
        .SubscribeOn(Scheduler.Default) 
        .Finally(() => semaphore.Release())
 select output)
 .Subscribe(x => Console.WriteLine(x), ex => {});

I've modelled your pipeline as 1 Observable (which in reality would be composed of several smaller observables chained together)

Key thing is to make sure that the semaphore gets released no matter how the pipeline terminates (Empty/Error) otherwise the stream might hang, and so a Finally() is used call Release() on the semaphore. (Might be worth considering adding a Timeout as well on the pipeline observable if it is liable to never OnComplete()/OnError().

Edit:

As per below comments, I've added some scheduling around the semaphore access so that we don't block whoever is pushing these inputs into our stream. I've used an EventLoopScheduler so that all requests for semaphore access will queue up and be executed on 1 thread.

Edit: I do prefer Paul's answer though - simple, less scheduling, less synchronisation (merge uses a queue internally).