Async with huge data streams

Question

We use IEnumerables to return huge datasets from database:

public IEnumerable Read(...)
{
    using(var connection = new SqlConnection(...))
             


        

Answer 1

I think Rx is definitely the way to go in this scenario, given an observable sequence is the formal dual to an enumerable one.

As mentioned in a previous answer you could re-write your sequence as an observable from scratch, but there are also a couple of ways to keep writing your iterator blocks but then just unwind them asynchronously.

1) Just convert the enumerable to an observable like so:

using System.Reactive.Linq;
using System.Reactive.Concurrency;

var enumerable = Enumerable.Range(10);
var observable = enumerable.ToObservable();
var subscription = observable.Subscribe(x => Console.WriteLine(x));

This will make your enumerable behave like an observable by pushing its notifications into any downstream observers. In this case, when Subscribe is called, it will synchronously block until all data has been processed. If you want it to be fully asynchronous, you can set it to a different thread, by using:

var observable = enumerable.ToObservable().SubscribeOn(NewThreadScheduler.Default);

Now the unwinding of the enumerable will be done in a new thread and the subscribe method will return immediately.

2) Unwind the enumerable using another asynchronous event source:

var enumerable = Enumerable.Range(10);
var observable = Observable.Timer(TimeSpan.Zero, TimeSpan.FromSeconds(1))
                           .Zip(enumerable, (t, x) => x);
var subscription = observable.Subscribe(x => Console.WriteLine(x));

In this case, I've setup a timer to fire every second and whenever it fires it moves the iterator forward. Now the timer could be easily replaced by any event source to control exactly when the iterator moves forward.

I find myself enjoying the syntax and semantics of iterator blocks (e.g. what happens with try/finally blocks and dispose), so I use these designs occasionally even when designing asynchronous operations.