Async with huge data streams

Question

We use IEnumerables to return huge datasets from database:

public IEnumerable Read(...)
{
    using(var connection = new SqlConnection(...))
             


        

Answer 1

This will consume a huge amount of resources on server, because all data must be in the list before return. What is the best and easy to use async alternative for IEnumerables to work with large data streams? I would like to avoid storing all the data in memory while processing.

If you don't want to send all data to the client at once, you may consider using Reactive Extensions (Rx) (on the client) and SignalR (on both client and server) to handle this.

SignalR would allow to send data to the client asynchronously. Rx would allow to apply LINQ to the asynchronous sequence of data items as they're arriving on the client. This would however change the whole code model of you client-server application.

Example (a blog post by Samuel Jack):

• Better Together – SignalR and the Rx Framework

Related question (if not a duplicate):

• Using async / await with DataReader ? ( without middle buffers!)