Is Rx extensions suitable for reading a file and store to database

问题

I have a really long Excel file wich I read using EPPlus. For each line I test if it meets certain criteria and if so I add the line (an object representing the line) to a collection. When the file is read, I store those objects to the database. Would it be possible to do both things at the same time? My idea is to have a collection of objects that somehow would be consumed by thread that would save the objects to the DB. At the same time the excel reader method would populate the collection... Could this be done using Rx or is there a better method?

Thanks.

回答1:

An alternate answer - based on comments to my first.

Create a function returning an IEnumberable<Records> from EPPlus/Xls - use yield return then convert the seqence to an observable on the threadpool and you've got the Rx way of having a producer/consumer and BlockingCollection.

function IEnumberable<Records> epplusRecords() 
{
  while (...)
     yield return nextRecord;
}
var myRecords = epplusRecords
   .ToObservable(Scheduler.ThreadPool)
   .Where(rec => meetsCritera(rec))
   .Select(rec => newShape(rec))
   .Do(newRec => writeToDb(newRec))
   .ToArray();

回答2:

Your case seems to be of pulling data (IEnumerable) and not pushing data (IObservable/Rx). Hence I would suggest LINQ to objects is something that can be used to model the solution. Something like shown in below code.

publis static IEnumerable<Records> ReadRecords(string excelFile)
{
  //Read from excel file and yield values
}

//use linq operators to do filtering
var filtered = ReadRecords("fileName").Where(r => /*ur condition*/)
foreach(var r in filtered)
   WriteToDb(r);

NOTE: In using IEnumerable you don't create intermediate collections in this case and the whole process looks like a pipeline.

回答3:

It doesn't seem like a good fit, as there's no inherent concurrency, timing, or eventing in the use case.

That said, it may be a use case for plinq. If EEPlus supports concurrent reads. Something like

epplusRecords
   .AsParallel()
   .Where(rec => meetsCritera(rec))
   .Select(rec => newShape(rec))
   .Do(newRec => writeToDb(newRec))
   .ToArray();

来源：https://stackoverflow.com/questions/7164483/is-rx-extensions-suitable-for-reading-a-file-and-store-to-database