Processing large text file in C#

Question

I have 4GB+ text files (csv format) and I want to process this file using linq in c#.

I run complex linq query after load csv and convert to class?

but file

Answer 1

If you are using .NET 4.0 you could use Clay and then write a method that returns an IEnumerable line for line and that makes code like the below possible

from record in GetRecords("myFile.csv",new []{"Foo","Bar"},new[]{","})
where record.Foo == "Baz"
select new {MyRealBar = int.Parse(record.Bar)

the method to project the CSV into a sequence of Clay objects could be created like:

 private IEnumerable GetRecords(
                    string filePath,
                    IEnumerable columnNames, 
                    string[] delimiter){
            if (!File.Exists(filePath))
                yield break;
            var columns = columnNames.ToArray();
            dynamic New = new ClayFactory();
            using (var streamReader = new StreamReader(filePath)){
                var columnLength = columns.Length;
                string line;
                while ((line = streamReader.ReadLine()) != null){
                    var record = New.Record();
                    var fields = line.Split(delimiter, StringSplitOptions.None);
                    if(fields.Length != columnLength)
                        throw new InvalidOperationException(
                                 "fields count does not match column count");
                    for(int i = 0;i