Is there a fast way to parse through a large file with regex?

前端 未结 4 1561
面向向阳花
面向向阳花 2021-02-20 17:49

Problem: Very very, large file I need to parse line by line to get 3 values from each line. Everything works but it takes a long time to parse through the whole file. Is it poss

4条回答
  •  囚心锁ツ
    2021-02-20 18:41

    Memory Mapped Files and Task Parallel Library for help.

    1. Create persisted MMF with multiple random access views. Each view corresponds to a particular part of a file
    2. Define parsing method with parameter like IEnumerable, basically to abstract a set of not parsed lines
    3. Create and start one TPL task per one MMF view with Parse(IEnumerable) as a Task action
    4. Each of worker tasks adds a parsed data into the shared queue of BlockingCollection type
    5. An other Task listen to BC (GetConsumingEnumerable()) and processes all data which already parsed by worker Tasks

    See Pipelines pattern on MSDN

    Must say this solution is for .NET Framework >=4

提交回复
热议问题