How to scan through really huge files on disk?

后端 未结 8 2106
北荒
北荒 2020-12-09 18:14

Considering a really huge file(maybe more than 4GB) on disk,I want to scan through this file and calculate the times of a specific binary pattern occurs.

My thought

8条回答
  •  感动是毒
    2020-12-09 18:51

    I would do it with asynchronous reads into a double buffer. So when one buffer has been read from file, start reading the next buffer while scanning the first buffer. This means you do CPU and IO in parallel. Another advantage is that you will always have data around data boundaries. However I don't know if double buffering is possible with memory mapped files.

提交回复
热议问题