How to scan through really huge files on disk?

后端 未结 8 2094
北荒
北荒 2020-12-09 18:14

Considering a really huge file(maybe more than 4GB) on disk,I want to scan through this file and calculate the times of a specific binary pattern occurs.

My thought

8条回答
  •  执念已碎
    2020-12-09 18:40

    Using a memory mapped file has the additional benefit of avoiding a copy from the filesystem cache memory to the (managed) application memory if you use a read-only view (although you have to use byte* pointers then to access the memory). And instead of creating many threads use one thread to sequentially scan through the file using for example 100MB memory mapped views into the file (don't map the entire file into the process address space at once).

提交回复
热议问题