memory-mapped-files | 易学教程

How to scan through really huge files on disk?

阅读更多关于 How to scan through really huge files on disk?

问题 Considering a really huge file(maybe more than 4GB) on disk,I want to scan through this file and calculate the times of a specific binary pattern occurs. My thought is: Use memory-mapped file(CreateFileMap or boost mapped_file) to load the file to the virtual memory. For each 100MB mapped-memory,create one thread to scan and calculate the result. Is this feasible?Are there any better method to do so? Update : Memory-mapped file would be a good choice,for scaning through a 1.6GB file could be

How to dynamically expand a Memory Mapped File

阅读更多关于 How to dynamically expand a Memory Mapped File

问题 I've used C# to solve the following requirement.. - create an app the can receive a lot of data fast - you must be able to analyse the received data while more are incoming. - use as little CPU and disk as possible My idea for an algorithm was.. SIZE = 10MB Create a mmf with the size of SIZE On data recived: if data can't fit mmf: increase mmf.size by SIZE write the data to mmf -> The size on the disc are increased in chuncks of 10MB when the previous "room/space" are used. How is the

Can multiple threads see writes on a direct mapped ByteBuffer in Java?

阅读更多关于 Can multiple threads see writes on a direct mapped ByteBuffer in Java?

问题 I'm working on something that uses ByteBuffers built from memory-mapped files (via FileChannel.map()) as well as in-memory direct ByteBuffers. I am trying to understand the concurrency and memory model constraints. I have read all of the relevant Javadoc (and source) for things like FileChannel, ByteBuffer, MappedByteBuffer, etc. It seems clear that a particular ByteBuffer (and relevant subclasses) has a bunch of fields and the state is not protected from a memory model point of view. So, you

How big can a memory-mapped file be?

阅读更多关于 How big can a memory-mapped file be?

What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits? You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's

How does mmap improve file reading speed?

阅读更多关于 How does mmap improve file reading speed?

问题 Assuming the address space can cover the file, it appears to me that mmap simply allocates a chunk of memory as large as the file about to be read, and creates a 1-to-1 relation between their corresponding blocks. However, why doing so speeds up file reading? It seems that in order to actually get the contents of the file, you still have to go to the disk, and read all the bytes on it. What difference does it make, compared to malloc'ing the same size of memory and manually read the entire

How big can a memory-mapped file be?

阅读更多关于 How big can a memory-mapped file be?

问题 What limits the size of a memory-mapped file? I know it can\'t be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits? 回答1: You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple

How to unmap a file from memory mapped using FileChannel in java?

阅读更多关于 How to unmap a file from memory mapped using FileChannel in java?

问题 I am mapping a file(\"sample.txt\") to memory using FileChannel.map() and then closing the channel using fc.close() . After this when I write to the file using FileOutputStream, I am getting the following error: java.io.FileNotFoundException: sample.txt (The requested operation cannot be per formed on a file with a user-mapped section open) File f = new File(\"sample.txt\"); RandomAccessFile raf = new RandomAccessFile(f,\"rw\"); FileChannel fc = raf.getChannel(); MappedByteBuffer mbf = fc.map