Sample of virtual memory using to work with Big Arrays

问题

I am dumb in C++/C# and WinAPI. Can someone share with me useful links or show simple sample of using virtual memory for working with big arrays( on C++ or on C#).

Thanks in advance.

回答1:

I think what you are after is using Memory Mapped Files which allows you to use the content of a file "as if" it was loaded into memory when in actuality it exists mostly on disk.

Take a look at "Creating a File View" on MSDN for windows, or man mmap for Linux.

回答2:

Virtual memory is not a property of your programming language. You cannot ever see virtual memory from a C++ program or a C# program. You are certainly not, on any modern (<20 years) hardware or operating system afforded the ability to control it directly.

But you're always, always using it.

Try the following program on your local machine:

#include <iostream>
int main(int, const char*[])
{
    const std::size_t one_megabyte = 1024 * 1024;
    char* gigantic_array[5*1024];   // 5GB in blocks of 1MB

    std::size_t counter = 0;
    while (true) {
        // Allocate and use the memory (prevents OS cheating)
        gigantic_array[counter] = new char[one_megabyte];
        for (std::size_t i = 0; i < one_megabyte; ++i)
             gigantic_array[counter][i] = 'F';

        ++counter;
        std::cout << "Allocated " << counter / 1024. << "GB of memory." << std::endl;
    }

    return 0;
}

Before you run this program, run top in a separate shell. Now run it. You're going to very quickly see the program you just ran race to the top of the list. Numbers will be scrolling by...

Now soon, you'll probably notice a pause. On my laptop, that happened at 1.3 GB, more or less. At this pause, you're out of physical memory on your machine and virtual memory starts swapping things out to disk. Again, you didn't just turn on virtual memory, you only made its job harder by actually making it evict things from RAM.

Now wait. How much RAM do you have on your machine? I have 4GB here. I killed the program by hand after it had allocated 5.5GB of memory. Again, you never "turn it on". It's a basic function of the operating system that your program can't easily tell how much physical memory it's using.

So I hope I've convinced you to look at virtual memory a little differently. Hopefully that helps you get on with your big array problem, as well.

回答3:

I know iam necroing this Q but it has no selected answer. Since virtual Memory is always and everywhere, you cant forgo using it. You can however control it more directly see the msdn for c# memorypages, or you can create a file and use it as buffer ("fileview"), check for exception and off you go, (with the help of a little serialization of your objects)

or, the more elegant way in my opinion: unsafe Code =) create an array each cell in that array holds a ref to an array that holds that sector of your big-picture array.

(With a little system magic you can calculate a good size for your part array -> get the size of 1 chunk of your filesystem, that way no space is lost.) if you trust your OS you can just take a typical base 2 number for size.

来源：https://stackoverflow.com/questions/8279197/sample-of-virtual-memory-using-to-work-with-big-arrays

标签

c++

virtual-memory