virtual-memory | 易学教程

How to avoid running out of memory in high memory usage application? C / C++

阅读更多关于 How to avoid running out of memory in high memory usage application? C / C++

I have written a converter that takes openstreetmap xml files and converts them to a binary runtime rendering format that is typically about 10% of the original size. Input file sizes are typically 3gb and larger. The input files are not loaded into memory all at once, but streamed as points and polys are collected, then a bsp is run on them and the file is output. Recently on larger files it runs out of memory and dies (the one in question has 14million points and 1million polygons). Typically my program is using about 1gb to 1.2 gb of ram when this happens. I've tried increasing virtual

Allocating largest buffer without using swap

阅读更多关于 Allocating largest buffer without using swap

问题 In C/C++ under Linux, I need to allocate a large (several gigabyte) block of memory, in order to store real-time data from a sensor connected to the ethernet port and streaming data at about 110MB/s. I'd like to allocate the largest amount of memory possible, to maximise the length of data sequence that I can store. However, I also need to make sure that there will be no disk-swapping, since the resulting delay and limited bandwidth of disk access causes the sensor's (very limited) buffer to

Zero a large memory mapping with `madvise`

阅读更多关于 Zero a large memory mapping with `madvise`

问题 I have the following problem: I allocate a large chunk of memory (multiple GiB) via mmap with MAP_ANONYMOUS . That chunk holds a large hash map which needs to be zeroed every now and then. Not the entire mapping may be used in each round (not every page is faulted in), so memset is not a good idea - takes too long. What is the best strategy to do this quickly? Will madvise(ptr, length, MADV_DONTNEED); guarantee me that any subsequent accesses provide new empty pages? From the Linux man

Array bounds checks on 64-bit hardware using hardware memory-protection

阅读更多关于 Array bounds checks on 64-bit hardware using hardware memory-protection

问题 I was reading a blog on 64-bit Firefox edition on hacks.mozilla.org. The author states: For asm.js code, the increased address space also lets us use hardware memory protection to safely remove bounds checks from asm.js heap accesses. The gains are pretty dramatic: 8%-17% on the asmjs-apps-*-throughput tests as reported on arewefastyet.com. I was trying to understand how 64-bit hardware have automatic bounds check (assuming compiler does with hardware support) for C/C++. I could not find any

Linux Zero-Copy: Transfer memory pages between two processes with vmsplice

阅读更多关于 Linux Zero-Copy: Transfer memory pages between two processes with vmsplice

问题 Currently, I am trying to understand the value of splice/vmsplice. Regarding the use case of IPC, I stumbled upon the following answer on stackoverflow: https://stackoverflow.com/a/1350550/1305501 Question: How to transfer memory pages from one process to another process using vmsplice without copying data (i.e. zero-copy)? The answer mentioned above claims that it is possible. However, it doesn't contain any source code. If I understand the documentation of vmsplice correctly, the following

When to do or not do INVLPG, MOV to CR3 to minimize TLB flushing

阅读更多关于 When to do or not do INVLPG, MOV to CR3 to minimize TLB flushing

问题 Prologue I am an operating system hobbyist, and my kernel runs on 80486+, and already supports virtual memory. Starting from 80386, the x86 processor family by Intel and various clones thereof has supported virtual memory with paging. It is well known that when the PG bit in CR0 is set, the processor uses virtual address translation. Then, the CR3 register points to the top-level page directory, that is the root for 2-4 levels of page table structures that map the virtual addresses to

A simple “Hello World” needs 10G virtual memory on a 64-bit machine vs 1G at 32-bit?

阅读更多关于 A simple “Hello World” needs 10G virtual memory on a 64-bit machine vs 1G at 32-bit?

问题 This question was migrated from Software Engineering Stack Exchange because it can be answered on Stack Overflow. Migrated 5 years ago . Running a simple Java program on our production machine, I noticed that this program eats up more 10G virt. I know that virtual memory is not that relevant, but at least I would like to understand why this is needed. public class Main { public static void main(String[] args) { System.out.println("Hello World!"); try { Thread.sleep(10000); } catch

How to calculate page table size?

阅读更多关于 How to calculate page table size?

问题 Given : 64 bit virtual byte address, 16 KB pages, 32-bit physical byte address. What is the total size of page table on this machine, assuming that the valid, protection, dirty and use bits take a total of 4 bits and all virtual pages are in use. So far I know the total number of page table entries : 2^64 / 2^14 = 2^50, but am not able to understand how to find the size of each entry. Each entry does contain 4 bits as said in the question, but can the size rest of the entry be found from the

Does the MMU mediate everything between the operating system and physical memory or is it just an address translator?

阅读更多关于 Does the MMU mediate everything between the operating system and physical memory or is it just an address translator?

问题 I'm trying to understand how does an operating system work when we want to assign some value to a particular virtual memory address. My first question concerns whether the MMU handles everything between the CPU and the RAM. Is this true? From what one can read from Wikipedia, I'd say so: A memory management unit (MMU), sometimes called paged memory management unit (PMMU), is a computer hardware component responsible for handling accesses to memory requested by the CPU. If that is the case,

Determine page table size for virtual memory

阅读更多关于 Determine page table size for virtual memory

问题 Consider a virtual memory system with a 38-bit virtual byte address, 1KB pages and 512 MB of physical memory. What is the total size of the page table for each process on this machine, assuming that the valid, protection, dirty and use bits take a total of 4 bits, and that all the virtual pages are in use? (assume that disk addresses are not stored in the page table.) 回答1: Well, if the question is simply "what is the size of the page table?" irrespective of whether it will fit into physical