mmap

Speeding up file I/O: mmap() vs. read()

南笙酒味 提交于 2019-11-28 05:14:35
I have a Linux application that reads 150-200 files (4-10GB) in parallel. Each file is read in turn in small, variably sized blocks, typically less than 2K each. I currently need to maintain over 200 MB/s read rate combined from the set of files. The disks handle this just fine. There is a projected requirement of over 1 GB/s (which is out of the disk's reach at the moment). We have implemented two different read systems both make heavy use of posix_advise : first is a mmap ed read in which we map the entirety of the data set and read on demand. The second is a read() / seek() based system.

Python mmap - slow access to end of files [with test code]

你离开我真会死。 提交于 2019-11-28 04:19:04
问题 I posted a similar question a few days ago but without any code, now I created a test code in hopes of getting some help. Code is at the bottom. I got some dataset where I have a bunch of large files (~100) and I want to extract specific lines from those files very efficiently (both in memory and in speed). My code gets a list of relevant files, the code opens each file with [line 1], then maps the file to memory with [line 2], also, for each file I receives a list of indices and going over

How does numpy handle mmap's over npz files?

大兔子大兔子 提交于 2019-11-28 03:40:43
问题 I have a case where I would like to open a compressed numpy file using mmap mode, but can't seem to find any documentation about how it will work under the covers. For example, will it decompress the archive in memory and then mmap it? Will it decompress on the fly? The documentation is absent for that configuration. 回答1: The short answer, based on looking at the code, is that archiving and compression, whether using np.savez or gzip , is not compatible with accessing files in mmap_mode . It

Write-only mapping a O_WRONLY opened file supposed to work?

你。 提交于 2019-11-28 03:30:29
问题 Is mmap() supposed to be able to create a write-only mapping of a O_WRONLY opened file? I am asking because following fails on a Linux 4.0.4 x86-64 system ( strace log): mkdir("test", 0700) = 0 open("test/foo", O_WRONLY|O_CREAT, 0666) = 3 ftruncate(3, 11) = 0 mmap(NULL, 11, PROT_WRITE, MAP_SHARED, 3, 0) = -1 EACCES (Permission denied) The errno equals EACCESS . Replacing the open-flag O_WRONLY with O_RDWR yields a successful mapping. The Linux mmap man page documents the errno as: EACCES A

Linux shared memory: shmget() vs mmap()?

一曲冷凌霜 提交于 2019-11-28 03:01:51
In this thread the OP is suggested to use mmap() instead of shmget() to get shared memory in Linux. I visited this page and this page to get some documentation, but the second one gives an obscure example regarding mmap() . Being almost a newbie, and needing to share some information (in text form) between two processes, should I use the shmget() method or mmap() ? And why? Sergey L. Both methods are viable. mmap method is a little bit more restrictive then shmget , but easier to use. shmget is the old System V shared memory model and has the widest support. mmap / shm_open is the new POSIX

mmap of /dev/mem fails with invalid argument for virt_to_phys address, but address is page aligned

白昼怎懂夜的黑 提交于 2019-11-28 02:17:53
For some reason my mmap failed with an Invalid argument message even though my offset is page aligned. Page size is 4096 bytes. Also CONFIG_STRICT_DEVMEM is disabled, i.e. I can access memory above 1MB. Here is my code: void *mmap64; off_t offset = 0x000000d9fcc000; int memFd = open("/dev/mem", O_RDWR); if (-1 == memFd) perror("Error "); mmap64 = mmap(0, getpagesize(), PROT_WRITE | PROT_READ, MAP_SHARED, memFd, offset); if (MAP_FAILED == mmap64) { perror("Error "); return -1; } Can someone explain why this is happening? EDIT Here is the strace of my code execve("./to_phys_test", ["./to_phys

How to write mmap input memory to O_DIRECT output file?

喜你入骨 提交于 2019-11-28 02:10:24
问题 why doesn't following pseudo-code work (O_DIRECT results in EFAULT) in_fd = open("/dev/mem"); in_mmap = mmap(in_fd); out_fd = open("/tmp/file", O_DIRECT); write(out_fd, in_mmap, PAGE_SIZE); while following does (no O_DIRECT) in_fd = open("/dev/mem"); in_mmap = mmap(in_fd); out_fd = open("/tmp/file"); write(out_fd, in_mmap, PAGE_SIZE); I guess it's something with virtual kernel pages to virtual user pages, which cannot be translated in the write call? Best regards, Friedrich 回答1: Using mmap()

mmap on /proc/pid/mem

▼魔方 西西 提交于 2019-11-28 01:04:15
Has anybody succeeded in mmap'ing a /proc/pid/mem file with Linux kernel 2.6? I am getting an ENODEV (No such device) error. My call looks like this: char * map = mmap(NULL, PAGE_SIZE, PROT_READ, MAP_SHARED, mem_fd, offset); And I have verified by looking at the /proc/pid/maps file while debugging that, when execution reaches this call, offset has the value of the top of the stack minus PAGE_SIZE. I have also verified with ptrace that mmap is setting errno to ENODEV. ephemient See proc_mem_operations in /usr/src/linux/fs/proc/base.c : /proc/.../mem does not support mmap . 来源: https:/

Does madvise(___, ___, MADV_DONTNEED) instruct the OS to lazily write to disk?

有些话、适合烂在心里 提交于 2019-11-28 00:37:29
问题 Hypothetically, suppose I want to perform sequential writing to a potentially very large file. If I mmap() a gigantic region and madvise(MADV_SEQUENTIAL) on that entire region, then I can write to the memory in a relatively efficient manner. This I have gotten to work just fine. Now, in order to free up various OS resources as I am writing, I occasionally perform a munmap() on small chunks of memory that have already been written to. My concern is that munmap() and msync()will block my thread

Why does mmap() fail with ENOMEM on a 1TB sparse file?

二次信任 提交于 2019-11-28 00:05:19
问题 I've been working with large sparse files on openSUSE 11.2 x86_64. When I try to mmap() a 1TB sparse file, it fails with ENOMEM. I would have thought that the 64 bit address space would be adequate to map in a terabyte, but it seems not. Experimenting further, a 1GB file works fine, but a 2GB file (and anything bigger) fails. I'm guessing there might be a setting somewhere to tweak, but an extensive search turns up nothing. Here's some sample code that shows the problem - any clues? #include