mmap

Speeding up file I/O: mmap() vs. read()

由 ∥☆過路亽．° 提交于 2019-11-27 00:56:59
问题 I have a Linux application that reads 150-200 files (4-10GB) in parallel. Each file is read in turn in small, variably sized blocks, typically less than 2K each. I currently need to maintain over 200 MB/s read rate combined from the set of files. The disks handle this just fine. There is a projected requirement of over 1 GB/s (which is out of the disk's reach at the moment). We have implemented two different read systems both make heavy use of posix_advise : first is a mmap ed read in which

numpy vs. multiprocessing and mmap

由 谁说胖子不能爱 提交于 2019-11-27 00:56:19
问题 I am using Python's multiprocessing module to process large numpy arrays in parallel. The arrays are memory-mapped using numpy.load(mmap_mode='r') in the master process. After that, multiprocessing.Pool() forks the process (I presume). Everything seems to work fine, except I am getting lines like: AttributeError("'NoneType' object has no attribute 'tell'",) in `<bound method memmap.__del__ of memmap([ 0.57735026, 0.57735026, 0.57735026, 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ], dtype

How to access(if possible) kernel space from user space?

由 廉价感情. 提交于 2019-11-27 00:23:34
How exactly is user memory and kernels memory differentiated inside the Linux kernel (in terms of giving security to kernel space)? What are the different ways I can write in kernel address space from user space? One way I know is through a system call . There are multiple system calls we can use, but at the end they are all system calls. Even in system calls, we send a data to kernel space, where it(driver or respective module) calls functions like copy_from_user() to copy data from user space to kernel space. Here we exactly are not writing into address space. we are just passing a user

Setting up Laravel on a Mac php artisan migrate error: No such file or directory [duplicate]

由 烈酒焚心 提交于 2019-11-27 00:09:09
This question already has an answer here: MySQL connection not working: 2002 No such file or directory 19 answers Pulled a perfectly working laravel project from a git into a mac running MAMP. Project ran perfectly on a linux machine. composer install php artisan migrate, got the following error: [PDOException] SQLSTATE[HY000] [2002] No such file or directory NB: php -v is 5.5 and mysql -v is 5.5 from the terminal Here is part of my config/database.php 'mysql' => array( 'driver' => 'mysql', 'host' => 'localhost', 'database' => 'essays', 'username' => 'root', 'password' => 'root', 'charset' =>

How to have a checkpoint file using mmap which is only synced to disk manually

由 孤者浪人 提交于 2019-11-26 23:29:27
问题 I need the fastest way to periodically sync file with memory. What I think I would like is to have an mmap'd file, which is only sync'd to disk manually. I'm not sure how to prevent any automatic syncing from happening. The file cannot be modified except at the times I manually specify. The point is to have a checkpoint file which keeps a snapshot of the state in memory. I would like to avoid copying as much as possible, since this will be need to called fairly frequently and speed is

mmap with /dev/zero

由 北城余情 提交于 2019-11-26 23:09:08
问题 Say I allocate a big memory (40MB) with mmap using /dev/zero as follows. fd = open("/dev/zero", O_RDWR); a = mmap (0, 4096e4, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FILE, fd, 0); What I understand is that the kernel will initialize memories to zero as the pages are brought into the physical memory (I suppose the modern Linux kernels use Demand paging ). So for example, when the first page is touched and therefore brought into the physical memory, kernel will initialize all of its 4096

How big can a memory-mapped file be?

由 最后都变了- 提交于 2019-11-26 22:24:02
What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits? You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's

mmap of /dev/mem fails with invalid argument for virt_to_phys address, but address is page aligned

由 社会主义新天地 提交于 2019-11-26 22:12:25
问题 For some reason my mmap failed with an Invalid argument message even though my offset is page aligned. Page size is 4096 bytes. Also CONFIG_STRICT_DEVMEM is disabled, i.e. I can access memory above 1MB. Here is my code: void *mmap64; off_t offset = 0x000000d9fcc000; int memFd = open("/dev/mem", O_RDWR); if (-1 == memFd) perror("Error "); mmap64 = mmap(0, getpagesize(), PROT_WRITE | PROT_READ, MAP_SHARED, memFd, offset); if (MAP_FAILED == mmap64) { perror("Error "); return -1; } Can someone

Driving Beaglebone GPIO through /dev/mem

由 假装没事ソ 提交于 2019-11-26 18:55:55
问题 I'm trying to write a C program for blinking a LED on the Beaglebone. I know I can use the sysfs way...but I'd like to see if it is possible to get the same result mapping the physical address space with /dev/mem. I have a header file, beaglebone_gpio.h wit the following contents: #ifndef _BEAGLEBONE_GPIO_H_ #define _BEAGLEBONE_GPIO_H_ #define GPIO1_START_ADDR 0x4804C000 #define GPIO1_END_ADDR 0x4804DFFF #define GPIO1_SIZE (GPIO1_END_ADDR - GPIO1_START_ADDR) #define GPIO_OE 0x134 #define GPIO

How does mmap improve file reading speed?

由 断了今生、忘了曾经 提交于 2019-11-26 16:56:47
问题 Assuming the address space can cover the file, it appears to me that mmap simply allocates a chunk of memory as large as the file about to be read, and creates a 1-to-1 relation between their corresponding blocks. However, why doing so speeds up file reading? It seems that in order to actually get the contents of the file, you still have to go to the disk, and read all the bytes on it. What difference does it make, compared to malloc'ing the same size of memory and manually read the entire
订阅 mmap