mmap

Resizing numpy.memmap arrays

由 时光毁灭记忆、已成空白 提交于 2020-01-01 01:15:48
问题 I'm working with a bunch of large numpy arrays, and as these started to chew up too much memory lately, I wanted to replace them with numpy.memmap instances. The problem is, now and then I have to resize the arrays, and I'd preferably do that inplace. This worked quite well with ordinary arrays, but trying that on memmaps complains, that the data might be shared, and even disabling the refcheck does not help. a = np.arange(10) a.resize(20) a >>> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 0, 0

How to read lines from a mmapped file?

由 家住魔仙堡 提交于 2019-12-29 14:15:12
问题 Is seems that the mmap interface only supports readline(). If I try to iterate over the object I get character instead of complete lines. What would be the "pythonic" method of reading a mmap'ed file line by line? import sys import mmap import os if (len(sys.argv) > 1): STAT_FILE=sys.argv[1] print STAT_FILE else: print "Need to know <statistics file name path>" sys.exit(1) with open(STAT_FILE, "r") as f: map = mmap.mmap(f.fileno(), 0, prot=mmap.PROT_READ) for line in map: print line # RETURNS

malloc vs mmap in C

由 空扰寡人 提交于 2019-12-29 10:28:52
问题 I built two programs, one using malloc and other one using mmap . The execution time using mmap is much less than using malloc . I know for example that when you're using mmap you avoid read/writes calls to the system. And the memory access are less. But are there any other reasons for the advantages when using mmap over malloc ? Thanks a lot 回答1: I assume that you are referring to using mmap and malloc for reading data from files. In that case you pretty much got the main point: using fread

malloc vs mmap in C

由 核能气质少年 提交于 2019-12-29 10:28:25
问题 I built two programs, one using malloc and other one using mmap . The execution time using mmap is much less than using malloc . I know for example that when you're using mmap you avoid read/writes calls to the system. And the memory access are less. But are there any other reasons for the advantages when using mmap over malloc ? Thanks a lot 回答1: I assume that you are referring to using mmap and malloc for reading data from files. In that case you pretty much got the main point: using fread

mmap for writing sequential log file for speed?

由 两盒软妹~` 提交于 2019-12-29 06:28:33
问题 I want to write log file, unstructured format (one line at a time), using mmap (for speed). What is the best procedure? Do I open empty file, truncate to 1 page size (write empty string to resize file?), then mmap - and repeat when mmaped area full? I usually use mmap for writing fixed size structures, usually just one page at a time, however this is for writing log files (anywhere from 0.5 - 10 Gb) using mmap but not sure what's the best practice once the first mmaped area is filled - munmap

Native memory allocation (mmap) failed to map xxx bytes for committing reserved memory

由 你说的曾经没有我的故事 提交于 2019-12-28 21:01:46
遇到问题 在服务器上运行 nexus 出现 Native memory allocation (mmap) failed to map 838860800 bytes for committing reserved memory 问题。 原因：查资料后是因为运行 nexus 需要 800m 的内存，而机器配置是1G,剩余可用的只有几十兆，所以导致无法启动 解决办法 方法一： 扩大机器配置； 方法二： 设置交换内存swap(一般设置为内存的两倍大小)； dd if=/dev/zero of=/tmp/swapfile bs=1024k count=256 #格式化文件 mkswap /tmp/swapfile #加载交换文件 swapon /tmp/swap #永久生效 vim /etc/fstab #最后一行添加 /swap1 swap swap defaults 0 0 方法三：设置虚拟机的启动参数 # Xms: 初始化堆内存(heap) # Xmx: 最大堆内存 # PermSize: 初始化永久内存，存储class类，不会被GC # MaxPermSize: 最大永久内存 # Xss: 每增加一个线程（thread)就会立即消耗的内存，而最佳值应该是128K,默认值好像是512 JAVA_OPTS=-Xss128k -Xms256m -Xmx512m -XX:PermSize

[linux内存]linux内存学习（一）

由 一曲冷凌霜 提交于 2019-12-28 13:21:27
1，内存中各个地址范围的含义 按照地址范围由低到高的顺序：0-3G的地址范围： 代码段： 代码的可执行文件，一般为只读并且是共享的。（RO code/data） 数据段 ：存已经被初始化的全局变量(RW data) static char *user="jiangsu" BSS段： 存一些未被初始化的全局变量(ZI data) static char*user 堆： maoolc()或者new（）申请的段，给程序员使用，地址向高地址范围增长(通过brk()函数扩展) mmap区间： （一些动态库文件） 栈： 函数参数等，由系统自动分配释放，地址向低地址范围增长（可以通过ulimit -s查看每个进程最大使用的栈的大小） 补充：正常代码段都不是从用户空间0地址开始的，比如arm linux嵌入式设备从0x8000（32K）开始的，前面的32k可以用来存储一些error code，如果一个函数返回一个地址，但是该地址大小小于0x8000，那么则认为此函数发生 了错误~ 正常stack和kernel space之间有一段空白，BSS和heap之间也有一段空白，heap和mmp之间也有一段空白，这个称为ASLR技术，防止地址被其他人知道而遭到攻击 3-4G的空间是内核空间，用户空间的代码不能访问到。 malloc函数详解~ malloc只是一个库函数，在不同的平台对malloc有不同的实现

Linux shared memory: shmget() vs mmap()?

由 三世轮回 提交于 2019-12-28 07:39:14
问题 In this thread the OP is suggested to use mmap() instead of shmget() to get shared memory in Linux. I visited this page and this page to get some documentation, but the second one gives an obscure example regarding mmap() . Being almost a newbie, and needing to share some information (in text form) between two processes, should I use the shmget() method or mmap() ? And why? 回答1: Both methods are viable. mmap method is a little bit more restrictive then shmget , but easier to use. shmget is

Efficiently reading a very large text file in C++

由 一个人想着一个人 提交于 2019-12-28 03:38:28
问题 I have a very large text file(45GB). Each line of the text file contains two space separated 64bit unsigned integers as shown below. 4624996948753406865 10214715013130414417 4305027007407867230 4569406367070518418 10817905656952544704 3697712211731468838 ... ... I want to read the file and perform some operations on the numbers. My Code in C++: void process_data(string str) { vector<string> arr; boost::split(arr, str, boost::is_any_of(" \n")); do_some_operation(arr); } int main() { unsigned

MMKV的原理与实现（一）

由 你说的曾经没有我的故事 提交于 2019-12-25 19:45:01
MMKV的原理与实现（一） 说到轻量级的数据持久化，大家最先想到的就是SharedPreferences（以下简称SP）了，SP存储方式为xml，直接使用I/O流进行文件的读写，这就形成了一个弊端：每次写入或修改都需要替换掉原来的数据，并将所有数据 重新写入文件。可想而知，如果一个sp文件的内容过多，那么再写入的时候会造成卡顿，甚至会有 ANR的风险。 一、I/O 1、先看一下SP的工作原理 虚拟内存被操作系统划分成两块：用户空间和内核空间，用户空间是用户程序代码运行的地方，内核空间是内核代码运行的地方。为了安全，它们是隔离的，即使用户的程序崩溃了，内核也不受影响。 2、使用I/O写入文件的流程 1、调用write，告诉内核需要写入数据的开始地址与长度 2、内核将数据拷贝到内核缓存 3、由操作系统调用，将数据拷贝到磁盘，完成写入 可见，将数据写入文件需要将数据拷贝两次，再写入到文件中，如果数据量过大，也会有很大的性能损耗。 二、MMKV 1、什么是MMKV 为了解决上述问题，微信团队基于MMAP研发了MMKV来代替SP。 MMKV 是基于 mmap 内存映射的 key-value 组件，底层序列化/反序列化使用 protobuf 实现，性能高，稳定性强。从 2015 年中至今在微信上使用，其性能和稳定性经过了时间的验证。近期也已移植到 Android / macOS /
订阅 mmap