ram

What is the fastest way to read several lines of data from a large file

可紊 提交于 2021-02-11 12:47:41
问题 My application needs to read like thousands of lines from a large csv file around 300GB with billion lines, each line contains several numbers. The data are like these: 1, 34, 56, 67, 678, 23462, ... 2, 3, 6, 8, 34, 5 23,547, 648, 34657 ... ... ... I tried fget reading file line by line in c, but it took really really really long, even with wc -l in linux, just to read all of the line, it took quite a while. I also tried to write all data to sqlite3 database based on the logics of the

What is the fastest way to read several lines of data from a large file

社会主义新天地 提交于 2021-02-11 12:45:45
问题 My application needs to read like thousands of lines from a large csv file around 300GB with billion lines, each line contains several numbers. The data are like these: 1, 34, 56, 67, 678, 23462, ... 2, 3, 6, 8, 34, 5 23,547, 648, 34657 ... ... ... I tried fget reading file line by line in c, but it took really really really long, even with wc -l in linux, just to read all of the line, it took quite a while. I also tried to write all data to sqlite3 database based on the logics of the

C++ Program immediately using 2 GB of RAM: how to find culprit?

女生的网名这么多〃 提交于 2021-02-10 17:52:24
问题 I have a program that, when queried on initialisation, is immediately using > 2 GB of RAM. Basically the code is like this: #include <blah> int main() { cout << get_mem_usage() << endl; //Lots of things happen, but no significant memory usage return 0; } Output: [2013-02-15 18:38:05.865283] 2147.71 Mb I am, however, linking to a lot of different shared object files: I checked ldd and I am linking to 58 libraries, with a combined .so size of 66 MB. I'm pretty sure that the get_mem_usage

C++ Program immediately using 2 GB of RAM: how to find culprit?

冷暖自知 提交于 2021-02-10 17:51:37
问题 I have a program that, when queried on initialisation, is immediately using > 2 GB of RAM. Basically the code is like this: #include <blah> int main() { cout << get_mem_usage() << endl; //Lots of things happen, but no significant memory usage return 0; } Output: [2013-02-15 18:38:05.865283] 2147.71 Mb I am, however, linking to a lot of different shared object files: I checked ldd and I am linking to 58 libraries, with a combined .so size of 66 MB. I'm pretty sure that the get_mem_usage

How Much Memory Can I Allocate?

狂风中的少年 提交于 2021-02-07 14:21:59
问题 If I have 16 GB of RAM on my machine, how much can I allocate to a java command line program I'm executing? I assume java -Xmx 16g... will crash my system? EDIT: In light of the comments, I tried java -Xmx16g... , and it did not crash my machine. The program still ran out of memory. I tried java -Xmx32g... , which did crash my machine. From the comments below (which have been really enlightening), I guess I just need to keep playing around with the allocations. 回答1: The size the heap memory

Virtuoso System Requirements

一笑奈何 提交于 2021-02-07 03:47:51
问题 We would be using Virtuoso for storing RDFs, the triple count will be 100 million to start with. I need to know what should be typical RAM, CPU, Disk etc for this. Querying will be with SPARQL and there will be a bit complex queries. Kindly provide your inputs. 回答1: The average size of a Virtuoso version 6.x triple (quad) is about 30bytes thus for 100 million triples you would need about 3GB RAM , this being the most critical component to enable the database working set to fit in memory ,

how to keep c++ variables in RAM securely?

随声附和 提交于 2021-02-06 10:15:34
问题 I'm working on a C++ application which is keeping some user secret keys in the RAM. This secret keys are highly sensitive & I must minimize risk of any kind of attack against them. I'm using a character array to store these keys, I've read some contents about storing variables in CPU registers or even CPU cache (i.e using C++ register keyword), but seems there is not a guaranteed way to force application to store some of it's variables outside of RAM (I mean in CPU registers or cache). Can

how to keep c++ variables in RAM securely?

≡放荡痞女 提交于 2021-02-06 10:12:33
问题 I'm working on a C++ application which is keeping some user secret keys in the RAM. This secret keys are highly sensitive & I must minimize risk of any kind of attack against them. I'm using a character array to store these keys, I've read some contents about storing variables in CPU registers or even CPU cache (i.e using C++ register keyword), but seems there is not a guaranteed way to force application to store some of it's variables outside of RAM (I mean in CPU registers or cache). Can

Make java understand that swap exists, and encourage him to use it [closed]

喜欢而已 提交于 2021-01-29 07:12:43
问题 Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers. Want to improve this question? Update the question so it's on-topic for Stack Overflow. Closed 2 months ago . Improve this question So, I have a java jar to run on my raspberry pi, and it requires a lot of memory. I have set up 2 gb swap on an external usb flash drive. But when I start the jar with -Xmx2048M, it says that it could not reserve enough space for 2097152KB object heap. I know

JVM - How much RAM should I allocate

自作多情 提交于 2021-01-28 18:35:58
问题 I didn't find a suggestion on how much RAM I actually should allocate to the JVM. Assuming that my computer has 4GB RAM and runs Windows 10. I nearly only use Java Applications on it. With how much RAM should I start the JVM? The system needs RAM for itself too, so in my logic, I should not allocate 4 GB even if that is possible. This isn't a black/white question, I just want to have an idea and how to come to an answer in this question. 回答1: I didn't find a suggestion on how much RAM I