large-files | 易学教程

Is O_LARGEFILE needed just to write a large file?

阅读更多关于 Is O_LARGEFILE needed just to write a large file?

Is the O_LARGEFILE flag needed if all that I want to do is write a large file ( O_WRONLY ) or append to a large file ( O_APPEND | O_WRONLY )? From a thread that I read titled " Cannot write >2gb index file " on the CLucene-dev mailing list, it appears that O_LARGEFILE might be needed to write large files, but participants in that discussion are using O_RDWR , not O_WRONLY , so I am not sure. O_LARGEFILE should never be used directly by applications. It's to be used internally by the 64-bit-offset-compatible version of open in libc when it makes the syscall to the kernel (Linux, or possibly

How to programmatically download a large file in C#

阅读更多关于 How to programmatically download a large file in C#

I need to programmatically download a large file before processing it. What's the best way to do that? As the file is large, I want to specific time to wait so that I can forcefully exit. I know of WebClient.DownloadFile(). But there does not seem a way to specific an amount of time to wait so as to forcefully exit. try { WebClient client = new WebClient(); Uri uri = new Uri(inputFileUrl); client.DownloadFile(uri, outputFile); } catch (Exception ex) { throw; } Another way is to use a command line utility (wget) to download the file and fire the command using ProcessStartInfo and use Process'

Advice on handling large data volumes

阅读更多关于 Advice on handling large data volumes

问题 So I have a "large" number of "very large" ASCII files of numerical data (gigabytes altogether), and my program will need to process the entirety of it sequentially at least once. Any advice on storing/loading the data? I've thought of converting the files to binary to make them smaller and for faster loading. Should I load everything into memory all at once? If not, is opening what's a good way of loading the data partially? What are some Java-relevant efficiency tips? 回答1: So then what if

How to read large text file on windows? [closed]

阅读更多关于 How to read large text file on windows? [closed]

问题 I have a large server log file (~750 MB) which I can't open with either Notepad or Notepad++ (they both say the file is too large). Can anyone suggest a program (for Windows) that will only read a small part of the file into memory at a time? Or do I need to write my own app to parse this file? 回答1: try this... Large Text File Viewer By the way, it is free :) But, I think you should ask this on serverfault.com instead 回答2: If all you need is a tool for reading, then this thing will open the

R:Loops to process large dataset(GBs) in chunks?

阅读更多关于 R:Loops to process large dataset(GBs) in chunks?

问题 I have a large data set in GBs that I'd have to process before I analyse them. I tried creating a connector, which allows me to loop through the large datasets and extract chunks at a time.This allows me to quarantine data that satisfies some conditions. My problem is that I am not able to create an indicator for the connector that stipulates it is null and to execute close(connector) when the end of the dataset is reached. Moreover, for the first chunk of extracted data, I'd have to skip 17

Binary search in a sorted (memory-mapped ?) file in Java

阅读更多关于 Binary search in a sorted (memory-mapped ?) file in Java

问题 I am struggling to port a Perl program to Java, and learning Java as I go. A central component of the original program is a Perl module that does string prefix lookups in a +500 GB sorted text file using binary search (essentially, "seek" to a byte offset in the middle of the file, backtrack to nearest newline, compare line prefix with the search string, "seek" to half/double that byte offset, repeat until found...) I have experimented with several database solutions but found that nothing

php uploading large files

阅读更多关于 php uploading large files

问题 I'm stuck here with file uploading problem. I've searched for answers but found only "increasing post_max_size and upload_max_filesize" suggestion and that doesn't work for me. I can't get large files uploaded(approx. around 150mb+), the following are my php.ini settings and my environments php.ini - max_input_time 300 - max_execution_time 600 - memory_limit 1024M - upload_max_filesize 1512M - post_max_size 2048M environments - Webserver: XAMPP - PHPFramwork: CodeIgniter I’ve also tried many

32 bit Windows and the 2GB file size limit (C with fseek and ftell)

阅读更多关于 32 bit Windows and the 2GB file size limit (C with fseek and ftell)

问题 I am attempting to port a small data analysis program from a 64 bit UNIX to a 32 bit Windows XP system (don't ask :)). But now I am having problems with the 2GB file size limit (long not being 64 bit on this platform). I have searched this website and others for possible sollutions but cannot find any that are directly translatable to my problem. The problem is in the use of fseek and ftell. Does anyone know of a modification to the following two functions to make them work on 32 bit Windows

Python - How to gzip a large text file without MemoryError?

阅读更多关于 Python - How to gzip a large text file without MemoryError?

问题 I use the following simple Python script to compress a large text file (say, 10GB ) on an EC2 m3.large instance. However, I always got a MemoryError : import gzip with open('test_large.csv', 'rb') as f_in: with gzip.open('test_out.csv.gz', 'wb') as f_out: f_out.writelines(f_in) # or the following: # for line in f_in: # f_out.write(line) The traceback I got is: Traceback (most recent call last): File "test.py", line 8, in <module> f_out.writelines(f_in) MemoryError I have read some discussion

Is git worth for managing many files bigger than 500MB

阅读更多关于 Is git worth for managing many files bigger than 500MB

问题 I would put under version control a big amount of data, i.e. a directory structure (with depth<=5) with hundreds files with size about 500Mb). The things I need is a system that help me: - to detect if an files has been changed - to detect if files were added/removed - to clone the entire repository in another location - to store a "checkpoint" and restore it later I don't need sha1 for change detect, something faster is acceptable. Is git worth for this? There is a better alternative? 回答1: