large-files

I'm a bioinformatician currently extracting normal-sized sequences from genomic files. Some genomic files are large enough that I don't want to put them into the main git repository, whereas I'm putting the extracted sequences into git. Is it possible to tell git "Here's a large file - don't store the whole file, just take its checksum, and let me know if that file is missing or modified." If that's not possible, I guess I'll have to either git-ignore the large files, or, as suggested in this question , store them in a submodule. I wrote a script that does this sort of thing. You put file

I have a auto-generated C++ source file, around 40 MB in size. It largely consists of push_back commands for some vectors and string constants that shall be pushed. When I try to compile this file, g++ exits and says that it couldn't reserve enough virtual memory (around 3 GB). Googling this problem, I found that using the command line switches --param ggc-min-expand=0 --param ggc-min-heapsize=4096 may solve the problem. They, however, only seem to work when optimization is turned on. 1) Is this really the solution that I am looking for? 2) Or is there a faster, better (compiling takes ages

I have a large file in windows XP - its 38GB. (a VM image) I cannot seem to copy it. Dragging on the desktop - gives error of "Insufficient system resources exist to complete the requested service" Using Java - FileChannel.transferTo(0, fileSize, dest) fails for all files > 2GB Using Java - FileChannel.transferTo() in chunks of 100Mb fails after ~18Gb java.io.IOException: Insufficient system resources exist to complete the requested service at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:44) at sun.nio.ch.IOUtil.writeFromNativeBuffer

In order to support upload of large (actually very large, up to several gigabytes) files with progress report we started using HttpClient with PushStreamContent, as described here . It works straightforward, we copy bytes between two streams, here is a code example: private void PushContent(Stream src, Stream dest, int length) { const int bufferLength = 1024*1024*10; var buffer = new byte[bufferLength]; var pos = 0; while (pos < length) { var bytes = Math.Min(bufferLength, length - pos); src.Read(buffer, 0, bytes); dest.Write(buffer, 0, bytes); pos += bufferLength; dest.Flush(); Console

I'm working with huge data CSV file. Each file contains milions of record , each record has a key. The records are sorted by thier key. I dont want to go over the whole file when searching for certian data. I've seen this solution : Reading Huge File in Python But it suggests that you use the same length of lines on the file - which is not supported in my case. I thought about adding a padding to each line and then keeping a fixed line length , but I'd like to know if there is a better way to do it. I'm working with python You don't have to have a fixed width record because you don't have to

I have two twins CentOS 5.4 servers with VMware Server installed on each. What is the most reliable and fast method for copying virtual machines files from one server to the other, assuming that I always use sparse file for my vmware virtual machines? The vm's files are a pain to copy since they are very large (50 GB) but since they are sparse files I think something can be done to improve the speed of the copy. If you want to copy large data quickly, rsync over SSH is not for you. As running an rsync daemon for quick one-shot copying is also overkill, yer olde tar and nc do the trick as