large-files

We're just starting a UNIX class and are learning a variety of Bash commands. Our assignment involves performing various commands on a directory that has a number of folders under it as well. I know how to list and count all the regular files from the root folder using: find . -type l | wc -l But I'd like to know where to go from there in order to find the largest file in the whole directory. I've seen somethings regarding a du command, but we haven't learned that, so in the repertoire of things we've learned I assume we need to somehow connect it to the ls -t command. And pardon me if my

Is there a Python file type for accessing random lines without traversing the whole file? I need to search within a large file, reading the whole thing into memory wouldn't be possible. Any types or methods would be appreciated. This seems like just the sort of thing mmap was designed for. A mmap object creates a string-like interface to a file: >>> f = open("bonnie.txt", "wb") >>> f.write("My Bonnie lies over the ocean.") >>> f.close() >>> f.open("bonnie.txt", "r+b") >>> mm = mmap(f.fileno(), 0) >>> print mm[3:9] Bonnie In case you were wondering, mmap objects can also be assigned to: >>>

How do I read a large CSV file (> 1 Gb) with a Scala Stream? Do you have a code example? Or would you use a different way to read a large CSV file without loading it into memory first? Just use Source.fromFile(...).getLines as you already stated. That returns an Iterator, which is already lazy (You'd use stream as a lazy collection where you wanted previously retrieved values to be memoized, so you can read them again) If you're getting memory problems, then the problem will lie in what you're doing after getLines. Any operation like toList , which forces a strict collection, will cause the

I'm using Python 2.6 on a Mac Mini with 1GB RAM. I want to read in a huge text file $ ls -l links.csv; file links.csv; tail links.csv -rw-r--r-- 1 user user 469904280 30 Nov 22:42 links.csv links.csv: ASCII text, with CRLF line terminators 4757187,59883 4757187,99822 4757187,66546 4757187,638452 4757187,4627959 4757187,312826 4757187,6143 4757187,6141 4757187,3081726 4757187,58197 So each line in the file consists of a tuple of two comma separated integer values. I want to read in the whole file and sort it according to the second column. I know, that I could do the sorting without reading the

I tried opening a huge (~2GB) file in VIM but it choked. I don't actually need to edit the file, just jump around efficiently. How can I go about working with very large files in VIM? Florian I had a 12GB file to edit today. The vim LargeFile plugin did not work for me. It still used up all my memory and then printed an error message :-(. I could not use hexedit for either, as it cannot insert anything, just overwrite. Here is an alternative approach: You split the file, edit the parts and then recombine it. You still need twice the disk space though. Grep for something surrounding the line

This question already has an answer here: Downloading large files reliably in PHP 13 answers I'm using php to download files, rather than the file itself opening in a new window. It seems to work ok for smaller files, but does not work for large files (I need this to work on very large files). Here's the code I have to download the file: function downloadFile($file) { if (file_exists($file)) { //download file header('Content-Description: File Transfer'); header('Content-Type: application/octet-stream'); header('Content-Disposition: attachment; filename='.basename($file)); header('Content