Python: handling a large set of data. Scipy or Rpy? And how?

Question

In my python environment, the Rpy and Scipy packages are already installed.

The problem I want to tackle is such:

1) A huge set of financial data are stored in

Answer 1

Python's File I/O doesn't have bad performance, so you can just use the file module directly. You can see what functions are available in it by typing help (file) in the interactive interpreter. Creating a file is part of the core language functionality and doesn't require you to import file.

Something like:

f = open ("C:\BigScaryFinancialData.txt", "r");
for line in f.readlines():
    #line is a string type
    #do whatever you want to do on a per-line basis here, for example:
    print len(line)

Disclaimer: This is a Python 2 answer. I'm not 100% sure this works in Python 3.

I'll leave it to you to figure out how to show the top 10 rows and find the row sums. This can be done with simple program logic that shouldn't be a problem without any special libraries. Of course, if the rows have some kind of complicated formatting that makes it difficult to parse out the values, you might want to use some kind of module for parsing, re for example (type help(re) into the interactive interpreter).