Read large txt file in c++

问题

I'd like to read a file of about 5MB in memory ... the file has this format (it is a text file)

ID 3:  0 itemId.1 0 itemId.2 0 itemId.5 1 itemId.7 ........................ 20 itemId.500
ID 50:  0 itemId.31 0 itemId.2 0 itemId.4 2 itemId.70 ........................ 20 itemId.2120
.....

how can I do this efficiently in c++?

回答1:

Reading a file line by line:

ifstream fin ("file.txt");
string     myStr;

while(getline(fin, myStr))   // Always put the read in the while condition.
{                            // Then you only enter the loop if there is data to
    //use myStr data         // processes. Otherwise you need to read and then
}                            //  test if the read was OK
                             //
                             // Note: The last line read will read up to (but not
                             //        past) then end of file. Thus When there is
                             //        no data left in the file its state is still
                             //        OK. It is not until you try and explicitly
                             //        read past the end of file that EOF flag is set.

For a reason to not explicitly call close see:
https://codereview.stackexchange.com/questions/540/my-c-code-involving-an-fstream-failed-review/544#544

If efficiency is your major goal (its probably not). Then read the whole file into memory and parse from there: see Thomas below: Read large txt file in c++

回答2:

Read the entire file into memory, then process the contents in memory.

A file resource (e.g. hard drive) is most efficient when the motor is kept spinning. So one large read of data is more efficient than 5 reads of small quantities of data.

On most platforms, memory is faster to access than a file. Using this information, one can make a program more efficient by reading data into memory then processing the memory.

Combining the two techniques will yield a greater performance: read as much data, in one transaction, into memory then process the memory.

Some people declare large arrays of char, or unsigned char (for binary data). Other people tell std::string or std::vector to reserve a large amount of memory, then read the data into the data structure.

Also, block reads (a.ka. istream::read()) will bypass most of the slow parts of the C++ stream facilities.

回答3:

Use a file stream:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main() {
    string line;
    ifstream myfile ("example.txt");
    if (myfile.is_open())
    {
        while ( getline(myfile, line) )
            cout << line << endl;

        myfile.close();
    }
    else 
    {
        cout << "Unable to open file"; 
    }

    return 0;
}

5MB really is not a large file. The stream will take care of reading chunks at a time for you, but really; almost any machine this runs on will likely be able to read 5MB right into memory no problem.

来源：https://stackoverflow.com/questions/7841903/read-large-txt-file-in-c

标签

c++

file

format