How to read a file and get words in C++

Question

I am curious as to how I would go about reading the input from a text file with no set structure (Such as notes or a small report) word by word. The text for example might be structured like this:

"06/05/1992
Today is a good day;
The worm has turned and the battle was won."

I was thinking maybe getting the line using getline, and then seeing if I can split it into words via whitespace from there. Then I thought using strtok might work! However I don't think that will work with the punctuation.

Another method I was thinking of was getting everything char by char and omitting the characters that were undesired. Yet that one seems unlikely.

So to sort the thing short: Is there an easy way to read an input from a file and split it into words?

Answer 1

Since it's easier to write than to find the duplicate question,

#include <iterator>

std::istream_iterator<std::string> word_iter( my_file_stream ), word_iter_end;

size_t wordcnt;
for ( ; word_iter != word_iter_end; ++ word_iter ) {
    std::cout << "word " << wordcnt << ": " << * word_iter << '\n';
}

The std::string argument to istream_iterator tells it to return a string when you do *word_iter. Every time the iterator is incremented, it grabs another word from its stream.

If you have multiple iterators on the same stream at the same time, you can choose between data types to extract. However, in that case it may be easier just to use >> directly. The advantage of an iterator is that it can plug into the generic functions in <algorithm>.

Answer 2

Yes. You're looking for std::istream::operator>> :) Note that it will remove consecutive whitespace but I doubt that's a problem here.

i.e.

std::ifstream file("filename");
std::vector<std::string> words;
std::string currentWord;
while(file >> currentWord)
    words.push_back(currentWord);

Answer 3

You can use getline with a space character, getline(buffer,1000,' ');

Or perhaps you can use this function to split a string into several parts, with a certain delimiter:

string StrPart(string s, char sep, int i) {
  string out="";
  int n=0, c=0;
  for (c=0;c<(int)s.length();c++) {
    if (s[c]==sep) {
      n+=1;
    } else {
      if (n==i) out+=s[c];
    }
  }
  return out;
}

Notes: This function assumes that it you have declared using namespace std;.

s is the string to be split. sep is the delimiter i is the part to get (0 based).

Answer 4

You can use the scanner technique to grabb words, numbers dates etc... very simple and flexible. The scanner normally returns token (word, number, real, keywords etc..) to a Parser.

If you later intend to interpret the words, I would recommend this approach.

I can warmly recommend the book "Writing Compilers and Interpreters" by Ronald Mak (Wiley Computer Publishing)