How do you search a large text file for a string without going line by line in C#?

Question

I have a large text file that I need to search for a specific string. Is there a fast way to do this without reading line by line?

This method is extremely slow beca

Answer 1

If you want to speed up the line-by-line reading you can create a queue-based application:
One thread reads the lines and enqeues them into a threadsafe Queue. A second one can then process the strings

Answer 2

You could buffer a large amount of data from the file into memory at one time, up to whatever constraint you wish, and then search it for the string.

This would have the effect of reducing the number of reads on the file and would likely be a faster method, but it would be more of a memory hog if you set the buffer size too high.

Answer 3

As Wayne Cornish already said: Reading line by line might be the best approach.

If you read for instance the entire file into a string and then search with a regular expression, it might be more elegant, but you'll create a large string object.

These kind of objects might cause problems, because they will be stored on the Large Object Heap (LOH, for objects above 85.000 bytes). If you parse many of these big files and your memory is limited (x86), you are likely to run into LOH fragmentation problems.

=> Better read line by line if you parse many big files!

Answer 4

The fastest method for searching is the Boyer-Moore algorithm. This method does not require to read all bytes from the files, but require random access to bytes. Also, this method is simple in realization.

Answer 5

If you're only looking for a specific string, I'd say line-by-line is the best and most efficient mechanism. On the other hand, if you're going to be looking for multiple strings, particularly at several different points in the application, you might want to look into Lucene.Net to create an index and then query the index. If this is a one-off run (i.e., you won't need to query the same file again later), you can create the index in a temporary file that will be cleaned up automatically by the system (usually boot time; or you can delete it yourself when your program exits). If you need to search the same file again later, you can save the index in a known location and get much better performance the second time around.

Answer 6

Here is my solution that uses a stream to read in one character at a time. I created a custom class to search for the value one character at a time until the entire value is found.

I ran some tests with a 100MB file saved on a network drive and the speed was totally dependent on how fast it could read in the file. If the file was buffered in Windows a search of the entire file took less than 3 seconds. Otherwise it could take anywhere from 7 seconds to 60 seconds, depending on network speed.

The search itself took less than a second if run against a String in memory and there were no matching characters. If a lot of the leading characters found matches the search could take a lot longer.

public static int FindInFile(string fileName, string value)
{   // returns complement of number of characters in file if not found
    // else returns index where value found
    int index = 0;
    using (System.IO.StreamReader reader = new System.IO.StreamReader(fileName))
    {
        if (String.IsNullOrEmpty(value))
            return 0;
        StringSearch valueSearch = new StringSearch(value);
        int readChar;
        while ((readChar = reader.Read()) >= 0)
        {
            ++index;
            if (valueSearch.Found(readChar))
                return index - value.Length;
        }
    }
    return ~index;
}
public class StringSearch
{   // Call Found one character at a time until string found
    private readonly string value;
    private readonly List<int> indexList = new List<int>();
    public StringSearch(string value)
    {
        this.value = value;
    }
    public bool Found(int nextChar)
    {
        for (int index = 0; index < indexList.Count; )
        {
            int valueIndex = indexList[index];
            if (value[valueIndex] == nextChar)
            {
                ++valueIndex;
                if (valueIndex == value.Length)
                {
                    indexList[index] = indexList[indexList.Count - 1];
                    indexList.RemoveAt(indexList.Count - 1);
                    return true;
                }
                else
                {
                    indexList[index] = valueIndex;
                    ++index;
                }
            }
            else
            {   // next char does not match
                indexList[index] = indexList[indexList.Count - 1];
                indexList.RemoveAt(indexList.Count - 1);
            }
        }
        if (value[0] == nextChar)
        {
            if (value.Length == 1)
                return true;
            indexList.Add(1);
        }
        return false;
    }
    public void Reset()
    {
        indexList.Clear();
    }
}