C# directory.getfiles memory help

Question

Here is the code I’m using:

using (StreamWriter output = new StreamWriter(Path.Combine(masterdestination, \"Master.txt\")))
{
     string masterfolders = sou         


        

Answer 1

If you are implementing search then I suggest you use Windows Search 4.0

Answer 2

As mentioned in the answer here if using .NET 4.0, you can use the static EnumerateFiles method on the Directory class to get an IEnumerable<string> instead of an string[], which is leading to all the memory consumption.

If you are working with a version of .NET before .NET 4.0, you can easily mimic this functionality by calling the FindFirstFileEx, FindNextFile, etc, etc, methods through the P/Invoke layer.

Then, for every file that is returned from a call to FindFirstFile/FindNextFile you would yield return the item.

This will cut down on the memory consumption as EnumerateFiles would for directories with large numbers of files because you aren't loading them all into an array up front, but yielding them for processing as you find them.

Answer 3

If you cannot use Fx4 you are best of to write your own FileEnumerator. Here is one example.

Answer 4

Directory.GetFiles has to build a list of all the matching files before it can return. Only then can you enumerate them. So of course, it is expensive when there are lots of matching files. It may even build a list of all files internally.

If you can use .NET 4.0, then you could use Directory.EnumerateFiles which avoids this problem by returing one file at a time. If you can't, then I would suggest you write this in C++ rather than C#.

In C++ you can use FindFirstFile which also returns the files to you one at at time.

// iterate though the files in this directory
//
TCHAR szWild[MAX_PATH];
PathCombine(szWild, masterfolders, _T("*.txt"));

WIN32_FIND_DATA fd;
HANDLE hFind = FindFirstFile(szWild, &fd);
if (INVALID_HANDLE_VALUE != hFind)
{
   do {
   TCHAR szFileName[MAX_PATH];
   PathCombine(szFileName, masterfolders, fd.cFileName);

   // write szFilename to output stream..

   } while (FindNextFile(hFind, &fd));

   FindClose (hFind);
}

Answer 5

Directory.GetFiles really sucks. If you can use .NET 4.0 you should look into using Directory.EnumerateFiles. From the docs:

The EnumerateFiles and GetFiles methods differ as follows: When you use EnumerateFiles, you can start enumerating the collection of names before the whole collection is returned; when you use GetFiles, you must wait for the whole array of names to be returned before you can access the array. Therefore, when you are working with many files and directories, EnumerateFiles can be more efficient.