How to traverse files (conditionally) faster than using FileSystemObject

Question

I\'ve written some VBA code using file objects to go into a folder, search for particular files (CSV) that meet certain criteria (contain \"HR\" in filename and created with

Answer 1

This ought to show some improvement, considering the ratio of HR files to total files (250 / 30,000).

Using Dir Function, Minimize reliance on FileSystemObject

The idea here is to use the Dir function first to get a list of all file names that contain the "HR" substring, and only use the FileSystemObject against those files to get the timestamp information -- there's no use incurring the overhead of FSO on every file in that directory.

Then, we process only those files which match the "HR" criteria:

Sub usingDir()
Dim folderPath As String
Dim fileName As String
Dim filesToProcess As New Collection
Dim item As Variant
Dim fileDate As Date
Dim firstDate As Date
Dim secondDate As Date

'Defining the user-input variables
firstDate = Cells(2, "E").Value
secondDate = Cells(3, "E").Value
folderPath = "\\SRV-1\process\DUMP\"

' Gets a collection of files matching the "HR" criteria
fileName = Dir(folderPath)
Do While Not fileName = ""
    If InStr(fileName, "HR") > 0 Then
        'Only processing files with "HR"
        filesToProcess.Add (folderPath & fileName)
    End If
    fileName = Dir
Loop

'Now we deal only with the "HR" files:
With CreateObject("Scripting.FileSystemObject")
    For Each item In filesToProcess
        ' Check the date last modified
        fileDate = .GetFile(item).DateLastModified ' modify as needed
        If firstDate < fileDate And secondDate > fileDate Then
            '
            '
            Debug.Print item
            'your code to Do Stuff goes here
            '
            '
            '
        End If
    Next
End With
End Sub

UPDATE: Without Using the FileSystemObject

This was nagging at me, and I figured there must be a way to get the timestamp information without relying on FileSystemObject. There is. We'll still use Dir to traverse the files, but now we'll eliminate any reference to FileSystemObject and replace with some fancy WinAPI function calls. Check out Chip Pearson's article here and download the .bas modules. You'll need the following two files imported to your VBProject:

• modGetSetFileTimes
• modTimeConversionFunctions

And then you can do something like this:

Option Explicit
Sub withoutFSO()
Dim folderPath As String
Dim FileName As String
Dim filesToProcess As New Collection
Dim item As Variant
Dim fileDate As Date
Dim firstDate As Date
Dim secondDate As Date

'Defining the user-input variables
firstDate = Cells(2, "E").Value
secondDate = Cells(3, "E").Value
folderPath = "\\Your\Path"

' Gets a collection of files matching the "HR" criteria and our Date range
FileName = Dir(folderPath)
Do While Not FileName = ""
    'Only processing files with "HR"
    If InStr(FileName, "HR") > 0 Then
        ' Only process files that meet our date criteria
        fileDate = CDate(modGetSetFileTimes.GetFileDateTime(CStr(item), FileDateLastModified))
        If firstDate < fileDate And secondDate > fileDate Then
            filesToProcess.Add (folderPath & FileName)
        End If
    End If
    FileName = Dir
Loop

'Now we deal only with the matching files:
For Each item In filesToProcess
    Debug.Print item
    Debug.Print fileDate
    'your code to Do Stuff goes here
    '
    '
    '
Next
End Sub

This should be an improvement even over my original answer, and, if combined with a more efficient manner of retrieving data (i.e., using ADO instead of Workbooks.Open, if possible) then you should be very optimized.