I had to implement some code to traverse a directory structure and return a list of files found. The requirements were pretty simple:
The mistake you have made is to mock File. There is a testing anti pattern that assumes that if your class delegates to class X, you must mock class X to test your class. There is also a general rule to be cautious of writing unit tests that do file I/O, because they tend to be too slow. But there is no absolute prohibition on file I/O in unit tests.
In your unit tests have a temporary directory set up and torn down, and create test files and directories within that temporary directory. Yes, your tests will be slower than pure CPU tests, but they will still be fast. JUnit even has support code to help with this very scenario: a @Rule on a TemporaryFolder.
Just this week I implemented, using TDD, some housekeeping code that had to scan through a directory and delete files, so I know this works.