I have a directory with 40 files with names from 0 to 39 (for example), I am trying to get the file with the largest number in its name (which means I need to get \"39\") I
It is only logical that they would be sorted that way, you would bring in some semantics to sort it by number, namely parse all the file names to numbers, then sort the files by that.
Something like
files.OrderBy(path => Int32.Parse(Path.GetFileNameWithoutExtension(path)))
Use Last() to get the file with the highest number.