multithread read from disk?

Question

Suppose I need to read many distinct, independent chunks of data from the same file saved on disk.

Is it possible to multi-thread this upload?

Related: Do al

Answer 1

As mentioned in the other answers a parallel read may be slower depending on the way the file is physically stored on disk. So if the head has to move a significant distance it can cause an actual slowdown. This being said there are however storage systems which can support multiple simultaneous reads and writes efficiently. The most simple one I can imagine is a SSD disk. I myself worked with magnificent storage systems from IBM which could perform simultaneous reads and writes with no slowdown. So let's assume you have such a file system and physical storage which will not slow down on parallel reads.

In that case parallel reads are very logical. In general there are two ways to achieve that:

1. If you want to use the standard C/C++ library to perform the IO then the only option you have is to keep one open file handle (descriptor) per thread. This is because the file pointer (which points to where to read or write from in the file) is kept per handle. So if you try to read simultaneously from the same file handle you will not have any way of knowing what you are actually reading.
2. Use platform specific API to perform asynchronous (OVERLAPPED) IO. On windows you use the WinAPI functions with what is called OVERLAPPED IO. On Unix/Linux you have posix AIO although I understand that it's use is discouraged although I didn't see any satisfactory explanation as to why that is the case.

I myself implemented the both fd/thread approach on both linux and windows and the OVERLAPPED approach on windows. Both work great.