Insert bytes into middle of a file (in windows filesystem) without reading entire file (using File Allocation Table)?

Question

I need a way to insert some file clusters into the middle of a file to insert some data.

Normally, I would just read the entire file and write it back out again with

Answer 1

No. What you are asking is not directly possible in Windows.

This is because in Windows, files are a logically contiguous collection of bytes, and it is not possible to insert bytes into the middle of the file without overwriting.

To understand why, let's conduct a thought experiment of what it would mean if it were possible.

Firstly, memory mapped files would suddenly become much more complicated. If we've mapped a file at a particular address, and then put some extra bytes in the middle of it, what would that mean for the memory mapping? Should the memory mapping now suddenly move? And if so, what happens to the program that doesn't expect it to?

Secondly, let's consider what happens with GetFilePointer if two handles are open to the same file, and one inserts extra bytes in the middle of that file. Let's suppose Process A has the file open for reading, and Process B has it open for reading and writing.

Process A wants to save it's location whilst doing a few reads, so it writes some code a bit like

DWORD DoAndThenRewind(HANDLE hFile, FARPROC fp){
   DWORD result;
   LARGEINTEGER zero = { 0 };
   LARGEINTEGER li;
   SetFilePointer(hFile, zero, &li, FILE_CURRENT);

   result = fp();

   SetFilePointer(hFile, &li, &li, FILE_BEGIN);
   return result;
}

Now what happens to this function if Process B wants to insert some extra bytes in the file? Well, if we add the bytes after where Process A currently is, all is fine - the file pointer (which is the linear address from the start of the file) remains the same before and after and all is well.

But if we add extra bytes in before where Process A is, well, suddenly our captured file pointers are all misaligned, and bad things start to happen.

Or to put it another way, adding bytes into the middle of the file means that we suddenly need to invent more clever ways of describing where we are in the file for rewinding purposes, since files are no longer a logically contiguous selection of bytes.

So heretofore we've discussed why it's probably a bad idea for Windows to expose this kind of functionality; but that doesn't really answer the question "is it actually possible". The answer here is still no. It is not possible.

Why? Because no such functionality is exposed to user mode programs to do this. As a user-mode program you have one mechanism for getting a handle to a file (NtCreateFile/NtOpenFile), you can read and write to it via NtReadFile/NtWriteFile, you can seek it and rename it and delete it via NtSetFileInformation, and you can release the handle reference via NtClose.

Even from kernel mode, you don't have many more options. The filesystem API is abstracted away from you, and filesystems treat files as logically contiguous collections of bytes, not as linked lists of byte ranges or anything that would make it easy to expose a method for you to insert non-overwriting bytes in the middle of a file.

That's not to say that it's not possible per-se. As others have mentioned, it's possible for you to open up the disk itself, pretend to be NTFS and alter the disk clusters assigned to a particular FCB directly. But doing so is brave. NTFS is barely documented, is complex, subject to change, and difficult to modify even when it's not mounted by the OS, never mind when it is.

So the answer, I'm afraid is no. It is not possible via normal safe Windows mechanisms to add extra bytes to the middle of a file as an insertion rather than as an overwriting operation.

Instead, consider looking at your problem to see whether it's appropriate for you to chunk up your files into smaller files and have an index file. That way you'll be able to modify the index file to insert extra chunks. By breaking your reliance on data needing to reside in one file, you'll find it easier to avoid the filesystem's requirement that a file is logically contiguous collection of bytes. You'll then be able to modify the index file to add extra chunks to your "pseduofile" without needing to read the entire pseudofile into memory.