Is it possible to efficiently insert or remove bytes from the middle of a large file, and if so how? Or am I stuck rewriting the entire file after the point
You have to copy the file. At best, you may be able to get away using Sparse Files, but only if the 'A lot of Bytes' is Zeros.