It would appear the correct answer is to use char * for pointer arithmetic, because sizeof(char) is always defined to be 1, and to be of the finest addressable granularity on any platform.
So in short, there is no way around the limitation, char * is in fact the proper way to do it.
Matthias Wandel had the right answer but with a different justification.