In a fresh VCL application Compile and Build operation produce the same binary and map file (with minor differences at the end of .exe file
There is two parts to this answer I believe.
Part of the problem your seeing, IIRC, is that the compiler doesn't zero out memory before doing a compile/build. Thus anything left in uninitialized memory becomes filler in the output for alignment purposes.
I also seem to recall that there is a date time stamp included in the pe header information for the application. that will cause a difference every time.
I'm not the best person to confirm this but this is what I seem to recall from past discussions.
People like Allen Bauer or Barry Kelly would probably be able to give better/more accurate information on this.