FFMPEG is really a great tool. I know it can edit ID3 tags and even remove all tags in a row :
ffmpeg -i tagged.mp3 -map_metadata -1 untagged.mp3
I tried llogan's solution with a small castle.mp3 file and found out that its size increased from 4448 to 4797 bytes! Further inspection in Audacity revealed that the signal has been slightly "delayed" as well - however the length of the file [castle2.mp3] remained the same.
After that, I used id3v2 -D castle.mp3
to delete all mp3 tags from the file, and the filesize went down to 4320 bytes, with no other noticeable (undesired) changes.