I am using 2 500K bitmaps in to display images on my WiX dialogs.
They dramatically increase the size of the installation package, and what is worse - it looks there
The Windows Installer documentation for the Bitmap control states that the image should be a "bitmap" -- presumably a .BMP file -- or a JPEG.
You have to remember, when working with WiX, that it's based on Windows Installer. This means that any limitations in WiX are often caused by limitations in the underlying Windows Installer implementation.
.BMP files can be RLE-compressed. I don't know if they support any other compression algorithms.