echo '#!'
Basically, with double quotes (") aka "weak quoting", Bash does some wacky things with the string, like variable substitution. With single (') aka "strong quoting" the string is taken literally.
See e.g. here for a more in-depth explanation of quoting.