Everybody knows how to set up a favicon.ico link in HTML:
Killer Solution in 2020
This solution necessarily comes nine years after the question was originally asked, because, until fairly recently, most browsers have not been able to handle favicons in .svg format.
That's not the case anymore.
See: https://caniuse.com/#feat=link-icon-svg
Right now, in June 2020, these browsers can handle SVG Favicons:
Note that these browsers still can't:
With the above in mind, we can now use SVG Favicons with a reasonable degree of confidence.
The main objective here is to avoid HTTP Requests.
As other solutions on this page have mentioned, a pretty smart way to do this is to use a Data URI rather than an HTTP URL.
SVGs (especially small SVGs) lend themselves perfectly to Data URIs, because the latter is simply plaintext (with any potentially ambiguous characters percentage-encoded) and the former, being XML, can be written out as a long line of plaintext (with a smattering of percentage codes) incredibly straightforwardly.
In December 2019, Leandro Linares was one of the first to realise that since Chrome had joined Firefox in supporting SVG Favicons, it was worth experimenting to see if a favicon could be created out of an emoji:
https://lean8086.com/articles/using-an-emoji-as-favicon-with-svg/
Linares' hunch was right.
Several months later (March 2020), Code Pirate Lea Verou realised the same thing:
https://twitter.com/leaverou/status/1241619866475474946
And favicons were never the same again.
Here's a simple SVG: