How to generate a unique hash for a URL?

Question

Given these two images from twitter.

http://a3.twimg.com/profile_images/130500759/lowres_profilepic.jpg
http://a1.twimg.com/profile_images/58079916/lowres_pr         


        

Answer 1

While CRC32 produces a maximum 2^32 values regardless of your input and so will not avoid conflicts, it is still a viable option for this scenario.

It is fast, so if you generate filename that conflicts, just add/change a character to your URL and simply re-calc the CRC.

4.3 billion possible checksums mean the likelihood of a filename conflict, when combined with the original filename, are going to be so low as to be be unimportant in normal situations.

I've used this approach myself for something similar and was pleased with the performance. See Fast CRC32 in Software.