You can use the mhash library (license is LGPL). On Debian systems:
sudo apt-get install libmhash-dev
See the man page man 3 mhash
But I don't think you can just give it the name of a file. You have to open the file yourself, read the data, and feed the data to this library's functions.