问题
I am trying to identify files with duplicate contents. Decided to do a comparison using a hashing mechanism (MD5, SHA1 or any other). Works fine with ".txt" files. However, with MS Office files (.doc,.docx,.xls, etc) this fails.
MD5/SHA1 hash is not constant for MS Office files, even if they have the same "text" content. I assume MS Office stores some kind of meta-data in the file, which changes each time you save the file. Thus the hash is different.
e.g. I have a file ABC.doc and I compute the hash (Hash1) for it. Open and change 1 word and save the file. Undo the change you made and save and compute hash (Hash2). Hash1 != Hash2 in this case. It is same if you try this on a ".txt" file
Is there a way to de-dupe MS Office documents based on hashing its contents? Can we hash only the contents of a file and not its meta-data?
回答1:
I don't think this can be done without extracting the text of the document using a tool and then hashing the text. I can recommend Stellent Outside In, now owned by Oracle. But that could be an overkill solution to your needs. They provide a tool to extract text from many types of files, including all office files and versions.
来源：https://stackoverflow.com/questions/22616462/c-sharp-hash-contents-of-ms-office-documents-without-metadata