checksum

I have to compare checksum of all files in /primary and /secondary folders in machineA with files in this folder /bat/snap/ which is in remote server machineB . The remote server will have lots of files along with the files we have in machineA . If there is any mismatch in checksum then I want to report all those files that have issues in machineA with full path and exit with non zero status code. If everything is matching then exit zero. I wrote one command (not sure whether there is any better way to write it) that I am running on machineA but its very slow. Is there any way to make it

What algorithm to use to calculate a check digit for a list of digits? The length of the list is between 8 and 12 digits. see also: How to generate a verification code/number? The Luhn algorithm is good enough for the credit card industry... As RichieHindle points out, the Luhn algorithm is pretty good. It will detect (but not correct) any one error or transposition (except a transposition of 0 and 9). You could also consider the algorithm for ISBN check digits , although for old-style ISBN, the check digit is sometimes "X", which may be a problem for you if you're using integer fields. New

I have a system with roughly a 100 million documents, and I'd like to keep track of their modifications between mirrors. In order to exchange information about modifications effectively, I want to send information about modified documents by days, not by each separate document. Something like this: [ 2012/03/26, cs26], [ 2012/03/25, cs25], [ 2012/03/24, cs24], ... where each cs is the checksum of timestamps of all documents created on a particular day. Now, the problem I'm running into is that I don't know of an algorithm that could "subtract" data from the checksum when a document is being

zlib's crc32_combine() takes crcA, crcB, and lengthB to calculate crcAB. # returns crcAB crc32_combine(crcA, crcB, lenB) Using concepts from Mark Adler's awesome posts here and here I was able to produce crc32_trim_trailing.pl which takes crcAB, crcB, and lengthB to calculate crcA (I use this to peel off padding of a known length and value). # prints crcA perl crc32_trim_trailing.pl $crcAB $crcB $lenB Unfortunately, this uses the principles of the slow method described, where each null byte must be peeled off one at a time. It's slow, but is a good proof of concept. I've been working to make a

I wonder how reliable the adler32 checksum is, compared to e.g. md5 checksums? It was told on wikipedia that adler32 is "much less reliable" than md5, so I wonder how much, and in which way? More specifically, I'm wondering if it is reliable enough as a consistency check for long-time archiving of (tar) files of size 20GB+? For details on the error-checking capabilities of the Adler-32 checksum, see for example Revisiting Fletcher and Adler Checksums. Maxino, 2006. This paper contains an analysis on the Hamming distance provided by these two checksums, and provides an indication of the

I'm interested in optimizing the hashing of some large files (optimizing wall clock time). The I/O has been optimized well enough already and the I/O device (local SSD) is only tapped at about 25% of capacity, while one of the CPU cores is completely maxed-out. I have more cores available, and in the future will likely have even more cores. So far I've only been able to tap into more cores if I happen to need multiple hashes of the same file, say an MD5 AND a SHA256 at the same time. I can use the same I/O stream to feed two or more hash algorithms, and I get the faster algorithms done for