Is it possible to do CRC-32 calculation in splits?

Question

I use this trivial function to calculate the CRC checksum of a given file:

long i, j = 0;
int k = 0;
uint crc = 0xFFFFFFFF;
FileInfo file_info = new FileInfo(file);
byte[] file_buffer = new byte[32768];

FileStream file_stream = new FileStream(@file, FileMode.Open);
while ((i = file_stream.Read(file_buffer, 0, file_buffer.Count())) > 0)
{
    for (j = 0; j < i; j++)
    {
        uint before = crc;
        k = (int)((crc ^ file_buffer[j]) & 0x000000FFL);
        uint after = (uint)((crc >> 8) & 0x00FFFFFFL) ^ crc32_table[k];
        crc = after;
        uint test = (uint)((crc << 8) & 0x00FFFFFFL) ^ crc32_table[k];
        MessageBox.Show((~crc).ToString("X"));
    }
}
file_stream.Close();
return ~crc;

My question is this: Say I have a large file, of say 100MB. Is there any link between a CRC-32 calculation of the first 50MB and the last 50MB and the CRC-32 calculation of the 100MB file?

The reason I'm asking, is I have some very large files (~10GB give or take) which take some time to generate, but while they're being generated, most parts remain static, however, parts in the middle (known point) and right at the start (header, also known part/length). Calculating a CRC-32 checksum of a 10GB file takes quite some time, so I was wondering if there was any way to do it in chunks?

Answer 1

It is indeed possible to parallelize the CRC-32 computation, but the details are messy and I'd need to spend about a day to come up with the code.

Let's look at the basic CRC algorithm, where there is no negation and no bit reversal.

For the string of bytes that you want to compute the CRC of, let's call it the message. The basic idea is that you treat the message as a polynomial in GF(2), and you compute its remainder modulo the CRC polynomial.

The basic CRC algorithm is additive/linear. If you have two messages a and b of the same length, then CRC(a XOR b) = CRC(a) XOR CRC(b).

Furthermore, if you pad the message on the right side with n zeros, the new CRC will be the old CRC times x^n mod the CRC polynomial.

 

With all that said, the only way to solve your problem is to really understand the mathematics behind the CRC algorithm and write your own custom code. Here's a long but very complete explanation of CRCs: http://www.ross.net/crc/download/crc_v3.txt

Answer 2

Yes. See crc32_combine_() in zlib for an example.

Answer 3

This equation is key:

CRC(a XOR b) == CRC(a) XOR CRC(b)

Suppose you want to compute the CRC of the following message:

"Always desire to learn something useful."

There exist functions to compute the CRC as:

crc_join(crc_part1("Always desire to lea"),
         crc_part2("rn something useful."))

If crc_part1 and crc_part2 zeros pad (\0) their arguments as shown, crc_join is just XOR.

crc_part1 = crc("Always desire to lea\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0")
crc_part2 = crc("\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0rn something useful.")

The trailing zeros can be accounted for in crc_part1 with lookup tables. The leading zeros can be ignored in crc_part2.

References:

1. High-speed Parallel Architecture for Software-based CRC
   Youngju. Do, Sung-Rok. Yoon, Taekyu. Kim, Kwang Eui. Pyun and Sin-Chong. Park
2. https://en.wikipedia.org/wiki/Cyclic_redundancy_check

Answer 4

I know this is an old question but this is what I use for "chunking" CRC calculations in case this helps anyone:

public static class Crc32Checksum
{
    private static readonly uint[] Table = GenerateTable();

    /// <summary>
    /// Calculates a CRC32 value for the data given.
    /// </summary>
    /// <param name="data">Data contents</param>
    /// <param name="offset">Byte offset to start reading</param>
    /// <param name="count">Number of bytes to read</param>
    /// <returns>The computed CRC32 value.</returns>
    public static int Calculate(byte[] data, int offset, int count) 
        => Calculate(0, data, offset, count);

    /// <summary>
    /// Calculates a new CRC32 value given additional data for the current CRC value.
    /// </summary>
    /// <param name="currentCrc">The current CRC value to start with</param>
    /// <param name="data">Additional data contents</param>
    /// <param name="offset">Byte offset to start reading</param>
    /// <param name="count">Number of bytes to read</param>
    /// <returns>The computed CRC32 value.</returns>
    public static int Calculate(int currentCrc, byte[] data, int offset, int count)
    {
        unchecked
        {
            uint crc = ~(uint)currentCrc;

            for (int i = offset, end = offset + count; i < end; i++)
                crc = (crc >> 8) ^ Table[(crc ^ data[i]) & 0xFF];

            return (int)~crc;
        }
    }

    private static uint[] GenerateTable()
    {
        unchecked
        {
            var table = new uint[256];

            const uint poly = 0xEDB88320;

            for (uint i = 0; i < table.Length; i++)
            {
                uint crc = i;

                for (int j = 8; j > 0; j--)
                {
                    if ((crc & 1) == 1)
                        crc = (crc >> 1) ^ poly;
                    else
                        crc >>= 1;
                }

                table[i] = crc;
            }

            return table;
        }
    }
}