Calculate and print SHA256 hash of a file using OpenSSL

Question

I'm trying to write a C function using OpenSSL/libcrypto to calculate the SHA256 sum of a file. I'm basing my code on Adam Lamer's c++ example here.

Here's my code:

int main (int argc, char** argv)
{
    char calc_hash[65];

    calc_sha256("file.txt", calc_hash);
}

int calc_sha256 (char* path, char output[65])
{
    FILE* file = fopen(path, "rb");
    if(!file) return -1;

    char hash[SHA256_DIGEST_LENGTH];
    SHA256_CTX sha256;
    SHA256_Init(&sha256);
    const int bufSize = 32768;
    char* buffer = malloc(bufSize);
    int bytesRead = 0;
    if(!buffer) return -1;
    while((bytesRead = fread(buffer, 1, bufSize, file)))
    {
        SHA256_Update(&sha256, buffer, bytesRead);
    }
    SHA256_Final(hash, &sha256);

    sha256_hash_string(hash, output);
    fclose(file);
    free(buffer);
    return 0;
}      

void sha256_hash_string (char hash[SHA256_DIGEST_LENGTH], char outputBuffer[65])
{
    int i = 0;

    for(i = 0; i < SHA256_DIGEST_LENGTH; i++)
    {
        sprintf(outputBuffer + (i * 2), "%02x", hash[i]);
    }

    outputBuffer[64] = 0;
}

The problem is this....take a look at the calculated sums below for an example file:

Known good SHA256: 6da032d0f859191f3ec46a89860694c61e65460d54f2f6760b033fa416b73866
Calc. by my code:  6dff32ffff59191f3eff6affff06ffff1e65460d54ffff760b033fff16ff3866

I also get * stack smashing detected * when the code is finished executing.

Does anyone see what I'm doing wrong?

Thanks!

Answer 1

Looks like there are a lot of '0xff' blocks in your output, and the corresponding blocks in the good string have the high bit set ... maybe a sign extension problem somewhere.

Does making:

char hash[SHA256_DIGEST_LENGTH];

unsigned, like:

unsigned char hash[SHA256_DIGEST_LENGTH];

help? (Especially in the signature of sha256_hash_string.)

Answer 2

You're printing out a signed char as an integer. If the byte is negative, it get's converted to a signed int (the default argument promotions in the call to sprintf), and then that gets converted to an unsigned int (via the %x format specifier) and printed out.

So, the byte A0 is -96 as a signed byte, which gets converted to -96 as a signed int, which is 0xFFFFFFA0 in hex, so it gets printed out as FFFFFFA0.

To fix this, case each byte to an unsigned char before printing:

sprintf(..., (unsigned char)hash[i]);

You're getting the warning about stack smashing because there's a signed byte near the end of the hash, so you're writing the 8 bytes FFFFFFB7 at offset 58 when you intended to only write 2 bytes. This results in a buffer overflow, which happens to be detected here because the compiler likely inserted a guard area or security cookie in the stack before the return value, and it detected that that guard area was inadvertently modified.