Simple XOR encryption routine in C/C++

Question

I'm trying to encrypt/decrypt a file using XOR. I have the following encryption/decryption routine where every byte is xor'd and the result is being subtracted by the value of the byte that is located at the previous location. The ASM representation is as follows

crypt:
mov dl, [eax+ecx]   ; read byte
xor dl, 0C5h        ; xor it with oxC5
sub dl, [eax+ecx-1] ; sub the previous byte
mov [eax+ecx], dl   ; save the new byte
dec eax             ; decrement pointer
test   eax, eax
jg   short crypt     ;

That is what my encryption routine should look like, I'm trying to port this this C/C++. My code is as follows

#include <stdio.h>

unsigned int xorkey = 0xC5;

int main(int argc, char *argv[])
{
if(argc < 3)
{
    printf("usage: encoder input output\n");
    return -1;
}

FILE *in = fopen(argv[1], "rb");
if(in == NULL)
{
    printf("failed to open: %s", argv[2]);
    return -1;
}

FILE *out = fopen(argv[2], "wb");

if(out == NULL)
{
    fclose(in);
    printf("failed to open '%s' for writing.",argv[2]);
    return -1;
}

int count;
char buffer[1024];

while(count = fread(buffer, 1, 1024, in))
{
    int i;
    int end = count;

    for(i = 0;i < end; ++i)
    {
            ((unsigned int *)buffer)[i] ^= xorkey;
    }
    if(fwrite(buffer, 1, count, out) != count)
    {
            fclose(in);
            fclose(out);

            printf("fwrite() error\n");

            return -1;
    }
}

fclose(in);
fclose(out);

return 0;
}

I cannot figure out how to do the subtracting of bytes in C++. The XOR routine itself looks correct though, no? Please note that I'm also trying to encrypt the file starting from the end of the file till the beginning. Any ideas?

Thanks!

Answer 1

Here's how you could write that assembly language function in C. I've kept the variable names the same as the register names so you can see how the various parts match up.

void do_xor_crypt(char *buffer, int count) {
    char *ecx = buffer;
    int eax = count - 1;
    if (eax > 0) {
        do {
            char dl = ecx[eax];
            dl ^= 0xC5;
            dl -= ecx[eax-1];
            ecx[eax] = dl;
            eax--;
        } while (eax > 0);
    }
}

Note that I have checked to make sure eax is greater than zero (meaning count is two or more) so that the loop has something to subtract. You could integrate this code into your reading loop like:

while (count = fread(buffer, 1, 1024, in))
{
    do_xor_crypt(buffer, count);
    if (fwrite(buffer, 1, count, out) != count)
    {
        // ...
    }
}

Answer 2

There are a couple things wrong with your C code.

The asm code starts at the end of the buffer and works its way down and stops when eax == 0. The asm code operates on a byte at a time, xor'ing and subtracting from the previous byte.

The asm code would seem to leave the first byte of the buffer untouched.

Your C code moves an index and xors the four bytes pointed by that byte index with 0xC5. That code is reading three bytes too many and only affecting the lowest byte with the XOR.

Plus your for-loop starts at the front and works its way to the end - the opposite of your asm routine.

Assuming chars are byte-sized, then to mimic the asm routine, your subtraction step would be:

buffer[i] = buffer[i] - buffer[i-1];

which can be rewritten as:

buffer[i] -= buffer[i-1];

...assuming you fix your for-loop to go from the end-1 of the array to index 1.

Answer 3

You need to change buffer to type unsigned char, and change your for loop to:

for (i = count - 1; i > 0; i--)
{
    buffer[i] ^= xorkey;
    buffer[i] -= buffer[i - 1];
}

Note though that this code works on the file in 1024-byte chunks from the start, and then works on each chunk in reverse. If you want to work on the whole file in reverse, you'll need to start reading from the end of it, and have special handling for the first character in each chunk.