check if carry flag is set

Question

Using inline assembler [gcc, intel, c], how to check if the carry flag is set after an operation?

Answer 1

The first function performs unsigned addition and then tests for overflow using the carry flag (CF). The volatile's must remain. Otherwise the optimizer will rearrange instructions, which pretty much ensures an incorrect result. I've seen the optimizer change the jnc to a jae (which is also based on CF).

/* Performs r = a + b, returns 1 if the result is safe (no overflow), 0 otherwise */
int add_u32(uint32_t a, uint32_t b, uint32_t* r)
{
    volatile int no_carry = 1;
    volatile uint32_t result = a + b;

    asm volatile
    (
     "jnc 1f          ;"
     "movl $0, %[xc]  ;"
     "1:              ;"
     : [xc] "=m" (no_carry)
     );

    if(r)
        *r = result;

    return no_carry;
}

The next function is for the signed ints. Same use of volatile applies. Note that signed integer math jumps on OF flag via jno. I've seen the optimizer change this to a jnb (which is also based on OF).

/* Performs r = a + b, returns 1 if the result is safe (no overflow), 0 otherwise */
int add_i32(int32_t a, int32_t b, int32_t* r)
{   
    volatile int no_overflow = 1;
    volatile int32_t result = a + b;

    asm volatile
    (
     "jno 1f          ;"
     "movl $0, %[xo]  ;"
     "1:              ;"
     : [xo] "=m" (no_overflow)
     );

    if(r)
        *r = result;

    return no_overflow;
}

In the big picture, you might use the functions as follows. In the same big picture, many folks will probably reject the extra work and aesthetic non-beauty until pwn'd by an overflow/wrap/underflow

int r, a, b;
...

if(!add_i32(a, b, &r))
    abort(); // Integer overflow!!!

...

The inline GCC assembly is available in GCC 3.1 and above. See Assembler Instructions with C Expression Operands, or search for 'GCC Extended Assembly'.

Finally, the same in Visual Studio would be as follows (not much difference in code generation), but syntax is much easier since MASM allows you to jump to a C label:

/* Performs r = a + b, returns 1 if the result is safe (no overflow), 0 otherwise */
int add_i32(__int32 a, __int32 b, __int32* r)
{   
    volatile int no_overflow = 1;
    volatile __int32 result = a + b;

    __asm
    {
        jno NO_OVERFLOW;
        mov no_overflow, 0;
    NO_OVERFLOW:
    }

    if(r)
        *r = result;

    return no_overflow;
}

On the bad side, the above MASM code is only applicable for x86 assembly. For x64 assembly, there is no inlining so you will have to code it up in assembly (in a separate file) and use use MASM64 to compile.