x86 Assembly: INC and DEC instruction and overflow flag

Question

In x86 assembly, the overflow flag is set when an add or sub operation on a signed integer overflows, and the carry flag is set when an operation o

Answer 1

The overflow flag is set when an operation would cause a sign change. Your code is very close. I was able to set the OF flag with the following (VC++) code:

char ovf = 0;

_asm {
    mov bh, 127
    inc bh
    seto ovf
}
cout << "ovf: " << int(ovf) << endl;

When BH is incremented the MSB changes from a 0 to a 1, causing the OF to be set.

This also sets the OF:

char ovf = 0;

_asm {
    mov bh, 128
    dec bh
    seto ovf
}
cout << "ovf: " << int(ovf) << endl;

Keep in mind that the processor does not distinguish between signed and unsigned numbers. When you use 2's complement arithmetic, you can have one set of instructions that handle both. If you want to test for unsigned overflow, you need to use the carry flag. Since INC/DEC don't affect the carry flag, you need to use ADD/SUB for that case.