How cmp assembly instruction sets flags (X86_64 GNU Linux)

Question

Here is a simple C program:

void main()
{
       unsigned char number1 = 4;
       unsigned char number2 = 5;

       if (number1 < number2)
       {
               number1 = 0;
       }
}

So here we are comparing two numbers. In assembly it will be done using cmp. cmp works by subtracting one operand from other.

Now how cmp is subtracting operands? Is it subtracting 1st operand from 2nd or vice versa? In any case, this should go like this:

case # 1:

4 - 5 = (0000 0100 - 0000 0101) = (0000 0100 + 1111 1010 + 1) = (0000 0100 + 1111 1011)

= 1111 1111 = -1

So since the sign bit = 1 so SF should be 1.

No carry, so CF should be = 0.

case # 2:

5 - 4 = (0000 0101 - 0000 0100) = (0000 0101 + 1111 1011 + 1)

= (0000 0101 + 1111 1100) = 1 0000 0001

so here, CF should be = 1

since result is positive, SF should be = 0

Now I compile and run program (linux x86_64, gcc, gdb), place a breakpoint after cmp instruction to see register states.

Breakpoint hit after cmp:

Breakpoint 2, 0x0000000000400509 in main ()
(gdb) disassemble
Dump of assembler code for function main:
   0x00000000004004f6 <+0>:     push   %rbp
   0x00000000004004f7 <+1>:     mov    %rsp,%rbp
   0x00000000004004fa <+4>:     movb   $0x4,-0x2(%rbp)
   0x00000000004004fe <+8>:     movb   $0x5,-0x1(%rbp)
   0x0000000000400502 <+12>:    movzbl -0x2(%rbp),%eax
   0x0000000000400506 <+16>:    cmp    -0x1(%rbp),%al
=> 0x0000000000400509 <+19>:    jae    0x40050f <main+25>
   0x000000000040050b <+21>:    movb   $0x0,-0x2(%rbp)
   0x000000000040050f <+25>:    pop    %rbp
   0x0000000000400510 <+26>:    retq
End of assembler dump.

Register dump after cmp has been executed:

(gdb) info reg
rax            0x4  4
rbx            0x0  0
rcx            0x0  0
rdx            0x7fffffffe608   140737488348680
rsi            0x7fffffffe5f8   140737488348664
rdi            0x1  1
rbp            0x7fffffffe510   0x7fffffffe510
rsp            0x7fffffffe510   0x7fffffffe510
r8             0x7ffff7dd4dd0   140737351863760
r9             0x7ffff7de99d0   140737351948752
r10            0x833    2099
r11            0x7ffff7a2f950   140737348041040
r12            0x400400 4195328
r13            0x7fffffffe5f0   140737488348656
r14            0x0  0
r15            0x0  0
rip            0x400509 0x400509 <main+19>
eflags         0x297    [ CF PF AF SF IF ]
cs             0x33 51
ss             0x2b 43
ds             0x0  0
es             0x0  0
fs             0x0  0
gs             0x0  0
(gdb)

So we can see that after cmp has been executed, both CF=1, SF=1.

So the actual resulted flags (CF=1 & SF=1) are not equal to flags we calculated in

Case # 1 (CF=0 & SF=1) or case # 2 (CF=1 & SF=0)

Whats happening then? How cmp is actually setting the flags?

you're mistaken the values of CF ... what makes you think that 4-5 is "no carry"? (and vice-versa), that's simply wrong! — Tommylee2k
@Tommylee2k yes i was doing it wrong. Thanks for pointing that out. — abs

Johan Johan · Accepted Answer · 2017-05-08T09:30:03

Operation of CMP
CMP performs a subtraction but does not store the result.
For this reason the effect on the flags is exactly the same between:

cmp eax,ecx
sub eax,ecx

As per the documentation:

Operation
temp ← SRC1 − SignExtend(SRC2);
ModifyStatusFlags; (* Modify status flags in the same manner as the SUB instruction*)
Flags Affected
The CF, OF, SF, ZF, AF, and PF flags are set according to the result.

Effects on the flags
So the following flags are affected like so:

Assume result = op1 - op2

CF - 1 if unsigned op2 > unsigned op1
OF - 1 if sign bit of OP1 != sign bit of result
SF - 1 if MSB (aka sign bit) of result = 1
ZF - 1 if Result = 0 (i.e. op1=op2)
AF - 1 if Carry in the low nibble of result
PF - 1 if Parity of Least significant byte is even

I suggest you read up on the OF and CF here: http://teaching.idallen.com/dat2343/10f/notes/040_overflow.txt

Order of the operands
I see that you like pain and are using the braindead variant of x86 assembly called ATT-syntax.
This being the case you need to take into account that

CMP %EAX, %ECX  =>  result for the flags = ECX - EAX
CMP OP2, OP1    =   flags = OP1 - OP2

Whereas Intel syntax is

CMP ECX, EAX    =>  result for the flags = ECX - EAX
CMP OP1, OP2    =>  flags = OP1 - OP2

You can instruct gdb to show you Intel syntax using: set disassembly-flavor intel