x86_64 Assembly Linux System Call Confusion

Question

I am currently learning Assembly language on Linux. I have been using the book \'Programming From the Ground Up\' and all the examples are 32-bit. My OS is 64-bit and I have

Answer 1

duskwuff's answer points out correctly the mechanism for system calls is different for 64-bit x86 Linux versus 32-bit Linux.

However, this answer is incomplete and misleading for a couple reasons:

• The change was actually introduced before 64-bit systems became popular, motivated by the observation that int 0x80 was very slow on Pentium 4. Linus Torvalds coded up a solution using the SYSENTER/SYSEXIT instructions (which had been introduced by Intel around the Pentium Pro era, but which were buggy and gave no practical benefit). So modern 32-bit Linux systems actually use SYSENTER, not int 0x80.
• 64-bit x86 Linux kernels do not actually use SYSENTER and SYSEXIT. They actually use the very similar SYSCALL/SYSRET instructions.

As pointed out in the comments, SYSENTER does not actually work on many 64-bit Linux systems—namely 64-bit AMD systems.

It's an admittedly confusing situation. The gory details are here, but what it comes down to is this:

For a 32bit kernel, SYSENTER/SYSEXIT are the only compatible pair [between AMD and Intel CPUs]

For a 64bit kernel in Long mode only… SYSCALL/SYSRET are the only compatible pair [between AMD and Intel CPUs]

It appears that on an Intel CPU in 64-bit mode, you can get away with using SYSENTER because it does the same thing as SYSCALL, however this is not the case for AMD systems.

Bottom line: always use SYSCALL on Linux on 64-bit x86 systems. It's what the x86-64 ABI actually specifies. (See this great wiki answer for even more details.)