How is the system call in Linux implemented?

Question

When I invoke a system call in user mode,how did the call get processed in OS?

Does it invoke some some executable binary or some standard library?

If yes,wh

Answer 1

It depends on what you mean by system call. Do you mean a C library call (through glibc) or an actual system call? C library calls always end up using system calls in the end.

The old way of doing system calls was through a software interrupt, i.e., the int instruction. Windows had int 0x2e while Linux had int 0x80. The OS sets up an interrupt handler for 0x2e or 0x80 in the Interrupt Descriptor Table (IDT). This handler then performs the system call. It copies the arguments from user-mode to kernel-mode (this is controlled by an OS-specific convention). On Linux, the arguments are passed using ebx, ecx, edx, esi, and edi. On Windows, the arguments are copied from the stack. The handler then performs some sort of lookup (to find the address of the function) and executes the system call. After the system call is completed, the iret instruction returns to user-mode.

The new way is sysenter and sysexit. These two instructions basically do all the register work for you. The OS sets the instructions up through the Model Specific Registers (MSRs). After that it's practically the same as using int.