A modification to %esp cause SIGSEGV

Question

Sometimes I use the following code to avoid stack overflow when taking part in coding competition.

int main()
{
  static const int _STACK_SIZE = MAXN*10;
  s         


        

Answer 1

It's because in x86-64 assembly modifying a 32-bit register such as esp zeroes the highest 32 bits of the corresponding 64-bit register, rsp in this case. In x86-64 the stack pointer rsp is a 64-bit register.

x86-64 Linux always puts the stack pointer near the top of the user-space range of virtual address space, like 0x7fffffffe6d0 for example, so it's always outside the low 32 bits of virtual address space. The high half of RSP is non-zero and ESP != RSP.

So by modifying esp you make rsp point to an address where your program has no access rights, and thus cause a segmentation fault. In x86-64 code you don't normally use esp at all, and you should replace all instances of esp with rsp in your x86-64 code.