Stack resident buffer overflow on 64-bit?

Question

I\'m studying some security related things and right now I\'m playing around with my own stack. What I\'m doing should be very trivial, I\'m not even trying to execute the s

Answer 1

Those two instructions are doing exactly what you expect them to do. You have overwritten the previous stack frame with 0x41's so when you hit the leaveq, you are doing this:

mov rsp, rbp
pop rpb

Now rsp points to where rbp did before. However, you have overwritten that region of memory, so when you do the pop rbp, the hardware is essentially doing this

mov rbp, [rsp]
add rsp,1

But [rsp] now has 0x41's. So this is why you're seeing rbp get filled with that value.

As for why rip isn't getting set like you expect, it's because ret is setting the rip to 0x41 and then generating an exception (page fault) on the instruction fetch. I wouldn't rely on GDB to show the right thing in this case. You should try overwriting the return value with a valid address within the program's text segment and you likely won't see this weird behavior.