x86 Calling Conventions
Calling conventions act as a contract between subroutines at the assembly level. They describe how and where arguments are passed and who is responsible for cleaning up the stack. Closely related considerations like which registers can be “clobbered”, which must be preserved and where return values are located are associated with a specific application binary interface (ABI). There’s an important distinction to be made between the responsibilities of an ABI and a calling convention, but those details are saved for a future article on ABIs in general.
I imagine it’s possible to have a long fulfilling career in C/C++ development without ever truly understanding the various calling conventions and their nuances as they are a concern of the compiler and generally not the developer (unless they are doing some low level debugging).
As a reverse engineer working on binary targets, it’s helpful to understand calling conventions and learn how to spot them in the code you’re reversing. Here we look at various calling conventions, the properties that make them unique and view some samples of what they look like when disassembled.
Caller/Callee
You’re going to see the terms caller and callee used a lot here which can be a little confusing. The caller is simply a function that calls another function. The callee is a function that is called from another function. Consider the following simple C program that contains two functions, main and printFavNums.
In the code above the caller is main and the callee is printFavNums. Why? Because main calls another function (printFavNums) and printFavNums is called from another function (main).
Who decides which calling convention is used?
The compiler determines which calling convention is used for all functions. Many compilers, however, allow the programmer to set specific calling conventions on a per-function basis as we’ll do with GCC in this article.
It’s important to remember that these are just conventions and a compiler has the choice to obey or disobey any convention. The calling conventions you’ll encounter are typically a result of the default calling convention set by the compiler, or a calling convention officially adopted by an API, like the Win32 API’s use of stdcall.
x86
The x86 architecture has numerous adopted calling conventions prior to 64-bit processors. Here we’ll take a look at some common and not so common 32-bit calling conventions, their defining characteristics and what they look like in Intel-flavor disassembly.
| Convention | Stack cleanup | Parameter passing |
|---|---|---|
| cdecl | Caller | Pushes parameters on the stack, in reverse order (right to left) |
| fastcall | Callee | First two in registers, the rest on the stack in reverse order |
| stdcall | Callee | Pushes parameters on the stack, in reverse order |
| thiscall | Callee | First param in ECX (usually this), the rest on the stack in reverse order |
cdecl
The cdecl (pronounced see-dec-el, short for “C declare”) is extremely common as it’s the default 32-bit x86 calling convention on today’s most popular compilers including GCC, MSVC, and LLVM. It has the following properties:
- The caller places all arguments to the callee on the stack
- Arguments are pushed to the stack from right to left
- Stack cleanup is performed by the caller
Let’s consider the simple C program we referenced earlier. Here it is again. We don’t need to explicitly set cdecl because it’s the default calling convention of GCC, the compiler we’re using here.
We use -m32 to tell GCC we want a 32-bit binary and -mpreferred-stack-boundary=2 to tell GCC we want 22 (4-byte) stack alignment. We set this stack alignment because it’s much easier to confirm who cleans up the stack. Without setting this option the ABI dictates a 24 (16-byte) stack alignment, which just adds unnecessary confusion. We’ll talk about all this in a future ABI article.
Let’s take a look at this simple program (just main and printFavNums) disassembled with Binary Ninja and look for the cdecl characteristics we expect to see.
Here we can see main (caller) is pushing arguments for printFavNums (callee) to the stack from right to left. It first pushes 8 then 2 and calls printFavNums. This confirms ‘the caller places all arguments to the callee on the stack’ and ‘arguments are pushed to the stack from right to left’.
But what about the last characteristic? Who is cleaning up the stack here? If you look just after the call to printFavNums, main cleans up 8-bytes of the stack frame, which accounts for both 4-byte arguments passed to the callee. This confirms the caller, main is cleaning up the stack for the callee. But just to make sure there’s no funny business here, let’s look at printFavNums.
We can see the three 4-byte arguments to printf are pushed to the stack from right to left and after the call is made, the caller, printFavNums cleans up the stack for the callee by adding 0xC (12) to the stack frame, just as we’d expect from cdecl. printFavNums never accounts for its own parameters when cleaning the stack frame.
fastcall
The fastcall calling convention gets its name because it’s faster than other calling conventions that pass their parameters via the stack. On 32-bit systems, fastcall has a limited number of registers to utilize for argument passing (compared to x86-64 and some other architectures). Because of this, only the first 2 arguments are passed by register, while any remaining arguments are passed on the stack. fastcall has the following properties:
- The caller places the first 2 arguments to the callee in registers, the rest on the stack
- Arguments are loaded to registers / pushed to the stack from right to left
- Stack cleanup is performed by the callee
Let’s consider a program very similar to the simple C program we referenced earlier. This time we’ve added one more parameter and we explicitly set fastcall as a function attribute (this is GCC-specific syntax).
Let’s take a look at what the compiler did in this version of our simple program, again disassembled with Binary Ninja.
Here we can see main (caller) is pushing the last argument to the stack, then loading the second and first arguments into the edx and ecx registers respectively. This confirms that in fastcall the *caller places the first 2 arguments to the callee in registers, the rest on the stack* and arguments are processed right to left.
Notice how main does not clean up the stack after calling printFavNums. Let’s take a look in printFavNums to make sure it cleans up its own parameters.
If we look below the call to printf (which is still cdecl) we see printFavNums cleaning up 0x10 (16-bytes) to account for the four 4-byte parameters passed to printf. But where is printFavNums cleaning up its own three 4-byte parameters? Well, remember that in fastcall two of those three were passed by register which don’t need to be cleaned up. This leaves only one 4-byte parameter to clean up, but where is that done?
The printFavNums function does this by using an optional operand to the retn instruction, of 0x4. This causes an extra 4-bytes to be cleaned up during the retn operation. It certainly looks a lot different than we’ve seen so far, with the stack pointer (esp) being directly manipulated, but the effect is the same. Hence, the callee cleans up after itself.
stdcall
stdcall (short for “standard call”) is a common x86 calling convention similar to cdecl however the callee is responsible for cleaning its own arguments off the stack. If you’ve ever done any Windows development you probably know this calling convention as WINAPI which is just an alias for stdcall set in windef.h (#define WINAPI __stdcall). This calling convention has the following properties:
- The caller places all arguments to the callee on the stack
- Arguments are pushed to the stack from right to left
- Stack cleanup is performed by the callee
Let’s consider the same simple C program we’ve been using, this time with stdcall set for printFavNums.
Now let’s check out our two disassembled functions.
As we expect, main pushes the arguments to printFavNums to the stack from right to left. Notice how main does not clean up the stack after calling printFavNums.
Taking a look in printFavNums we can see retn 0x8 which accounts for the two 4-byte arguments passed into the function. As we expected, this stdcall function cleans up after itself.
thiscall
The thiscall calling convention is a pretty interesting beast. Its intent is to be used on C++ class member functions that need to reference their “this pointer”. This is so the function can access class instance variables. Here are the properties for this calling convention:
- Arguments are pushed to the stack from right to left
- The this pointer is passed to the callee via the ECX register (not on the stack)
- Stack cleanup is performed by the callee
So this is sort of a hybrid of fastcall and stdcall. I say that because a register is used to pass a parameter (like fastcall, though only one not two) and the callee cleans up the stack (like stdcall).
Let’s consider a C++ version of the C program we’ve been using where printFavNums is a member of the FavoriteNumbers class.
Based on, well, everything I’ve never read and known about thiscall, GCC should make printFavNums a thiscall function. So let’s see…
Wait a minute. There’s no use of ecx for a this pointer, and the stack is clearly being cleaned by the caller main. printFavNums fits the description of cdecl to a tee. So what happened here? This one really threw me for a loop.
It turns out G++ made this function a cdecl function but inserted the this pointer as the first argument to pritnFavNums, so the function actually has three arguments even though the prototype only has two.
Let’s use that same code, but this time we’ll explicitly set printFavNums to thiscall.
Now we’ll compile and disassemble it to see what happened this time.
That’s more like it! The printFavNums function is passed two arguments on the stack, and ecx is loaded with the this pointer just before the call. We can also see that main is not cleaning up the stack of arguments after the call to printFavNums. printFavNums is clearly a thiscall function!
Special Considerations
The subject of calling conventions can become complex under certain conditions. In many cases the information presented up to this point holds true, but in some cases, there are catches you must consider. Here we discuss some of those special cases.
Variable Argument Functions
In this article, you’ve learned about some calling conventions that are “callee cleanup”, meaning the callee cleans up the stack itself, but there’s a catch. Callees cannot cleanup their own stack if they are passed a variable number of functions!
Variable argument functions (also called “vararg” or “variadic” functions) are functions that take a variable number of arguments. And example of a vararg function is given below.
The above code will compile with no issues because the default calling convention is cdecl, a caller cleanup convention. However, if we attempt to set this function as any of the callee cleanup functions we’ve learned about, say stdcall…
… the compiler will throw a warning and fall back to cdecl because it cannot make a variadic function a callee cleanup convention.
Calling thiscall Anytime!
So we learned about the intended use of thiscall which is intended to be used with C++ class member functions that need access to their own class instance. But thiscall can be used wherever you want. In fact, this code is perfectly valid.
You might think the compiler (GCC 5.4.0) would complain, I mean, this isn’t even C++ and there’s no class in sight. Why would we need thiscall? But it works, check out the disassembly.
Since there’s no this pointer the compiler passes arg1 through ecx and arg2 on the stack. The callee is cleaning up its own arguments from the stack as we expect in thiscall functions.
来源：oschina
链接：https://my.oschina.net/zengfr/blog/4268203