x86 find out operand size of instruction given only the hex machine code?

Question

For example, given a hex: 83 E4 F0

By looking at the intel developer's manual, I can figure out that 83 means and and FO means the -16. Looking at E4, I can decode that the source/destination register is either SP or ESP.

Therefore, I can conclude that the hex means either and $-16, %ESP or and $-16, %SP. However, in the manual, both of those are listed as 83 /4 ib.

How can I differentiate between those two?

Answer 1

As harold says, the default operand size is not encoded in the instruction but depends on the current processor mode.

In real mode and 16-bit protected mode, the default operand size is 16-bit, so 83 E4 F0 decodes to and $-16, %sp.

In 32-bit mode operand size defaults to 32-bit, so it's and $-16, %esp.

In x64 mode, most instructions again default to 32-bit operand size (except branches and those that indirectly use the stack, such as pushes, pops, calls and returns), so it again decodes to and $-16, %esp.

It is possible to override the default operand size using prefixes. For example, prefix 66h switches between 32-bit and 16-bit operand size, so 66 83 E4 F0 decodes to and $-16, %esp in 16-bit mode and to and $-16, %sp in 32-bit or 64-bit mode. To get 64-bit operand size, you need to use the REX prefix with the W bit set, so 48 83 E4 F0 decodes to and $-16, %rsp (but only in 64-bit mode!).

Answer 2

Under protected mode, it can only be the 32bit version, both 16 and 64 bit versions require a prefixed size override byte, in this case the 16bit version requires the 0x66 prefix override, so you get 66:83 E4 F0. Intel clearly states this in the description for AND:

In 64-bit mode, the instruction’s default operation size is 32 bits.

and the reference for 066H, Chapter 2.2.1:

The operand-size override prefix allows a program to switch between 16- and 32-bit operand sizes. Either size can be the default; use of the prefix selects the non-default size.