Code to illustrate :
int i = 5;
object obj = i;
byte b = (byte)obj; // X
When run, this generates a System.InvalidC
MSDN explicitly says that unboxing to a different type will throw an InvalidCastException.
My understanding is that the type to which a variable is unboxed is actually a parameter to the underlying CIL assembly command. It is unbox opcode that actually throws the InvalidCastException.
InvalidCastException is thrown if the object is not boxed as valType.
The difference in behaviour you're seeing is the difference between identity and representation.
Unboxing is an identity cast, and a representation-preserving operation. Casting an int to a byte, however, is representation-changing (since there is a potential loss of precision).
You get an InvalidCastException when you try to unbox the int as a byte because the identity of the boxed value is not a byte, it is an int. When you write byte b = (byte)obj, you are telling the runtime, I know that what's in there is a byte, but what you really mean to say is, I think that what's in there can be converted to a byte.
In order to make the latter statement, you first have to declare the identity of the object, which is an int. Then and only then can you make a representation-changing conversion to byte.
Note that this applies even if the target type is "larger" - i.e. an Int64. All explicit conversions for which the destination type is not in the inheritance tree of the source type are considered to be representation-changing. And since all types derive from System.Object, unboxing by definition cannot change the representation.