Say I have 12 registers. How many bits must be reserved in the machine code instruction order to address any of these 12 registers?
It takes ceil(log2(12)) bits to encode 1 of 12 possibilities in a normal fixed-width field.
But does having fewer than a power of 2 registers let us save any bits in an instruction with multiple register operands? Yes, in this case.
A 3-register instruction has 12^3 = 1728 possible permutations of registers. But using 3 separate 4-bit fields would give us 2^(4*3) = 4096 possible encodings. So there's 1 redundant bit, because 2^11 = 2048 which is still greater than 1728. However, encoding all 3 register selectors into one 11-bit field would require much more complex decoding.
A 2-register instruction needs 12^2 = 144 unique register encodings. But here, 2^(4*2) = 256, and the next lowest power of 2 (128) isn't big enough.
Probably your best bet is to use 4-bit fields, and use the 13..15 register encodings for something else. e.g. an escape code that means it's actually a different instruction. Or if you don't need that much instruction coding space, simplify the decoders and leave your instruction format redundant.
Or really your best bet is to have a power-of-2 number of registers, so you don't waste coding space. There's a reason that basically every modern register machine has a power-of-2 number of registers.