So I\'ve been working recently on an implementation of the Miller-Rabin primality test. I am limiting it to a scope of all 32-bit numbers, because this is a just-for-fun pr
Exponentiation by squaring still "works" for modulo exponentiation. Your problem isn't that 2 ^ 168277
is an exceptionally large number, it's that one of your intermediate results is a fairly large number (bigger than 2^32), because 673109 is bigger than 2^16.
So I think the following will do. It's possible I've missed a detail, but the basic idea works, and this is how "real" crypto code might do large mod-exponentiation (although not with 32 and 64 bit numbers, rather with bignums that never have to get bigger than 2 * log (modulus)):
Obviously that's a bit awkward if your C++ implementation doesn't have a 64 bit integer, although you can always fake one.
There's an example on slide 22 here: http://www.cs.princeton.edu/courses/archive/spr05/cos126/lectures/22.pdf, although it uses very small numbers (less than 2^16), so it may not illustrate anything you don't already know.
Your other example, 4000111222 ^ 3 % 1608
would work in your current code if you just reduce 4000111222
modulo 1608
before you start. 1608
is small enough that you can safely multiply any two mod-1608 numbers in a 32 bit int.