Calculating pow(a,b) mod n

Question

I want to calculate a^b mod n for use in RSA decryption. My code (below) returns incorrect answers. What is wrong with it?

unsigned long i         


        

Answer 1

Calculating pow(a,b) mod n

1. A key problem with OP's code is a * a. This is int overflow (undefined behavior) when a is large enough. The type of res is irrelevant in the multiplication of a * a.

   The solution is to ensure either:

   • the multiplication is done with 2x wide math or
   • with modulus n, n*n <= type_MAX + 1

2. There is no reason to return a wider type than the type of the modulus as the result is always represent by that type.

   // unsigned long int decrypt2(int a,int b,int n)
   int decrypt2(int a,int b,int n)
   

3. Using unsigned math is certainly more suitable for OP's RSA goals.

---

Also see Modular exponentiation without range restriction

// (a^b)%n
// n != 0

// Test if unsigned long long at least 2x values bits as unsigned
#if ULLONG_MAX/UINT_MAX  - 1 > UINT_MAX
unsigned decrypt2(unsigned a, unsigned b, unsigned n) {
  unsigned long long result = 1u % n;  // Insure result < n, even when n==1
  while (b > 0) {
    if (b & 1) result = (result * a) % n;
    a = (1ULL * a * a) %n;
    b >>= 1;
  }
  return (unsigned) result;
}

#else
unsigned decrypt2(unsigned a, unsigned b, unsigned n) {
  // Detect if  UINT_MAX + 1 < n*n
  if (UINT_MAX/n < n-1) {
    return TBD_code_with_wider_math(a,b,n);
  }
  a %= n;
  unsigned result = 1u % n;
  while (b > 0) {
    if (b & 1) result = (result * a) % n;
    a = (a * a) % n;
    b >>= 1;
  }
  return result;
}

#endif