Finding the nth term in Fibonacci series f(n) = f(n-1) + f(n-2) can be solved in O(n) time by memoization.
A more efficient way would be to find the nth power of mat
As you are already suspecting, this will work very similar. Use the n-th power of the x * x matrix
|1 0 0 0 .... 1 1|
|1
| 1
| 1
| 1
| 1
...................
...................
| ... 1 0|
This is easy to understand if you multiply this matrix with the vector
f(n-1), f(n-2), ... , f(n-x+1), f(n-x)
which results in
f(n), f(n-1), ... , f(n-x+1)
Matrix exponentiation can be done in O(log(n)) time (when x is considered to be constant).
For the Fibonacci recurrence, there is also a closed formula solution, see here http://en.wikipedia.org/wiki/Fibonacci_number, look for Binet's or Moivre's formula.