DTMC Markov Chain - How to get the stationary vector

问题

For a Discrete Time Markov Chain problem, i have the following:

1) Transition matrix:

0.6  0.4  0.0  0.0
0.0  0.4  0.6  0.0
0.0  0.0  0.8  0.2
1.0  0.0  0.0  0.0

2) Initial probability vector:

1.0  0.0  0.0  0.0

So, i wrote the following SciLab code to get to the stationary vector:

P = [0.6, 0.4, 0, 0; 0, 0.4, 0.6, 0; 0, 0, 0.8, 0.2; 1,0,0,0]
PI = [1,0,0,0]
R=PI*P
count=0;
for i = 1 : 35  // stationary vector is obtained at iteration 33, but i went futher to be sure
    R=R*P;
    count=count+1
    disp("count = "+string(count))
end

PI // shows initial probability vector
P // shows transition matrix
R // shows the resulting stationary vector

After iteration number 33, the following resulting stationary vector is obtained:

0.2459016    0.1639344    0.4918033    0.0983607

What manual calculations do i have to perform in order to get to the stationary vector above without having to multiply the transition matrix 33 times then multiply the result by the initial vector?

I was told that the calculations are quite simple but i just could not realize what to do even after reading some books.

Of course explanations are welcome, but above all things i would like to have the exact answer for this specific case.

Thanks for all your help!

La-Roque

回答1:

You can solve DTMC on Octave by using this short code:

P = [
  0.6, 0.4,   0,   0;
    0, 0.4, 0.6,   0;
    0,   0, 0.8, 0.2;
    1,   0,   0,   0
  ]

pis = [P' - eye(size(P)); ones(1, length(P))] \ [zeros(length(P), 1); 1]

Or with SAGE with this code:

P = matrix(RR, 4, [
  [0.6, 0.4,   0,   0],
  [  0, 0.4, 0.6,   0],
  [  0,   0, 0.8, 0.2],
  [  1,   0,   0,   0]
  ])

I = matrix(4, 4, 1); # I; I.parent()
s0, s1, s2, s3 = var('s0, s1, s2, s3')
eqs = vector((s0, s1, s2, s3)) * (P-I); eqs[0]; eqs[1]; eqs[2]; eqs[3]
pis = solve([
  eqs[0] == 0,
  eqs[1] == 0,
  eqs[2] == 0,
  eqs[3] == 0,
  s0+s1+s2+s3==1], s0, s1, s2, s3)

On both, the result of the steady state probabilities vector is:

I hope it helps.

WBR, Albert.

来源：https://stackoverflow.com/questions/16888303/dtmc-markov-chain-how-to-get-the-stationary-vector

标签

markov-chains