Got confused with a vector indexed by a matrix, in Matlab

That's a very standard MATLAB operation that you're doing. When you have a vector or a matrix, you can provide another vector or matrix in order to access specific values. Accessing values in MATLAB is not just limited to single indices (i.e. A(1), A(2) and so on).

For example, what you have there is a vector of a = [1 2 3 4]. When you try to use b to access the vector, what you are essentially doing is a lookup. The output is basically the same size as b, and what you are doing is creating a matrix where there are 3 rows, and each element accesses the first, second and third element. Not only can you do this for a vector, but you can do this for a matrix as well.

Bear in mind that when you're doing this for a matrix, you access the elements in column major format. For example, supposing we had this matrix:

 A = [1 2
      3 4
      5 6
      7 8]

A(1) would be 1, A(2) would be 3, A(3) would be 5 and so on. You would start with the first column, and increasing indices will traverse down the first column. Once you hit the 5th index, it skips over to the next column. So A(5) would be 2, A(6) would be 4 and so on.

Here are some examples to further your understanding. Let's define a matrix A such that:

 A = [5 1 3
      7 8 0
      4 6 2]

Here is some MATLAB code to strengthen your understanding for this kind of indexing:

 A = [5 1 3; 7 8 0; 4 6 2]; % 3 x 3 matrix
 B = [1 2 3 4];
 C = A(B); % C should give [5 7 4 1]
 D = [5 6 7; 1 2 3; 4 5 6];
 E = A(D); % E should give [8 6 3; 5 7 4; 1 8 6]
 F = [9 8; 7 6; 1 2];
 G = A(F); % G should give [2 0; 3 6; 5 7]

As such, the output when you access elements this way is whatever the size of the vector or matrix that you specify as the argument.

In order to be complete, let's do this for a vector:

 V = [-1 9 7 3 0 5]; % A 6 x 1 vector
 B = [1 2 3 4];
 C = V(B); % C should give [-1 9 7 3]
 D = [1 3 5 2];
 E = V(D); % E should give [-1 7 0 9]
 F = [1 2; 4 5; 6 3];
 G = V(F); % G should give [-1 9; 3 0; 5 7]

NB: You have to make sure that you are not providing indexes that would make the accessing out of bounds. For example if you tried to specify the index of 5 in your example, it would give you an error. Also, if you tried anything bigger than 9 in my example, it would also give you an error. There are 9 elements in that 3 x 3 matrix, so specifying a column major index of anything bigger than 9 will give you an out of bounds error.

Notice that the return value of a(b) is the same size as b.

a(b) simply takes each element of b, call it b(i,j), as an index and returns the outputs a(b(i,j)) as a matrix the same size as b. You should play around with other examples to get a more intuitive feel for this:

b = [4 4 4; 4 4 4];
a(b) % Will return [4 4 4; 4 4 4]
c = [5; 5];
a(c) % Will error as 5 is out of a's index range