Why is array broadcasting far slower than matrix inverses in Julia?

Question

My code essentially finds the column vector k using this equation:

Calculating the matrix takes three orders of magnitude longer than calculating