I use MPI and like it very much. It does force you to think about the memory hierarchy, but in my experience, thinking about such things is important for high performance anyway. In many cases, MPI can largely be hidden behind domain-specific parallel objects (e.g. PETSc for solving linear and nonlinear equations).
