Since you mention high-level operations on matrices and vectors, ATLAS, Intel's MKL, PLASMA, and FLAME may be of interest.
Some C++ matrix math libraries include uBLAS from Boost, Armadillo, Eigen, IT++, and Newmat. The POOMA library probably also includes some of these things. This question also refers to MTL.
If you're looking for lower-level portability primitives, a colleague of mine has developed a wrapper around SSE2, Altivec, VSX, Larrabee, and Cell SPE vector operations. It can be found in our source repository, but its licensing (academic) may not be appropriate if you want to distribute it as part of your work. It is also still under significant development to cover the range of application needs that it's targeted at.