I think Eigen is what you're looking for.
http://eigen.tuxfamily.org/index.php?title=Main_Page
It is a headers only library and compiles on many compilers. It even uses exotic assembly for faster math.
This is the page that shows off the linear solver api.
http://eigen.tuxfamily.org/dox-2.0/TutorialAdvancedLinearAlgebra.html
It has a few solvers with a simple api.