I was doing the mathematical calculations with python or octave because of availability of really nice functions and libraries at hand. But recently I gained interest in rub
SciRuby is working on this. NMatrix and Statsample are probably the most useful SciRuby gems for linear algebra and statistics, respectively, though you might also find these useful: distribution, minimization, and integration.
As far as visualization and plotting goes, check out Rubyvis (a Ruby port of Protovis). Nyaplot is also coming along nicely.
If you need something more stable than NMatrix, there's always NArray.
I'm also surprised no one has mentioned Ruby/GSL. There's also a fork which supports NMatrix over in the SciRuby github account.
Full disclosure: I'm an admin for SciRuby, but I don't make any money off of it. =)