'Mathematics for Computer Graphics Applications' is an introductory level textbook and takes a classroom appropriate approach to all of the basic math that you need to be acquainted with for 3D programming (matrices and vectors mostly)
And a note regarding quaternions: They're very useful for certain applications. SLERP (Spherical Linear intERPolation) can be very handy for producing smooth/appealing camera movements (among other things). SLERP is a pain (expensive) to do with matrices, but cheap and easy with Quaternions. Learn to use and love them - even if you don't fully understand them.
