Fredrik - the short answer is that, yes, you must learn Matrices and Vectors as they are the mathematical underpinnings for 3D work.
While Linear algebra is definitely not doctorate-level mathematics, it will take a bit of work. To get started, check out this book on Amazon: it looks like it is exactly what you are looking for. I haven't read this particular book (the one I used in grad school is a bit out of date) but it is particularly well rated.
One other thing: there are various 3D modeling engines that do this work for you on the market. The most famous of these is arguably the Source Engine from Valve. You can use this Engine (built for HalfLife2 & CounterStrike) to create some pretty sophisticated games while working above the level of 3D modeling. In fact, one of the most popular games on the Steam network, Garry's mod started with someone just playing with cool things you can do with the Steam Engine. Here's a link to a site that provides tutorials for building your own worlds using the Source Engine in case you are interested.
