I'm more involved in the managed server optimization in my company but I may have a few tips for you. First, you can look at the code more closely using the code tracing feature of Zend server. It will allow you to see where and when the things get dirty.
I totally share benlumley's consideration regarding the cache. Most of the sites we host doesn't even have the block caching enable. This cache has to be explicitly called and not "assumed". So if you code hasn't yet took part of this mechanism, it's something you definitely want to try. If you have a EE version, you can get the Full page up in order to get the best of the beast.
A reverse proxy will also help a lot. It'll cache the static ressources, significantly lowering the pressure on the php interpretation stack of your front servers.
Don't forget to write the sessions & Magento cache to a RAM disk. This will also definitely get you to another level of performances.
There's still a lot to be said here but I'm running out of time. You have to know that a good site, well coded in a 1.4.1 CE version, running on a 2x5650 Xeon + 16 GB RAM server and having a Rproxy on top can take up to 50 000 unique visitors a day with smooth pages to everybody.