Doxygen is a bit slow - it takes about a couple of minutes to process my whole project, so for small incremental changes this is longer than actually building the rest of my
I don't think having Doxygen run on a normal dev cycle is a good idea. Our Doxygen build runs as part of our Continuous Integration server's responsibilities.
That said, there are some benefits of running doxygen every build to catch missing docs. So I would trim the doxygen config for dev builds removing diagrams, and even stop apple importing it into xcode.