Ok, I think none of the answers below tell the whole story, so here's my take on a response to my question:
LLVM compiles code faster than GCC, may create code that runs faster, and the Clang frontend provides more accurate error messages than GCC – so there are definitely reasons for switching;
that said, the version provided with latest stable Xcode (LLVM 1.6) is not 100% stable yet, you can run into some minor bugs if you're unlucky. So if you want to be safe, you should either compile the latest LLVM (2.0) from source, or stick to GCC for the next few months;
in a few months, probably when Apple releases Xcode 4, LLVM 2.0 will be the version that ships with Xcode by default, and then we should all be able to switch to it safely.
Thanks to everyone who responded, feel free to correct me if I got something wrong.
