You're trying too hard to solve something that isn't actually a problem. Use the header files you need, and reduce - WHERE POSSIBLE - the requirement for them. But don't try and take it to extremes because you'll fail.
In some cases, the PIMPL idiom may help you, but not here.