I want to use the pimpl idiom to avoid having users of my library need our external dependencies (like boost, etc) however when my class is templated that seems to be imposs
There are two general solutions:
while the interface depends on some type T, it defers to a more weakly typed implementation (e.g. one using void* pointers directly or trough type erasure), or
you support only a specific and quite limited number of types.
The second solution is relevant for e.g. char/wchar_t-dependent stuff.
The first solution was quite common in the early days of C++ templates, because at that time compilers were not good at recognizing commonalities in the generated machine code, and would introduce so called “code bloat”. Today, much to the surprise of any novice who tries it out, a templated solution can often have smaller machine code footprint than a solution relying on runtime polymorphism. Of course, YMMV.
Cheers & hth.,