Personally, I quite like header-only libraries, but there are claims they cause code bloat due to over-inlining (as well as the other obvious problem of longer compile times
In my experience bloat hasn't been a problem:
Header only libraries give compilers greater ability to inline, but they do not force compilers to inline - many compilers treat the inline keyword as nothing more than a command to ignore multiple identical definitions.
Compilers usually have options to optimize to control the amount of inlining; /Os on Microsoft's compilers.
It's usually better to allow the compiler to manage the speed vs. size issues. You'll only see bloat from calls that have actually been inlined, and the compiler will only inline them if its heuristics indicate that it inlining will improve performance.
I wouldn't consider code bloat as a reason to stay away from header only libraries - but I would urge you to consider how much a header only approach will increase compile times by.