tl:dr; use -fvisibility=hidden as a gcc and llvm compiler switch in everything you compile, including your dependent libraries, unless you have a reason not to.
A good introduction to the -fvisibility and -fvisibility-inline-hidden compilation flags is available on Apple's web site, as of this writing. The article also goes into some detail on the __attribute__((visibility("hidden"))) and __attribute__((visibility("default"))) declarations.