Several shared object using same proto leading the the error: file already exists in database

Question

An error related to protobuf3

I have a project that have an c++ executable core, and several shared objects (.so, .dll) called plugins. When the core launches, it will load those plugins with dlopen. The core and plugins using protobuf as communication protocol, so they have to compile the generated .pb.cc and .ph.h files into their binaries to have the copy of the serializer/deserializer. And libprotobuf.so link to both the core and plugins. When I launch the core, it crushes with error: file already exists in database, same error in #863

I'm using protobuf-3 beta2, and Ubuntu 14.04. This error only happens on Linux. The program works fine on Windows and OS X.

I have also tried another way which compile all the generated protobuf files into a dynamic library (protocol.so), then the core and plugins were linked to protocol.so and libprotobuf.so. This works fine. Of course, because in #1062 the bug has been fixed. But when I changed the protocol.so into protocol.a, it failed again. I think it is same as compile generated .pb.cc separately.

I don't want to compile a protocol.so, because it is inconvenient for me to extend the communication protocol when I add more and more plugins. I think compile the generated .pb.cc into the plugin's binary is better (this work well on windows and OS X).

Any suggestions to fix this error are appreciated.

Answer 1

The problem happens when you have multiple compiled copies of the same .pb.cc file sharing a single copy of libprotobuf.so. There are two ways to avoid this:

1. The way you already found: factor out the .pb.cc files into a shared library.
2. Link a separate copy of libprotobuf into each plugin. You'll need to use static linking for this library, i.e. use libprotobuf.a rather than libprotobuf.so. Note that with this option, it is unsafe to pass a pointer to a protobuf class between the plugins and the base application, because they are using separate copies of the protobuf library, which can lead to crashes. You will have to pass serialized messages as byte blobs instead. Luckily, that's the whole point of protobuf.

Answer 2

I was able to get around this problem by adding RTLD_GLOBAL to dlopen which takes existing known symbols into account.