ZeroMQ + Protocol Buffers

Question

ZeroMQ FAQ page suggest use of Google's protobuf as a way to serialise message content.

Has anyone see a good usage example?

I also need to get the answer to "What is the biggest advantage of serialising messages?" - whether it may be something I can live without and take the advantage of slimmer pipeline.

I quite like the idea of .proto files and the protoc compiler.

Also, it seem that another great tool to throw at the playground would be libev, any comments are welcome :)

Answer 1

If you are 100% certain that the programs that are going to communicate over ZMQ will at all times be capable of understanding each other's binary format (eg because they are always distributed together and were all compiled with the same compiler options anyways) I see no benefit to the overhead that's added by serialization.

As soon as the above condition cannot be satisfied (like partner programs running on different host types, programs written in different languages or even partner programs that can evolve independently in time - which may cause incompatibilities in their raw binary structures) serialization becomes quite probably a must.

It seems that nowadays everybody and their brother is creating serialization solutions, which may be an indication that there's no one size fits all solution. This page contains a pretty thorough benchmarking of serialization time, deserialization time and sizes for 27 (!!) different serialization systems. Don't skip the first paragraph of that page, it says "Warning, benchmarks can be misleading". Your application, your data are what counts for you, but the data presented there may help you narrow down the choices you want to study in detail.

Answer 2

Here is a sample which send and receive messages through java and in C++:

Serializing in java:

Person person = Person.newBuilder().setName("chand")
    .setEmail("chand@test.com").setId(55555).build();
socket.send(person.toByteArray(), 0);

De-serialize in java:

byte[] reply = socket.recv(0);
Person person2 = Person.parseFrom(reply);

Serializing in C++:

Person p = Person();
std::string str;
p.SerializeToString(&str);
int sz = str.length();
zmq::message_t *query = new message_t(sz);
memcpy(query->data (), str.c_str(), sz);
socket->send (*query);

De-serializign in C++

zmq::message_t resultset(100);
socket->recv (&resultset);

Person p = Person();
p.ParseFromArray(resultset.data(), resultset.size());
printf("\n Server : %s", p.name().c_str());

Answer 3

I am not sure PUB/SUB in 0mq will work with protobuf, because 0mq expects a string topic at head of the msg.. but protobuf puts a field descriptor first.

actually here is a link with a solution.

http://www.dotkam.com/2011/09/09/zeromq-and-google-protocol-buffers/

cheers

Answer 4

You always need to serialize when communicating. Structures are random access. Communication layers, like ZeroMQ are serial.

You can use the "default serialization" that comes with your language.

For example in C++ a structure with no pointers will have a certain binary layout that can be directly turned into a byte array. This binary layout is, indirectly, your serialization layer, and is both language and compiler specific.

As long as you limit yourself to structures that have no pointers and are using the same compiler and language on both ends of the pipe... feel free to avoid a library that does additional serialization on top of the default layout provided.