raw decoder for protobufs format

Question

I\'d like to find a way to convert a binary protobuf message into a human readable description of the contained data, without using the .proto files.

The background

Answer 1

As noted in Michel de Ruiter's answer, it's possible that your protobuf message has a length-prefix. Assuming it does, this answer should help.

(NOTE: For most of the commands below, I'm assuming your protobuf message is stored in a file called input.)

protoc --decode_raw + dd for a single message:

If it's simply a single message, then you can indeed leverage protoc --decode_raw, but you need to strip off the length-prefix header first. Assuming the header is 4 bytes long you can use dd to strip the header off of input and then feed the output into protoc.

dd bs=1 skip=4 if=input 2>/dev/null | protoc --decode_raw

protoc-decode-lenprefix --decode_raw for a single message:

I also wrote a script that handles the header stripping automatically:

protoc-decode-lenprefix --decode_raw < input

This script is simply a wrapper on top of protoc --decode_raw, but handles parsing out the length-prefix header and then invoking protoc.

Now, this script isn't terribly useful in this case, because we can just use the dd trick above to strip the header off. However, say we have a data stream (e.g., a file or TCP stream) containing multiple messages that are framed with length-prefix headers....

protoc-decode-lenprefix --decode_raw for a stream of messages:

Instead of a single protobuf message in the input file, lets say input contained multiple protobuf messages which are framed by length-prefix headers. In this case its not possible to just use the dd trick, because you need to actually read the contents of the length-prefix header to determine how long the subsequent message in the stream is, and thus how many bytes ahead the next header+message lies. So instead of worrying about all of that, you can simply use protoc-decode-lenprefix again:

protoc-decode-lenprefix --decode_raw < input

protoc-decode-lenprefix --decode ... foo.proto for a stream of messages

This script can also be used to fully decode length-prefixed messages (instead of just "raw decode" them). It assumes you have access to the .proto files that define the protobuf message, just like the wrapped protoc command. The invocation syntax is identical to protoc --decode. For example, using the dd trick with protoc --decode, along with input being a Mesos task.info file, the syntax looks like this:

dd bs=1 skip=4 if=task.info 2>/dev/null | \
protoc --decode mesos.internal.Task \
                      -I MESOS_CODE/src -I MESOS_CODE/include \
                      MESOS_CODE/src/messages/messages.proto

And the parameters are identical when using protoc-decode-lenprefix

cat task.info | \
protoc-decode-lenprefix --decode mesos.internal.Task \
                      -I MESOS_CODE/src -I MESOS_CODE/include \
                      MESOS_CODE/src/messages/messages.proto