Protobuf streaming (lazy serialization) API

Question

We have an Android app that uses Protocol Buffers to store application data. The data format (roughly) is a single protobuf (\"container\") that contains a list of protobufs (\"

Answer 1

For serialization:

protobuf is an appendable format, with individual items being merged, and repeated items being appended

Therefore, to write a sequence as a lazy stream, all you need to do is repeatedly write the same structure with only one item in the list: serializing a sequence of 200 x "Container with 1 Item" is 100% identical to serializing 1 x "Container with 200 Items".

So: just do that!

---

For deserialization:

That is technically very easy to read as a stream - it all, however, comes down to which library you are using. For example, I expose this in protobuf-net (a .NET / C# implementation) as Serializer.DeserializeItems, which reads (fully lazy/streaming) a sequence of messages of type T, based on the assumption that they are in the form you describe in the question (so Serializer.DeserializeItems would be the streaming way that replaces Serializer.Deserialize - the outermost object kinda doesn't really exist in protobuf)

If this isn't available, but you have access to a raw reader API, what you need to do is:

• read one varint for the header - this will be the value 10 (0x0A), i.e. "(1 << 3) | 2" for the field-number (1) and wire-type (2) respectively - so this could also be phrased: "read a single byte from the stream , and check the value is 10"
• read one varint for the length of the following item
• now:
  • if the reader API allows you to restrict the maximum number of bytes to process, use this length to specify the length that follows
  • or wrap the stream API with a length-limiting stream, limited to that length
  • or just manually read that many bytes, and construct an in-memory stream from the payload
• rinse, repeat