Protocol Buffers with Extensions

Question

I'm perhaps overlooking something, but I'm attempting to wrestle protocol buffers into an easy method for providing extensions later. That seems a bit unclear so I'll jump directly into the problem.

I am writing an assembly to support various tasks, one of which includes describing structured data. Perfect time to use protocol buffers. The primary class to use protocol buffers is called StateDefinition. Here's the .proto file I came up with for it:

package Kannon.State;
message StateDefinition {
    enum StateTypes {
        GRAPHICS = 0;
        AUDIO = 1;
        MIND = 2;
        PHYSICS = 3;
        NETWORK = 4;
        GENERIC = 5;
    }
    repeated StateTypes requiredStates = 1; 
    optional GraphicsStateDef Graphics = 2;
    optional AudioStateDef Audio = 3;
         (etc)
}

message GraphicsStateDef {
    extensions 100 to max;
}

message AudioStateDef {
    extensions 100 to max;
}
    (etc)

My goal was to allow those _StateDef messages to be extended later with what the fields it would need. However, this extension would happen independent of the library I'm currently writing.

Kagents.dll -> Handles StateDefinition parsing and such.

Something Referencing Kagents.dll -> Has a protobuff file with "extend GraphicsStateDef" to define the state needed.

I was hoping that defining the "extend GraphicsStateDef" would generate code that would allow me to use properties to access these fields, and avoid the cumbersome "Extendible.AppendValue()" and GetValue() syntax.

One solution I devised, which seems hackish, is to define a class in the referencing DLL with extension methods, like so:

    public static class GraphicsExt
    {
        enum Fields
        {
            someValue = 1,
            someOtherValue = 2
        }

        public static Int32 someValue(this State.GraphicsStateDef def)
        {
            return Extensible.GetValue(def, Fields.someValue);
        }
        public static void someValue(this State.graphicsStateDef def, Int32 value)
        {
            Extensible.AppendValue(def, fields.someValue, value);
        }
    }

If anyone can think of a better way, I would be much obliged. =) Also, I'm not sure how lucid my description of the problem came out, so if there's any clarification or further information I can provide, please let me know. =)

EDIT: So, After thinking a lot about this and realized I'm approaching the problem wrong. StateReference is supposed to store a list of different GameState's. As well, it stores a StateDefinition, which should describe the state of this state reference. Currently, I'm trying to deserialize the state buffers into different classes (GraphicsStateDef), when I really should be deserializing into the state objects themselves.

Therefore, I need to rethink the design such that StateDefinition becomes a container for the stream and extracts only enough information for the "repeated StateTypes requiredStates=1" field. Then, in the referencing assembly, the rest of the stream can be deserialized into the respective states.

Does anyone have reccomendations for how to approach this? A few ideas are formulating, but nothing concrete, and I'd love the input of others.

Answer 1

I'm the author of protobuf-net. I haven't added anything to address the scenario as directly presented (other than the Extensible code), but I'm open to suggestion on what you think it should do.

I would also need to check whether "protoc" (the .proto compiler that I use to parse .proto prior to code generation) allows me to distinguish between regular and extended members.

Answer 2

Final answer:

Alright, so, A few days ago I settled on a solution and I'm just updating this in case anyone else runs into the same issue.

The whole problem stemmed from the fact that I didn't realize protobuf-net could support byte[]. So, here's my solution:

namespace Kannon.State
{
    /// <summary>
    /// ReferenceDefinition describes the layout of the reference in general.
    /// It tells what states it should have, and stores the stream buffers for later serialization.
    /// </summary>
    [ProtoBuf.ProtoContract]
    public class ReferenceDefinition
    {
        /// <summary>
        /// There are several built in state types, as well as rudimentary support for a "Generic" state.
        /// </summary>
        public enum StateType
        {
            Graphics=0,
            Audio,
            Mind,
            Physics,
            Network,
            Generic
        }

        /// <summary>
        /// Represents what states should be present in the ReferenceDefinition
        /// </summary>
        [ProtoBuf.ProtoMember(1)]
        List<StateType> m_StatesPresent = new List<StateType>();

        /// <summary>
        /// Represent a list of StateDefinitions, which hold the buffers for each different type of state.
        /// </summary>
        [ProtoBuf.ProtoMember(2)]
        List<StateDefinition> m_StateDefinition = new List<StateDefinition>();

        /// <summary>
        /// Add a state, mapped to a type, to this reference definition.
        /// </summary>
        /// <param name="type">Type of state to add</param>
        /// <param name="def">State definition to add.</param>
        public void AddState(StateType type, StateDefinition def)
        {
            // Enforce only 1 of each type, except for Generic, which can have as many as it wants.
            if (m_StatesPresent.Contains(type) && type != StateType.Generic)
                return;
            m_StatesPresent.Add(type);
            m_StateDefinition.Add(def);
        }
    }

    /// <summary>
    /// Represents a definition of some gamestate, storing protobuffered data to be remapped to the state.
    /// </summary>
    [ProtoBuf.ProtoContract]
    public class StateDefinition
    {
        /// <summary>
        /// Name of the state
        /// </summary>
        [ProtoBuf.ProtoMember(1)]
        string m_StateName;
        /// <summary>
        /// Byte array to store the "data" for later serialization.
        /// </summary>
        [ProtoBuf.ProtoMember(2)]
        byte[] m_Buffer;

        /// <summary>
        /// Constructor for the state definition, protected to enforce the Pack and Unpack functionality to keep things safe.
        /// </summary>
        /// <param name="name">Name of the state type.</param>
        /// <param name="buff">byte buffer to build state off of</param>
        protected StateDefinition(String name, byte[] buff)
        {
            m_StateName = name;
            m_Buffer = buff;
        }

        /// <summary>
        /// Unpack a StateDefinition into a GameState
        /// </summary>
        /// <typeparam name="T">Gamestate type to unpack into.  Must define Protobuf Contracts.</typeparam>
        /// <param name="def">State Definition to unpack.</param>
        /// <returns>The unpacked state data.</returns>
        public static T Unpack<T>(StateDefinition def) where T:GameState
        {
            // Make sure we're unpacking into the right state type.
            if (typeof(T).Name == def.m_StateName)
                return ProtoBuf.Serializer.Deserialize<T>(new MemoryStream(def.m_Buffer));
            else
                // Otherwise, return the equivalent of Null.
                return default(T);
        }

        /// <summary>
        /// Pack a state type into a State Definition
        /// </summary>
        /// <typeparam name="T">Gamestate to package up.  Upst define protobuf contracts.</typeparam>
        /// <param name="state">State to pack up.</param>
        /// <returns>A state definition serialized from the passed in state.</returns>
        public static StateDefinition Pack<T>(T state) where T:GameState
        {
            // Using a memory stream, to make sure Garbage Collection knows what's going on.
            using (MemoryStream s = new MemoryStream())
            {
                ProtoBuf.Serializer.Serialize<T>(s, state);
                // Uses typeof(T).Name to do semi-enforcement of type safety.  Not the best, but it works.
                return new StateDefinition(typeof(T).Name, s.ToArray());
            }
        }
    }
}