Update existing field in List and re-save into same protobuf.net file

问题

Got POCO with 100+ fields (using DataMember(Order=X) attribute to mark them for ProtoBuf.Net serialisation).

We need to do some post processing of the file where one field's value needs to calced (which we can only do once we have read through it once) be updated and saved down again. Atm we are resaving into new file and this obviously has some IO/file overhead.

Is there any way I can update existing file where only Field "X"'s value needs to be updated and saved down. Hopefully this should be quicker than generating new file from scratch.

Sample POCO Data Structure (and each file will have ~3mln of those items)

[DataContract]
public class DataItem
{
    [DataMember(Order=1)]
    public string ProfitCentre {get; set;}

    [DataMember(Order=2)]
    public string BusinessFunction {get; set;}

    //this is the value that needs to be updated once file is written
    [DataMember(Order=3)]
    public double AdjustedAmount {get; set; } 
}

回答1:

That depends a bit on the structure of the model. The protobuf format allows for append-as-update, meaning: a value can be written at the end of the file that is then applied to the code model. There is, however, a limitation: it doesn't work with repeated elements (which includes map<...> elements). It can apply to the root object or any number of optional / required sub-objects, but not repeated. The other limitation is that you need an API that writes that value and only that value. There are various ways of achieving this, for example if we have:

Foo
 - Bar (field 3 of Foo)
   - Blap (field 7 of Bar)
     - X (string, field 4 of Blap)

then one of many ways of doing that would be to have a second model just for this purpose that updates the X field, as:

[ProtoContract]
class FooUpdate {
    [ProtoMember(3)]
    public BarUpdate Bar {get;set;} = new BarUpdate();
}
[ProtoContract]
class BarUpdate {
    [ProtoMember(7)]
    public BlapUpdate Blap {get;set;} = new BlapUpdate();
}
[ProtoContract]
class BlapUpdate {
    [ProtoMember(4)]
    public string X {get;set;}
}

now; assuming we previously wrote a regular Foo to a file, we can construct:

var obj = new FooUpdate { Bar = { Blap = { X = "abcdef" } } };
Serializer.Serialize(existingFile, obj);

which appends just this data (I'm assuming here that existingFile is positioned at the end, for append purposes).

Note that something similar can also be achieved using "conditional serialization", but it seems a lot of work to turn everything off when we can just... not have anything to turn off.

If the data you want to write is in a repeated / map<...> field, however: you'll just have to rewrite the entire file. In that context, append would result in additional list elements, or the loss of data from map elements.

来源：https://stackoverflow.com/questions/44883721/update-existing-field-in-listt-and-re-save-into-same-protobuf-net-file

标签

protobuf-net

Update existing field in List<T> and re-save into same protobuf.net file

问题

回答1: