Changing the input data layer during training in Caffe

Question

Is it possible to change the input source of ImageData layer or a MemoryData layer on the fly?

I am trying to shuffle the data every epoch but I have both image and some other non-image features that I want to concatenate at a later stage in the network. I could not find a reliable way to shuffle both the image and my other data in a way that preserves the alignment of the two.

So, I am thinking of re-generating the imagelist.txt as well as nonimage data (in Memory) every epoch and attach the new file to the ImageData layer and initialize MemoryDataLayer with the new data.

How can I make sure that I re-initialize the network with the new text file without restarting the training process. (I want the network to continue training at the same stage, momentum etc.., only start reading the image files from new file instead of one that was originally compiled).

layer {
name: "imgdata"
type: "ImageData"
top: "imgdata"
top: "dlabel"
transform_param {
  # Transform param here
}
image_data_param {
source: "path to imagelist.txt" ## This file changes after n iterartions
batch_size: XX
new_height: XXX
new_width: XXX
}
}

And Similarly, I want to be able to copy the re-shuffled data into MemoryData layer. Can I call Net.set_input_arrays during middle of training?

layers {
  name: "data"
  type: MEMORY_DATA
  top: "data"
  top: "label"
  memory_data_param {
  batch_size: XX
  channels: X
  height: XXX
  width: XXX
  }

Answer 1

Your problem could be solved with the help of Python layers, as suggested in comments. An example of the usage of python layer can be found within caffe here.

Within the python script you could write the code to shuffle both the data by preserving their alignments.

Answer 2

I want to provide an answer without python interface, I did it in C++ source code.

Intel/caffe is used as example because currently I am working on it, but you can easily try with blvc/caffe. I assume you can compile caffe source code.

Here is one proper place to change in the source code(right after loading the train_text.prototxt):

    https://github.com/intel/caffe/blob/master/src/caffe/solver.cpp#L156

inside the block, add following:

    net_param.mutable_layer(0)->mutable_data_param()->set_source("/test/data/path/");
    net_param.mutable_layer(1)->mutable_data_param()->set_source("/test/data/path/");
    LOG(INFO) << "+++> " << net_param.layer(0).data_param().source();
    LOG(INFO) << "+++> " << net_param.layer(1).data_param().source();

You will change the data source of first two layers(train data layer and test data layer).

So if you want to change the data source in other scenarios, just find the context of net_param, layer and data_param, then you can use above function calls to change your own data path.

Actually as caffe uses google protobuffer, so if you want to understand the full interfaces of net_param, layer and data_param, you need to build caffe, then check file: .build_release/src/caffe/proto/caffe.pb.h(I used Makefile to build, not cmake). I don't really understand protobuffer, if you know it, you can check source code: src/caffe/proto/caffe.proto.