The way I did it now is with a postprocessing step and a recursive function that's doing its own reference saving and rewiring:
private static void Reffing(this IDictionary current, Action
used as:
var test = JsonConvert.DeserializeObject(...);
var dictionary = new Dictionary();
Reffing(test,null,dictionary);