I\'m wondering why I can\'t just cast (I have a vague idea this might have something to do with that co/contravariance stuff?), and am I forced to copy the elements of the f
You cannot do this because they aren't the same type. Consider:
        var x = new Dictionary>();
        // won't compile, but assume it could...
        Dictionary> xPrime = x;
        // uh oh, would allow us to legally add array of int!
        xPrime["Hi"] = new int[13];
  
Does this make sense?  Because Dictionary says the TValue is List which means you can Add() a List as a value.  If you could cast this to a Dictionary it would mean the value type is IEnumerable which would mean you could Add() any IEnumerable (int[], HashSet, etc) which would violate the original type.
So, a List can be converted to a IEnumerable reference because List implements IEnumerable, but that does not mean that Dictionary implements/extends Dictionary.
Put more simply:
Dictionary 
 
Can't convert to
Dictionary
 
Because the latter would allow you to add Cat, Squirrel, etc.