IMO, defaultdict is unnecessary here and doing it as a list comprehension sacrifices readability (though that's not generally the case). Unless profiling indicates that this is really a bottleneck, I would do it as follows:
def invert_to_lists(dct):
inverted_dict = {}
for key in dct:
inverted_dict.setdefault(dct[key], []).append(key)
return inverted_dict
defaultdict is one more complication. Using setdefault is fine in this case because it only needs to be typed out once. After going through the rigmarole of importing and instantiating a defaultdict, you will have typed more than making the one call to setdefault.