python: remove duplicate values from dict

Question

I'm trying to remove duplicate values in my dict but its not working:

samples_antibiotics_with_duplicates = {'S00541-09': ['Streptomycin', 'Sulfamethoxazole', 'Trimethoprim', 'Spectinomycin', 'Streptomycin', 'Streptomycin', 'Trimethoprim']}
samples_antibiotics = {}
    for key,value in samples_antibiotics_with_duplicates.iteritems():
      if value not in samples_antibiotics.values():
        samples_antibiotics[key] = value
print samples_antibiotics

This prints:

{'S00541-09': ['Streptomycin', 'Sulfamethoxazole', 'Trimethoprim', 'Spectinomycin', 'Streptomycin', 'Streptomycin', 'Trimethoprim']}

Any help will be appreciated.

Answer 1

You can try this:

samples_antibiotics_with_duplicates = {'S00541-09': ['Streptomycin', 'Sulfamethoxazole', 'Trimethoprim', 'Spectinomycin', 'Streptomycin', 'Streptomycin', 'Trimethoprim']}

new_dict = {a:list(set(b)) for a, b in samples_antibiotics_with_duplicatates.items()}

Answer 2

If you don't care about retaining original order then set(my_list) will remove all duplicates.

If you want to retain original order then list(OrderedDict.fromkeys(my_list))

Answer 3

The below dict comprehension will create a new dict from the original one without any duplicate values:

samples_antibiotics = {k: list(set(v)) for k, v in samples_antibiotics_with_duplicates.items()}

The set version of a list (or any container) does not contain any duplicates since sets do not allow any (that is why they require hashable items as do dicts).

As @CoryKramer says in the comments, the solution given here will not (generally speaking) preserve the order of the items in the values-list. If that is important to you, you would have to go with something else.

Answer 4

There are better ways to do this as seen in the other posts. But to retain as much of your original code as possible while explaining why it doesn't work use this instead:

samples_antibiotics_with_duplicates = {'S00541-09': ['Streptomycin', 'Sulfamethoxazole', 'Trimethoprim', 'Spectinomycin', 'Streptomycin', 'Streptomycin', 'Trimethoprim']}
samples_antibiotics = {}
for key,value in samples_antibiotics_with_duplicates.items():
    samples_antibiotics[key] = set(value)
print(samples_antibiotics)

The problem is that you iterate through each key in the dictionary in your for loop (so only the 'S00541-09') and then you check if the value is in the values (which obviously it has to be). What I did was essentially iterate the values within the key itself.