How to remove these duplicates in a list (python)

Question

biglist = 

[ 

    {\'title\':\'U2 Band\',\'link\':\'u2.com\'}, 
    {\'title\':\'ABC Station\',\'link\':\'abc.com\'}, 
    {\'title\':\'Live Concert by U2\',\'link         


        

Answer 1

You can sort the list, using the link field of each dictionary as the sort key, then iterate through the list once and remove duplicates (or rather, create a new list with duplicates removed, as is the Python idiom), like so:

# sort the list using the 'link' item as the sort key
biglist.sort(key=lambda elt: elt['link'])

newbiglist = []
for item in biglist:
    if newbiglist == [] or item['link'] != newbiglist[-1]['link']:
        newbiglist.append(item)

This code will give you the first element (relative ordering in the original biglist) for any group of "duplicates". This is true because the .sort() algorithm used by Python is guaranteed to be a stable sort -- it does not change the order of elements determined to be equal to one another (in this case, elements with the same link).