How to remove these duplicates in a list (python)

Question

biglist = 

[ 

    {\'title\':\'U2 Band\',\'link\':\'u2.com\'}, 
    {\'title\':\'ABC Station\',\'link\':\'abc.com\'}, 
    {\'title\':\'Live Concert by U2\',\'link         


        

Answer 1

Probably the fastest approach, for a really big list, if you want to preserve the exact order of the items that remain, is the following...:

biglist = [ 
    {'title':'U2 Band','link':'u2.com'}, 
    {'title':'ABC Station','link':'abc.com'}, 
    {'title':'Live Concert by U2','link':'u2.com'} 
]

known_links = set()
newlist = []

for d in biglist:
  link = d['link']
  if link in known_links: continue
  newlist.append(d)
  known_links.add(link)

biglist[:] = newlist