Algorithm - How to delete duplicate elements in a list efficiently?

Question

There is a list L. It contains elements of arbitrary type each. How to delete all duplicate elements in such list efficiently? ORDE

Answer 1

Delete duplicates in a list inplace in Python

Case: Items in the list are not hashable or comparable

That is we can't use set (dict) or sort.

from itertools import islice

def del_dups2(lst):
    """O(n**2) algorithm, O(1) in memory"""
    pos = 0
    for item in lst:
        if all(item != e for e in islice(lst, pos)):
            # we haven't seen `item` yet
            lst[pos] = item
            pos += 1
    del lst[pos:]

Case: Items are hashable

Solution is taken from here:

def del_dups(seq):
    """O(n) algorithm, O(log(n)) in memory (in theory)."""
    seen = {}
    pos = 0
    for item in seq:
        if item not in seen:
            seen[item] = True
            seq[pos] = item
            pos += 1
    del seq[pos:]

Case: Items are comparable, but not hashable

That is we can use sort. This solution doesn't preserve original order.

def del_dups3(lst):
    """O(n*log(n)) algorithm, O(1) memory"""
    lst.sort()
    it = iter(lst)
    for prev in it: # get the first element 
        break
    pos = 1 # start from the second element
    for item in it: 
        if item != prev: # we haven't seen `item` yet
            lst[pos] = prev = item
            pos += 1
    del lst[pos:]