Algorithm - How to delete duplicate elements in a list efficiently?

Question

There is a list L. It contains elements of arbitrary type each. How to delete all duplicate elements in such list efficiently? ORDE

Answer 1

If the order does not matter, you might want to try this algorithm written in Python:

>>> array = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6]
>>> unique = set(array)
>>> list(unique)
[1, 2, 3, 4, 5, 6]

Answer 2

My code in Java:

ArrayList<Integer> list = new ArrayList<Integer>();

list.addAll({1,2,1,3,4,5,2,3,4,3});

for (int i=0; i<list.size(); i++)
{
    for (int j=i+1; j<list.size(); j++)
    {
        if (list.get(i) == list.get(j))
        {
            list.remove(i);
            j--;
        }
    }
}

or simply do this:

SetList<Integer> unique = new SetList<Integer>();

unique.addAll(list);

Both ways have Time = nk ~ O(n^2)

where n is the size of input list,

k is number of unique members of the input list

Answer 3

Maybe you should look into using associate arrays (aka dict in python) to avoid having duplicate elements in the first place.

Answer 4

Algorithm delete_duplicates (a[1....n])

//Remove duplicates from the given array

//input parameters :a[1:n], an array of n elements

{

temp[1:n]; //an array of n elements

 temp[i]=a[i];for i=1 to n

     temp[i].value=a[i]

        temp[i].key=i

*//based on 'value' sort the array temp.*

//based on 'value' delete duplicate elements from temp.

//based on 'key' sort the array temp.//construct an array p using temp.

p[i]=temp[i].value

return p

In other of elements is maintained in the output array using the 'key'. Consider the key is of length O(n), the time taken for performing sorting on the key and value is O(nlogn). So the time taken to delete all duplicates from the array is O(nlogn).

Answer 5

It depends on what you mean by "efficently". The naive algorithm is O(n^2), and I assume what you actually mean is that you want something of lower order than that.

As Maxim100 says, you can preserve the order by pairing the list with a series of numbers, use any algorithm you like, and then resort the remainder back into their original order. In Haskell it would look like this:

superNub :: (Ord a) => [a] -> [a]
superNub xs = map snd 
              . sortBy (comparing fst) 
              . map head . groupBy ((==) `on` snd) 
              . sortBy (comparing snd) 
              . zip [1..] $ xs

Of course you need to import Data.List (sort), Data.Function (on) and Data.Ord (comparing). I could just recite the definitions of those functions, but what would be the point?

Answer 6

One line solution in Python.
Using lists-comprehesion:

>>> L = [2, 1, 4, 3, 5, 1, 2, 1, 1, 6, 5]
>>> M = []
>>> zip(*[(e,M.append(e)) for e in L if not e in M])[0]
(2, 1, 4, 3, 5, 6)