Removing duplicate elements from a List

Question

I have developed an array list.

ArrayList<String> list = new ArrayList<String>();

list.add("1");
list.add("2");
list.add("3");
list.add("3");
list.add("5");
list.add("6");
list.add("7");
list.add("7");
list.add("1");
list.add("10");
list.add("2");
list.add("12");

But as seen above it contains many duplicate elements. I want to remove all duplicates from that list. For this I think first I need to convert the list into a set.

Does Java provide the functionality of converting a list into a set? Are there other facilities to remove duplicates from a list?

Answer 1

You can convert to a Set with:

Set<String> aSet = new HashSet<String>(list);

Or you can convert to a set and back to a list with:

list = new ArrayList<String>(new HashSet<String>(list));

Both of these, however, are not likely to preserve the order of the elements. To preserve order, you can use a HashSet as an auxiliary structure while iterating:

List<String> list2 = new ArrayList<String>();
HashSet<String> lookup = new HashSet<String>();
for (String item : list) {
    if (lookup.add(item)) {
        // Set.add returns false if item is already in the set
        list2.add(item);
    }
}
list = list2;

In the case of duplicates, only the first occurrence will appear in the result. If you want only the last occurrence to appear, that's a tougher problem. I'd tackle it by reversing the input list, applying the above, and then reversing the result.

Answer 2

This:

Set<String> set = new HashSet<String>();
set.addAll(list);
list.clear();
list.addAll(set);

Answer 3

Java 8 way: list.stream().distinct().collect(Collectors.toList());

done :)

Answer 4

If you need to preserve elements order then use LinkedHashSet instead of HashSet

Set<String> mySet = new LinkedHashSet<String>(list);

Answer 5

Just use the normal constructor:

ArrayList<T> yourList;
HashSet<T> set = new HashSet<T>(yourList);

And you'll have a new view of the items, with duplicates removed, but you will lose ordering. This is true in every answer posted so far. To keep ordering you should iterate on the existing list and remove an element only if it's a duplicate (which can be done using a set to check if an element was already found).

Answer 6

You can use a set in the first place or convert into it:

 Set<String> set = new TreeSet<String>(list);

Answer 7

package com.scjp.dump.test;

import java.util.ArrayList;

import java.util.Iterator;

import java.util.List;

public class ArrayListTest {

    public static void main(String[] args) {

        List<Integer> mylist2 = new ArrayList<Integer>();

        List<Integer> mylist1 = new ArrayList<Integer>();
        mylist1.add(3);
        mylist1.add(3);
        mylist1.add(5);
        mylist1.add(9);
        mylist1.add(2);
        mylist1.add(5);
        mylist1.add(5);
        mylist1.add(3);
        mylist1.add(3);
        mylist1.add(3);
        mylist1.add(9);
        mylist1.add(56);
        System.out.println(mylist1);
        Iterator<Integer> itr1 = mylist1.listIterator();
        while (itr1.hasNext()) {
            Integer itn1 = (Integer) itr1.next();
            if (mylist2.contains(itn1) == false)
                mylist2.add(itn1);
        }

        System.out.println(mylist2);

    }

}

Answer 8

Here are some way you can achieve this.

Using Java 8:

List<String> distinctLambda=originalList.stream()
           .distinct().collect(Collectors.toList());
 System.out.println(distinctLambda);

Using Set:

Set<String> distinctSet=new HashSet<>(originalList);
        System.out.println(distinctSet);

Normal for loop:

List<String> distinctNewList=new ArrayList<>();
        for (String temp:originalList) {
            if(distinctNewList.size()==0){
                distinctNewList.add(temp);
                continue;
            }

            if(!distinctNewList.contains(temp)){
                distinctNewList.add(temp);
            }
        }

        System.out.println(distinctNewList);

Here is your data set:

ArrayList<String> originalList = new ArrayList<>();
        originalList.add("1");
        originalList.add("2");
        originalList.add("3");
        originalList.add("3");
        originalList.add("5");
        originalList.add("6");
        originalList.add("7");
        originalList.add("7");
        originalList.add("1");
        originalList.add("10");
        originalList.add("2");
        originalList.add("12");