Finding Duplicate Data in Oracle

Question

I have a table with 500,000+ records, and fields for ID, first name, last name, and email address. What I'm trying to do is find rows where the first name AND last name are both duplicates (as in the same person has two separate IDs, email addresses, or whatever, they're in the table more than once). I think I know how to find the duplicates using GROUP BY, this is what I have:

SELECT first_name, last_name, COUNT(*)
FROM person_table
GROUP BY first_name, last_name
HAVING COUNT(*) > 1

The problem is that I need to then move the entire row with these duplicated names into a different table. Is there a way to find the duplicates and get the whole row? Or at least to get the IDs as well? I tried using a self-join, but got back more rows than were in the table to begin with. Would that be a better approach? Any help would be greatly appreciated.

Answer 1

An index on (first_name, last_name) or on (last_name, first_name) would help:

SELECT t.*
FROM 
    person_table t
  JOIN      
      ( SELECT first_name, last_name
        FROM person_table
        GROUP BY first_name, last_name
        HAVING COUNT(*) > 1
      ) dup
    ON  dup.last_name = t.last_name
    AND dup.first_name = t.first_name

or:

SELECT t.*
FROM person_table t
WHERE EXISTS
      ( SELECT *
        FROM person_table dup
        WHERE dup.last_name = t.last_name
          AND dup.first_name = t.first_name
          AND dup.ID <> t.ID
      )

Answer 2

The most effective way to remove duplicate rows is with a self-join:

DELETE FROM person_table a
 WHERE a.rowid > 
       ANY (SELECT b.rowid
              FROM person_table b
             WHERE a.first_name = b.first_name
               AND a.last_name  = b.last_name);

This will remove all duplicates even if there are more than one duplicate row.

There is more on removing duplicates and differing methods here: http://www.dba-oracle.com/t_delete_duplicate_table_rows.htm

Hope it helps...

EDIT: As per your comments, if you want to select all but one of the duplicates then

SELECT *
  FROM person_table a
 WHERE a.rowid > 
       ANY (SELECT b.rowid
              FROM person_table b
             WHERE a.first_name = b.first_name
               AND a.last_name  = b.last_name);

Answer 3

This will give you an ID you want to move/delete/etc. Note that it does not work if count(*) > 2, as you get only 1 ID (you could re-run your query for these cases).

SELECT max(ID), first_name, last_name, COUNT(*)
FROM person_table
GROUP BY first_name, last_name
HAVING COUNT(*) > 1

Edit: You can use COLLECT to get all IDs at once (but be careful, as you only want to move/delete all but one)

Answer 4

To add another option, I usually use this one to remove duplicates:

delete from person_table
where rowid in (select rid
                  from (select rowid rid, row_number() over
                         (partition by first_name,last_name order by rowid) rn
                          from person_table
                        )
                 where rn <> 1 )