duplicate-removal

I have a data.table table with about 2.5 million rows. There are two columns. I want to remove any rows that are duplicated in both columns. Previously for a data.frame I would have done this: df -> unique(df[,c('V1', 'V2')]) but this doesn't work with data.table. I have tried unique(df[,c(V1,V2), with=FALSE]) but it seems to still only operate on the key of the data.table and not the whole row. Any suggestions? Cheers, Davy Example >dt V1 V2 [1,] A B [2,] A C [3,] A D [4,] A B [5,] B A [6,] C D [7,] C D [8,] E F [9,] G G [10,] A B in the above data.table where V2 is the table key, only rows 4

I am using postgres. I want to delete Duplicate rows. The condition is that , 1 copy from the set of duplicate rows would not be deleted. i.e : if there are 5 duplicate records then 4 of them will be deleted. Try the steps described in this article: Removing duplicates from a PostgreSQL database . It describes a situation when you have to deal with huge amount of data which isn't possible to group by . A simple solution would be this: DELETE FROM foo WHERE id NOT IN (SELECT min(id) --or max(id) FROM foo GROUP BY hash) Where hash is something that gets duplicated. delete from table where not id

I have to add a unique constraint to an existing table. This is fine except that the table has millions of rows already, and many of the rows violate the unique constraint I need to add. What is the fastest approach to removing the offending rows? I have an SQL statement which finds the duplicates and deletes them, but it is taking forever to run. Is there another way to solve this problem? Maybe backing up the table, then restoring after the constraint is added? just somebody For example you could: CREATE TABLE tmp ... INSERT INTO tmp SELECT DISTINCT * FROM t; DROP TABLE t; ALTER TABLE tmp