Fix DB duplicate entries (MySQL bug)

Question

I'm using MySQL 4.1. Some tables have duplicates entries that go against the constraints.

When I try to group rows, MySQL doesn't recognise the rows as being similar.

Example:

Table A has a column "Name" with the Unique proprety.
The table contains one row with the name 'Hach?' and one row with the same name but a square at the end instead of the '?' (which I can't reproduce in this textfield)
A "Group by" on these 2 rows return 2 separate rows

This cause several problems including the fact that I can't export and reimport the database. On reimporting an error mentions that a Insert has failed because it violates a constraint.

In theory I could try to import, wait for the first error, fix the import script and the original DB, and repeat. In pratice, that would take forever.

Is there a way to list all the anomalies or force the database to recheck constraints (and list all the values/rows that go against them) ?

I can supply the .MYD file if it can be helpful.

Answer 1

To list all the anomalies:

SELECT name, count(*) FROM TableA GROUP BY name HAVING count(*) > 1;

There are a few ways to tackle deleting the dups and your path will depend heavily on the number of dups you have.

See this SO question for ways of removing those from your table.

Here is the solution I provided there:

-- Setup for example
create table people (fname varchar(10), lname varchar(10));

insert into people values ('Bob', 'Newhart');
insert into people values ('Bob', 'Newhart');
insert into people values ('Bill', 'Cosby');
insert into people values ('Jim', 'Gaffigan');
insert into people values ('Jim', 'Gaffigan');
insert into people values ('Adam', 'Sandler');

-- Show table with duplicates
select * from people;

-- Create table with one version of each duplicate record
create table dups as 
    select distinct fname, lname, count(*) 
    from people group by fname, lname 
    having count(*) > 1;

-- Delete all matching duplicate records
delete people from people inner join dups 
on people.fname = dups.fname AND 
   people.lname = dups.lname;

-- Insert single record of each dup back into table
insert into people select fname, lname from dups;

-- Show Fixed table
select * from people;

Answer 2

Create a new table, select all rows and group by the unique key (in the example column name) and insert in the new table.

Answer 3

To find out what is that character, do the following query:

SELECT HEX(Name) FROM TableName WHERE Name LIKE 'Hach%'

You will se the ascii code of that 'square'.

If that character is 'x', you could update like this:(but if that column is Unique you will have some errors)

UPDATE TableName SET Name=TRIM(TRAILING 'x' FROM Name);

Answer 4

I'll assume this is a MySQL 4.1 random bug. Somes values are just changing on their own for no particular reason even if they violates some MySQL constraints. MySQL is simply ignoring those violations.

To solve my problem, I will write a prog that tries to resinsert every line of data in the same table (to be precise : another table with the same caracteristics) and log every instance of failures.

I will leave the incident open for a while in case someone gets the same problem and someone else finds a more practical solution.