When I execute the following query:
SELECT * FROM `table1`
INNER JOIN table2 ON table2.number = table1.number
I get the result within 2 s
The reason this isn't working is cause your basiclly joining every row of table1 with every row with table 2. You need something to still join it with. The best way to do this is to do a left join (Meaning it'll join table1 no matter what, but not table2) and then check to make sure there isn't an entry for table2 with the is null. You'll then need to do the same thing for table2.
SELECT * FROM `table1`
LEFT JOIN table2 ON table2.number = table1.number
WHERE table2.number is NULL
UNION
SELECT * FROM `table2`
LEFT JOIN table1 ON table2.number = table1.number
WHERE table1.number is NULL