Too many tables; MySQL can only use 61 tables in a join

Question

What is the best way to export data from multiple tables in MySQL. I\'m basically working with product details. Say a product has 150 attributes of data. How can I export t

Answer 1

If you're using EAV and you want to export a large number of attributes at once, the best way is actually to use multiple temporary tables.

Each temporary table will have the same primary key column. Then join all of them and export into csv.

I don't know if I want to do a fully fleshed out example, but I will try to do an outline that will hopefully make things clearer.

1.) Get your list of attributes you want to export. You will use their attribute_ids in the join to your EAV attribute_values table.

2.) Split up the attributes so that you will not exceed the join limit. You need the original table, and 1 table per join, so you can have 60 attributes per table in this scheme.

3.) Create "flat" temporary tables for each group of attributes. It would go something like this.

CREATE TEMPORARY TABLE temp1
[(create_definition,...)]
SELECT t1.product_id, t1.sku, t2.color, GROUP_CONCAT(t3.sizes SEPARATOR ',') as sizes,
...

#( suppose the product has multiple sizes and you want them shown comma-separated in your export)

FROM products t1
LEFT JOIN eav_attribute_values t2 ON t1.product_id = t2.product_id AND t2.attribute_id = 55
LEFT JOIN eav_attribute_values t3 ON t1.product_id = t2.product_id AND t2.attribute_id = 76
... etc for up to 60 attributes

CREATE TEMPORARY TABLE temp2 ... # repeat for next 60 attributes

4.) Now you have temporary tables temp1, temp2, temp3, etc. They all share the same primary key (product_id and/or product_sku for example). Assuming you have less than 60 temporary tables (which would be absurd), you can now join all of those and create a single table.

In my system, I don't think I've exceeded 3 temporary tables and that is quite a lot.

CREATE TEMPORARY TABLE export_data
[(create_definition,...)]
SELECT t1.*, t2.*, t3.* FROM # though I would not actually use * here b/c it would cause repeated key fields. I would list out all the columns
temp1 t1 LEFT JOIN temp2 t2 ON t1.product_id = t2.product_id
LEFT JOIN temp3 t3 ON t1.product_id = t3.product_id # etc for more joins

5.) Export. Use MySQL's file export feature to create a CSV. Send it to the user with PHP.

I hope that helps.

Also note that the above process executes fairly quickly for me. The reason to use temporary tables is because they will be automatically dropped after use, and because multiple users can all run the same type of process without interfering with each other since temporary tables only exist for the user who created them.