MYSQL query performs very slow

Question

I have developed a user bulk upload module. There are 2 situations, when I do a bulk upload of 20 000 records when database has zero records. Its taking about 5 hours. But w

Answer 1

If I understand, for all the result of SELECT * FROM AdvanceBulkInsert ... you run a request SELECT cf.*, and for all the SELECT cf.*, you run the SELECT * FROM User

I think the issue is that you send way too much requests to the base.

I think you should merge all your select request in only one big request.

For that:

• replace the SELECT * FROM AdvanceBulkInsert by a EXISTS IN (SELECT * FROM AdvanceBulkInsert where ...) or a JOIN

• replace the SELECT * FROM User by a NOT EXISTS IN(SELECT * from User WHERE )

Then you call the update on all the result of the merged select.

You should too time one by one your request to find which of this requests take the most time, and you should too use ANALYSE to find what part of the request take time.

Edit:

Now I have see your code :

Some lead:

• have you index for cf.customTypeId , cfv.customFieldId , cfsa.customFieldId, user. dateOfBirth ,user. firstName,user.lastName ?

• you don't need to do a LEFT JOIN CustomFieldSubArea if you have a WHERE who use CustomFieldSubArea, a simple JOIN CustomFieldSubArea is enougth.

• You will launch the query 2 a lot of time with relatedId = 0 , maybe you can save the result in a var?

• if you don't need sorted data, remove the "ORDER BY cf.sortOrder, cf.label" . Else, add index on cf.sortOrder, cf.label