MySQL huge tables JOIN makes database collapse

Question

Following my recent question Select information from last item and join to the total amount, I am having some memory problems while generation tables

I have two tabl

Answer 1

You can make this puppy scream. Dump the whole inner join query. Really. This is a trick virtually no one seems to know about.

Assuming dates is a datetime, convert it to a sortable string, concatenate the values you want, max (or min), substring, cast. You may need to adjust the date convert function (this one works in MS-SQL), but this idea will work anywhere:

SELECT customer, count(sale), max_sale = cast(substring(max(convert(char(19), dates, 120) + str(sale, 12, 2)), 20, 12) as numeric(12, 2))
FROM sales a 
group by customer

Voilá. If you need more result columns, do:

SELECT yourkey
            , maxval = left(val, N1)                  --you often won't need this
            , result1 = substring(val, N1+1, N2)
            , result2 = substring(val, N1+N2+1, N3)   --etc. for more values
FROM ( SELECT yourkey, val = max(cast(maxval as char(N1))
                               + cast(resultCol1 as char(N2))
                               + cast(resultCol2 as char(N3)) )
       FROM yourtable GROUP BY yourkey ) t

Be sure that you have fixed lengths for all but the last field. This takes a little work to get your head around, but is very learnable and repeatable. It will work on any database engine, and even if you have rank functions, this will often significantly outperform them.

More on this very common challenge here.