What generic techniques can be applied to optimize SQL queries?

Question

What techniques can be applied effectively to improve the performance of SQL queries? Are there any general rules that apply?

Answer 1

Some other points (Mine are based on SQL server, since each db backend has it's own implementations they may or may not hold true for all databases):

Avoid correlated subqueries in the select part of a statement, they are essentially cursors.

Design your tables to use the correct datatypes to avoid having to apply functions on them to get the data out. It is far harder to do date math when you store your data as varchar for instance.

If you find that you are frequently doing joins that have functions in them, then you need to think about redesigning your tables.

If your WHERE or JOIN conditions include OR statements (which are slower) you may get better speed using a UNION statement.

UNION ALL is faster than UNION if (And only if) the two statments are mutually exclusive and return the same results either way.

NOT EXISTS is usually faster than NOT IN or using a left join with a WHERE clause of ID = null

In an UPDATE query add a WHERE condition to make sure you are not updating values that are already equal. The difference between updating 10,000,000 records and 4 can be quite significant!

Consider pre-calculating some values if you will be querying them frequently or for large reports. A sum of the values in an order only needs to be done when the order is made or adjusted, rather than when you are summarizing the results of 10,000,000 million orders in a report. Pre-calculations should be done in triggers so that they are always up-to-date is the underlying data changes. And it doesn't have to be just numbers either, we havea calculated field that concatenates names that we use in reports.

Be wary of scalar UDFs, they can be slower than putting the code in line.

Temp table tend to be faster for large data set and table variables faster for small ones. In addition you can index temp tables.

Formatting is usually faster in the user interface than in SQL.

Do not return more data than you actually need.

This one seems obvious but you would not believe how often I end up fixing this. Do not join to tables that you are not using to filter the records or actually calling one of the fields in the select part of the statement. Unnecessary joins can be very expensive.

It is an very bad idea to create views that call other views that call other views. You may find you are joining to the same table 6 times when you only need to once and creating 100,000,00 records in an underlying view in order to get the 6 that are in your final result.

In designing a database, think about reporting not just the user interface to enter data. Data is useless if it is not used, so think about how it will be used after it is in the database and how that data will be maintained or audited. That will often change the design. (This is one reason why it is a poor idea to let an ORM design your tables, it is only thinking about one use case for the data.) The most complex queries affecting the most data are in reporting, so designing changes to help reporting can speed up queries (and simplify them) considerably.

Database-specific implementations of features can be faster than using standard SQL (That's one of the ways they sell their product), so get to know your database features and find out which are faster.

And because it can't be said too often, use indexes correctly, not too many or too few. And make your WHERE clauses sargable (Able to use indexes).