Bulletin board - Database optimisation

Question

This question is a follow on from this Question

The project and problem

The project I am currently working on is a bulletin board for a large non-profit

Answer 1

Subquery First, then the RANK() Function

Relax, son, we'll get there! Your speed is fine.

Preparation

The first thing, you really need to get access to a decent set of manuals, for your specific flavour of MySQL. I found ▶this one◀. As before, you have to do your own debugging, but I am now providing SQL that is as close to generic MySQL as possible. I've confirmed that everything we are going to be doing is entirely possible in that flavour of MySQL (I don't know what flavour/version yours is, except ENGINE=MyISAM).

Subquery

Ok, let's start again. I have written a ▶series of SELECTS◀, to lead your through the process. Please complete each one, and understand it completely before progressing to the next. If you have any questions, stop, and post the question.

The code is written and tested in Sybase; then downgraded for MySQL (from perusing the web, eg. the above site), and tested as much as possible in that state.

The first bit creates and loads three tables for use.

1. The first SELECT is a straight join of the three tables, no subquery. You need to get that to work; that is, understand what is does, fix any syntax problems; figure out the differences between the SQL I provide and the SQL runs on your server. And get used to making those changes. We can't keep stopping for that.

2. The second SELECT produces exactly the same result set. It introduces the concept of a Subquery, which is used to populate a single column.

Drive that bus. Respond when you're done or if your have problems.

Responses to Your Comments of 03 Dec 10 17:51

1. Straight Join
   I have never seen that way of doing joins before, I have always used left join, right join or inner join. Ok so for this first query we are just joining the two tables student and course with the studentcourse table sitting in the middle as the associative table. Results are repeated as expected because one student might be on more that one course and they will have a result for that course.

Yes.

That ( x=y in the WHERE clause ) is the traditional way of identifying joins, it is much more clear; the LEFT/RIGHT/INNER/OUTER JOIN syntax is the "new" way. Much more cumbersome AFAIC, but the learning is relevant because it is fundamental to what comes later. Feel free to convert to the latter syntax, and back again, for purposes of understanding.

Repeats ? That is not what repeats or duplicates mean. All the rows are discrete, true rows in CS. You should get the same 15 rows in every report (as we progress).

(ps when i direclty create the tables using queries you provided, the names are converted to all lowercase while the column names can still be camel case.)

MySQL is very strange. (It appears to be doing the naming conventions for us!)

.
2. Simple scalar query
A few issues with query. You use the alias(in the scalar subquery) before you have defined what it is?(StudentCourse sc) I guess I always incorrectly assumed that you have to say define an alias before you use it.

You are thinking procedurally. SQL is a set-oriented language, for manipulating Relational sets of data.

The whole query gets evaluated and optimised in one pass. There is no "before" or "after". I am defining it in the same batch of SQL that I am using it.

I don't entirely understand the use of the alias 'in-ner' in the scalar subquery, is this to say that you want it to check each row individually(not sure how to explain this) instead of on a table wide check?Ie when you are doing this check make it local to the particular row you are on?(terrible explanation sorry).

For purposes of understanding/debugging, evaluate the subquery first (the contents of the brackets), alone. Understand it fully. Note the use of "sc" and keep it in your hat.

• in_ner and sc are ALIASES, that is, handles for the table name that it sits next to in the FROM clause; that we use elsewhere in the code for convenience
• in_ner is a descriptive name for the table referenced in the Inner Query, the Subquery
• sc is a descriptive name for the table referenced in the Outer Query, which is only Outer because it has an Inner query, otherwise it would be a flat query
• we could just as easy use fred and sally
• Aliases such as in_ner and out_er are meaningful when the same table is referenced in both the Inner and Outer queries.
• notice the join between the Inner query and the Outer query WHERE in_ner.CourseId = sc.CourseId
• I have related the table referenced in the in_nerquery to the table sc referenced in Out_er query
• Such a subquery is called a Correlated Subquery

See if you can visualise the Outer query (result set) as a grid, a spreadsheet, 15 rows by 4 columns.

• Make sure you understand that Outer query, "easy" as it is. Notice that it is the same as (1. Straight Join), with a different method of populating one column.

As i understand it the scalar subquery asks for Name where the courseId's in Course and studentcourse are the same.(pretty straight forward) and is an alternative to saying that in the where,

Yes, exactly.

And notice that we are after only the Course.Name which is a 1::1 join from StudentCourse to Course, on CourseId. Notice exactly the WHERE clause in (1) that we are replacing in (2); in (1) it applies to all rows.

But because we are grabbing one datum; one cell; one item for a specific row/column; not all rows; not all columns, it is called a Scalar.

We are obtaining it using a subquery, which has to be constrained to the specific row. Therefore we need to relate the row from the outer query to the row in the Inner query.

• so the Correlation between the Inner Subquery and Outer (specific row) is required.

• And if we did not have that identification of the specific row, we would be loading rubbish into the Scalar, or it would return a Table (not a Scalar value) and the query would fail.

  • Try that, take the WHERE CourseId = sc.CourseId out
  • So that you know what the error message is, so that when it happens in future, you will know "Aha, I am returning a table, not a scalar; I am missing something in my Inner WHERE clause; I am not identifying a specific Correlated row".
    .
• it is not quite "asks for Name where the courseId's in Course and studentcourse are the same"; it is getting the Course.Name for a specific StudentCourse.CourseId, which is identified from the outside, whatever sc row it is.

with the differnece that you can make this check row by row before the where.

you are thinking procedurally; there is no "row-by-row"; the dbms is set-oriented; the result set you are building is a set. Re-state the question is set terminology.

I used Course instead or in-ner, what is the point of using an alias in this case, is it just to show that aliases can be used?

Yes. And to highlight issues. And to differentiate the Inner Query from the Outer query. In the Inner query, the "inner" Alias, or any alias is not demanded. Only the Alias relating to the outer query is demanded.

Something I don't understand here is that when I try to do this, 'course.Name' it says unknown Course.Name in field list. this is the way that I have always defined that i mean Name in the Course table and not some other table. What would happen if I had two tables with a name column?

Exactly. If it were ambiguous, then you would have to supply the table name or alias; where it is not ambiguous, it is not demanded, but nice to have for documentary, clarity, purposes. You have to figure out why MySQL is not accepting it. Mixed case/lower case madness ?

I have also never seen that order by syntax, I can see that 1 and 4 mean the column numbers but why bother passing it two columns?

Huh ? Because I want the result set ordered by Course.Name in ascending order, and within that, by StudentCourse.Mark in Descending order.

If I did not state the order, MySql would produce the result set in whatever order it gets it from StudentCourse (chronological ?; by index ?). Whatever that default order is, find that out, you need to know it, and thus avoid an ORDER BY, when it is unnecessary.

• Take the ORDER BY out and play with it.

• Try ORDER BY 4 DESC, 1

It is not "passing", I am telling it what to do with my result set, in the one SQL command. The only passing you are doing is between your app (PHP ?) and MySQL.

2.1. Ok, when you finished with (2), and completely happy that you understand it, do this exercise.

SELECT  (SELECT Name
            FROM  Course
            WHERE CourseId = sc.CourseId
            ) AS CourseName,
        () AS FirstName,
        () AS LastName,
        Mark
    FROM StudentCourse sc
    ORDER BY 1, 4 DESC

• Produce the same grid format, we want the exact same result set as (1) and (2).

• Fill in the two pairs of empty brackets with the appropriate subquery; ie. write a subquery to populate the FirstName column, and another to populate the LastName column

Responses to your Comments re Third Data Model

2.1. Perfect, yes, we move on.
.
You are cooking with gas, so if you don't mind, I will take your text, and annotate it a bit; notice the differences, they may or may not be subtle.

The correlated scalar subquery says that for each course id we need the highst mark, as opposed to the highest mark for all the courses. This is where the correlated aspect of this subquery comes into play because we are relating the outer query to the inner query for this particular row. [Yes!] The way that I am currently visualizing [That's it, use the visual part of your mind, not the serial part] it is that the outer query runs through the tables putting together the result table set, and each time it creates a row it runs the scalar subquery and picks out [a single value to fill the cell; here it is] the highest mark where the courseId's match, so when it is on a row where the course id is 66 then the scalar subquery is only looking for the max mark where the courseId is 66.

I could hardly have said it better myself.

There is no such thing as "result table".

Add one more definitive item.

• The outer query defines the result set.

  • The subquery is independent of that; it is merely Correlated or Indexed.

Ok, so you have that SQL working, right ?

Now that you understand that, the next step is to visualise the result set, and to visulaise the subquery (3, unchanged) filling the entire column. if the above text was a balloon filling one cell at a time, then visualise hundreds of ballons, filling consecutive cells. Then visualise a bucket poured into the column.

Now leave that two dimensional result set alone for a minute, and visualise another layer on top of it. This is the parallel layer, where you write your subquery code.

If ever you have difficulty getting a subquery to work, go back to this, your way of visualising, one result set, and another layer for the subquery, which pours a bucket of scalars in, to fill the column. It eliminates all the well-known subquery coding bugs; removes the use of GROUP BY, DISTINCT, and all those ham-fisted methods of getting a long angry snake to fit into a jam jar.
.
Three more small steps before you proceed to (4).

2.2 Re-read my response (2) above, all the way down to this point. No skimming. This is because when you teach your mind something new and different, you need to re-inforce it. It is an officially recognised and labelled technique.

Responses to Comments of 08 Dec 10 20:49

2.3. Write that query (3) without using subqueries, and ensure you check the results. If you catch yourself laughing when you are writing the code, it is a good sign. As long as you produce the correct result set, you pass, but try to write the most efficient code (fewest COUNTS and GROUP BYs, etc). Do this only if you want to run circles around your peers, to be able to answer any "how do I code ..." question on your database.

I'm not sure what you mean by write that query without using sub-queries? I thought we wanted to avoid the use of group by's etc

Yes. Absolutely. You've walked forward. Now walk backward without tripping. This will really help your understanding of walking forwards, when it is better to use a subquery vs a join. Code the query with GROUP BYs and COUNTs. The fewest. Don't laugh.

2.4. Write the subquery (3) on your database, to produce a list of Bulletins, the outer query has to be FROM bbs only; with a count of likes, and a count of dislikes. So trunacte the tables and do 10 or 12 meaningful INSERTS, fibe minutes, big deal.

I used the method of using sub-queries on my database to put together a list of bulletins replies, count the number of reply likes and dislikes and get a particular users rating. it was great because I didn't have to use any group by's or counts and I didn't have to create temporary tables like I did for the bulletins.

Well, that's perfect. Now we are getting a bit of Relational Power in your spinach.

Now, go and look at this question and answer; ensure you compare the code. You've come a long way in just a few days.

When you finish (2.3), read your (2.4) query again, to refresh yourself, and move onto (4).

If you get stuck, replace the word "Rank" with "CountOfStudsWithHigherMark", and give it another go.

Responses to Comments of 11 Dec 10 13:14

2.3 I am having trouble writing that query without a scalar subquery. Scalar subqueries always made more logical sense to me even before I knew how to do them. That is why I said "I guess the problem I am running into here is, how do you refer to user-id = x in this particular row, not in all the row" in that previous question. Correlating the scalar subquery to the main query with and alias was the answer.

The (2.3) exercise is intended for you to:

• really understand the incorrectness of the fat query with the GROUP BY (in a relational database using a set-processing relational engine) vs the correctness, elegance, and speed of the Correlated Subquery. You have achieved that. That will place you above your peers, in terms of SQL coding ability.

• be able to identify when a fat WHERE clause and when a Correlated Subquery is appropriate. I am not sure, but it looks like you have achieved that.

• be able to correct and debug this kind of issue when maintaining code written by others, and to be able to teach them the distinction. It sounds like you have a good visual, relational ability; which has been re-inforced by the exercise; and now you cannot go back to inferior methods. That is, you can understand and fix incorrect SQL code, but you cannot communicate that to others.

As long as you understand those distinctions and accept that, I am happy to drop (2.3) and move on.

Read your (2.4) query again, to refresh yourself, and move onto (4).

If you get stuck, replace the word "Rank" with "NumStudentsWithHigherMark", and give it another go.

---

Don't read further. The following is "old code"

Here's a ▶Quick Tutorial◀ on the RANK() operator (as it is commonly known). It is not ANSI SQL; it is an Oracle and MS extension. However it is not required if you understand Subqueries, which is why Sybase does not have it. I doubt MySQL has it, so you need to get your head around it. Understanding Scalar Subqueries is a pre-requisite. Sybase syntax, so whack your semi-colons in, etc. Feel free to ask specific questions.

I have never seen that approach of writing Rank = (SELECT.... Is that the same as (SELECT ...) as Rank?

Yes, () AS Rank instead of Rank = () are both legal SQL; MySQL may not like the latter form. The brackets containing the Subquery, of course. Note that Rank is the name of the derived column.

I have already stated that understanding subqueries is prerequisite. That means that millions before you have had this problem, and the lecturers figured out that you would suffer less frustration if you followed the lessons in the prescribed order. So forget RANK for now, and learn subqueries.

Try this (I supply ANSI Standard SQL; I do not have MySQL; you will have to syntax-fix it for MySQL; I don't fix syntax problems; that's your job):

SELECT  COUNT(*)+1 AS Id_iot  -- not you, everyone who uses them blindly
        (SELECT title in_ner FROM bb_locations WHERE out_er.bb_locations_id = in_ner.id) AS Location,
        title AS Bulletin,
        created_date AS Date
    FROM bbs out_er

• in_ner and out_er are ALIASES, that is, handles for the table name that it sits next to in the FROM clause; that we use elsewhere in the code for convenience
• in_ner is a descriptive name for the table referenced in the Inner Query, the Subquery
• out_er is a descriptive name for the table referenced in the Outer Query, which is only Outer because it has an Inner query, otherwise it would be a flat query
• we could just as easy use fred and sally
• notice the join
• I have related the table referenced in the in_nerquery to the table referenced in out_erquery
• Such a subquery is called a Correlated Subquery
• This is just an example, simple, so that you can learn Subqueries; purposely chosen to provide the same result set as one you are familiar with producing, using straight joins (bbs and bb_locations in the FROM clause, joining via the WHERE clause or JOIN syntax).
• Because it produces a single value, it is called a Scalar Subquery (those that produce rows are Table Subqueries; and cannot be used like this, to load a single value into each row)
• There is no suggestion that anyone should "use Subqueries instead of Joins". Absurd. Subqueries have their place, and Joins have theirs. Misue is a different thing.

Now, drive that bus. And don't talk to me about RANK until you can drive that bus around every corner in your database neighbourhood without killing any children.

I don't understand inner and outer, when I google them I get INNER JOIN what are they called so I can research further

Aliases. Refer above.

When I run that select statement I get this error You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE inner.Mark >= outer.Mark ) FROM studentmark outer ORDER B' at line 5

• first, as per reasons detailed above, I can't write MySQL syntax, and debugging is your job
• second, I realiise that you can't debug what you can't understand, so drop it for now (it has to do with RANK) and as you learn the MySQL flavour of SQL, all these things will be resolved
• third, let me assure you that it runs on any Standard SQL server. It gets used in about 10 courses a year, so hundreds of participants per year. I just ran it again on Sybase, just to check.
• first thing I would suggest is, since the MySQL optimiser sucks dead bears; it does not understand context, inner and outer are probably being treated as reserved words. So change that as per the above code.