Bulletin board - Database optimisation

Question

This question is a follow on from this Question

The project and problem

The project I am currently working on is a bulletin board for a large non-profit

Answer 1

Part II

Continuation of Part I, due to that Answer reaching maximum length.

Revised 14 Jan 11 - 05:40 PST

Comments re 11 Dec 10 13:14, Fifth Data Model and Responses

a. IDEF1X Design/Diagramming tool.
I do not know of any freeware options. The MySQL design tool reportedly crashes often. If you are happy with my diagrams, I am happy to work with you for the duration, until the final model is resolved; ie. I provide the Data Model, and you can skip that task. For ongoing work, yes, you need a diagramming tool, perhaps not a database design tool. Refer my comments at the end of p2 in the Notation doc.
.
29. Are you clear about the PKs and FKs in each table, as per the coloured tabs in the Fifth Data Model; can I remove the tabs now ?
.
38. Closed.
.
39. All the Dtm columns will be MySQL DATETIMEdatatypes. The variables you use for those columns should be the same. TimeStamp has a different meaning. Using the correct Datatypes is the first (big) step towards ensuring that the data is coorect and no illegal values are allowed to enter the db. Ie, only valid dates and times will be allowed. Further, you can interrogate any date or time component (eg. month or day name) from it. Check this document.
.
40. No Problem. Instead of having just the one category hardcoded, how about (like the handling your of Permission), we implement an Category.IsRestricted and then Permission 5 becomes Post Restricted Bulletins.
.
41. Done.

You should think about doing the same for Category and User. You want to be able to delete them by setting the Indicator, without removing the entry (and all the Bulletins, Responses, Replies, etc) fro the database. It has to be retained for historical purposes, but you need to disallow the User from logging in and doing anything. I have included this in the DM.

For such column names and Booleans in general, personally I prefer to identify the minority or exception case, as in IsObsolete.

20.2. Done. Table and column naming now progressed to InnoDB format.
.
Subquery responses in Subquery Answer.

Comments 13 Dec 10 13:14 EST and Responses

.
41. See (41) and next para, above.
.
42. I meant: either Title or Description is enough; we do not need both.
.
43.1. Implementation of Data Model. Go ahead. That's why I gave you the Physical yesterday.
.
43.2. Design/Drawing. Go ahead. I have already commented in (a) above.

Data Model

Sixth Data Model supplied, containing all changes as per above.

The Physical means a lot more detail required for implementation/coding: Datatypes; n::n Relations implemented as Associative tables; etc. You are pretty much ready to implement the Data Model, which means you need the Physical. And you already have the Associative tables figured out. Therefore I have taken the liberty of providing you with the Physical DM, even though you said you were in no hurry.

• Note that Domains (User Defined Datatyptes) should always be used in a database, both for the DDL; the $variables you use. And a private Domain for each Primary Key. But this is not possible in MySQL, therefore the Datatypes are raw, regrettably.

• Fixed length columns are much faster than variable length; I do not provide (advise) Var length. You are free to implement what you like.

• Are you sure you need both Category.Title and Description ? I think not, but I have left it in until you confirm.

• Enjoy the little blue glass buttons, and the navigation from the Collapsed Entities.

Please read the IDEF1X Notation document again, I expanded it last week.

Depending on how the Open Issues close, and any issues you may have, we can progress another edition, in the next day/night.

Comments 28 Dec 10 10:34 and Response

I have begun implementing the data model. I assume that the 6th data model is the physical model because it contains the associative tables.

Yes, I supplied that, and the Datatypes, because you said you were ready to implement.

There are still a few minor outstanding items. May be a good time to go through your question; all three of my answers, and check. Category.Title and Description, for instance.

I will put up a database dump once I am done.

That is not necessary, given that the model has the Dataypes defined; but if you do post it, sure, I will check it for you. Email may be better.

I will then put up a list of all the queries that I need to run on the database and begin writing them.

Very good idea, to take a structured and planned approach to the job.

Implementation of Physical model

(39) With mysql I am not able to assign more than one primary key so I am just going to make them unique and not null as you suggest in the documentation. Do you think it would be a good idea to index them as well?

Not sure what you mean, what is "them" ?:

• you can never have more than one Primary Key on a table; the Alternate Keys are Unique, one of them is "primary"; that is carried as FK in the child tables.
• with InnoDB (what you said you will get), you can define PRIMARY KEY constraints (which is equivalent to UNIQUE, NOT NULL)
• with MyISAM (what you have now), you need an Index, UNIQUE, NOT NULL for the Primary Key (above the line in the model)
• for either InnoDB or MyISAM, each Alternate Key (AKx[.y] in the model) must be defined as an additional index, UNIQUE, NOT NULL.

Comments 07 Jan 11 14:08 and Response

(40) Could you explain why the category.CategoryCode is a char of 4 characters. Why not just use an number like we do for user?

40.1. The idea is to use good natural Identifiers. Numbers are meaningless to users. If we didn't have a large no of Users, and User churn, I would not have used a number there either. A CHAR(2) or (3) or (4) allows them to pick meaningful short code for the long Category.Description, and it is small enough to be carried as a Foreign Key in user_category and bulletin.

For the developer, when testing and debugging, that short code in a list of say bulletins, will be very handy.

(40) I don't quite understand permission for category and location. Lets say that I want all users to be able to post to the Travel category. Would I set the permission of this category at 4? Why do we need to bool IsRestricted when we are giving a permission to the category and location?

40.2. I have not changed the concept or essence of Permission; it was your idea, and it remains exactly as you explained it to me.

(All I did was implement permission as a table.)

40.3. Refer (13) and (40) original exchange. category.IsRestricted defines restricted categories; there are two categories of categories, Restricted and Unrestricted. The users need a permission of 5 to post bulletins re Restricted categories, and 4 to post bulletins re Unrestricted categories.

40.4. But whoa, son, are you introducing a change or extension; eg. match the permission of the user to the permission of the category, thus allowing far more than two categories of categories ? Please don't. That would mean permission means one thing re category and a different thing re the rest of the system. Or if you do, then we have to resolve the exact need first, then implement it as a change.

40.5. Location (now office) is exactly the same for that bit (40.2) and (40.3). If you are referring to the text at the bottom, it is my small mistake, will correct it.

40.6. As per (14), office additionally has a single AdministratorId (UserId). Shown on the model as (permitted) user.

40.7. But that brings up an issue: who can administer categories ? Right now anyone with permission 5 or greater, which is a different thing. I think we need something explicit, a permission 6 = Administer Category.

Other

• Processed your DDL and returned.

• Data Model Updated. Number of small clarifications and two minor errors corrected.

Comments 08 Jan 11 14:08 and Response

(I think that was 09 Jan not 08 Jan ... I did check for updates.)

(40) I don't quite understand permission for category and location. I haven't changed anything. Disregard the content of the previous question as explain what permission would be set to allow users to post to the 'Travel' Category. In my implementation I simply had a permission column. If a given user had the required permission or greater then they could post to a category, is that how the new system works?

Yes. Unchanged. category.permission has nothing to do with it. They need user.permission 4 for unrestricted categories.

If the category.IsRestricted, they need user.permission 5.

Quite separately, an user needs user.Permission of whatever category.Permission is, in order to administer category. Do not use values less than 4.

(41) Query Delete a bulletin and all its associated replies and ratings.

I did not expect that, are you sure they have no need to keep all past bulletins for historical or audit purposes ?

Anyway, let's deal with that on the basis that deleting bulletins is allowed ...

I don't even know where to get started on this. In the past I have joined would have joined up the bulletin table with the response table and the response table with the response rating table and the bulletin-rating table where the bulletin id = x and deleted them. But now any one particular bulletin is identified by three columns: the OfficeCode,IssuerId and BulletinDtm. Which are carried to the child tables as foreign keys. For a start, how do I store indicate which bulletin is to be deleted in my php? Usually I would have a link like this index.php?action=delete&bulletin-id=5. Now will I have to have a link that is index.php?action=delete&OfficeCode=20001&IssuerId=34&BulletinDtm=14:02... I really have no clue how to do this?

a. I can't help you there, I am a database and SQL expert, not a php or MyISAM expert. You will need to post that as a new question on SO or the MySQL boards.

b. As far as my reading of that subject has taken me, I don't know the syntax, but yes, it can be done, it is normal. I checked before recommending composite keys to you. The corrected DDL succeeded, and the indices are confirmed, correct ?

c. The issue is simply the syntax required for composite or compound keys; and working with index.php. Something like:

index.php? action=delete & OfficeCode=x and IssuerId=y and BulletinDtm=z

d. Why can't you use mysql_query instead of index.php and thus use full SQL ? As I understand it, it works with MyISAM. Then you can use:

`$sql = "DELETE $table WHERE OfficeCode=$OfficeCode AND IssuerId=$IssuerId AND BulletinDtm=$BulletinDtm";`

e. Delete response_rating first; then response; then bulletin_rating; then bulletin. When they switch to InnoDB they will have less changes to make.

f. Most important, you will have to get them to identify the basis on which a bulletin can be deleted. ANy and all bulletins should not be deleted. Something like "no activity for one year" or "closed" (which means an added column), etc.

Comments 10 Jan 11 14:08 EST and Response

(41.10-Jan-11) No problem, the method is fine, and I have detailed related issues which need address under (41.*) above. (41.f) still needs an answer ... other than permissions, is there any basis for deleting bulletins ?

Comments 10 Jan 11 13:48 pst and Response

SO Editing. Don't worry, it is not you. The site is of poor technical quality. The editing is hopeless (and believe me, I have tried to work with it and around it, to make my Answers appear even somewhat like I want them to appear). It cannot handle indents or more than one level of numbering correctly.

Delete Basis. Ok, you have a valid basis. And the users who wrote responses would not mind if they were deleted without being asked ?

(41) What you are looking for is a "cascading delete" in Standard SQL, which is defined in the Foreign Key clause (which you do not have in MyISAM). Each INSERT/UPDATE/DELETE verb applies to one table only, and may affect other tables by REFERENCE.

For non-standard SQLs, you have the DELETE multiple_table method (non-standard syntax).

First, it is very important to understand this, before anything else. The FROM and WHERE (or JOIN) clauses in a DELETE command are separate to the DELETE itself; they are in fact a SELECT. The idea is: DELETE table_one (SELECT FROM table_one, table_two WHERE join_conditions).

Therefore:

• name the four tables in the DELETE (target)
• name the four tables in the FROM (how you find them, via SELECT)
• ensure you have the correct (complete) JOIN clauses for the four tables; which you can test via a SELECT
  • which means, JOIN ON OfficeCode, IssuerId, BulletinDtm (the bulletin PK affecting the thre child tables)
  • use NATURAL or INNER joins, not left joins (be explicit, do not mix them up, as a general rule)
• ensure the WHERE identifies the specific bulletin composite Primary Key to the deleted.

Here's a link to the DELETE syntax and the JOIN syntax.

Comments 12 Jan 11 21:48 and Response

a. Don't be lazy. Write four delete statements, bottom up. That's what we have to do in the big end of town, where we do not have "cascading deletes". Write the delete for rating_response; then copy-and-paste, and delete one line of code each time. I do not understand the angst or avoidance.

b. I repeat, do not use left, right, or any kind of outer join (which is only required for the single all-encompassing delete). Use straight inner joins only (which is not a problem with 4 delete statements). Any and all upset you are experiencing is due to your need to use one delete. Give that up, and the upset and complication disappear.

• I just wrote this code for another question. That is a single SELECT command. Three-column PK times four Subqueries. I do not understand the need to avoid long (demanded, again, due to SQL being cumbersome) commands. And I didn't even use the JOIN syntax. Took me all of ten mins to write, plus five mins to test. What exactly, is the big deal ?

c. You have not forgotten the power of Relational keys, that you recognised some weeks ago, have you. Eg. ability to grab bulletin from rating_response, without having to join with rating. If you succumb to your single-column-key desires, you will lose all that. SQL is cumbersome. But that is all we have. Deal with it. The non-SQLs try to "make life easy" but in fact, introduce all sorts of unnecessary and avoidable complications. Case in point.

Comments of 13 Jan 2011 21:18 PST and Response

Deletes. Three flavours. Great. Hopefully you will have the data values in $variables, so there will not be that form of repetition. For testing, that is fine.

1. Delete x Four Tables. (not "Indiviudally deleting records", which is a different thing altogether; each delete except the last could net hundreds of rows). I trust the cut-and-paste took seconds. You need to be careful about forgetting to change the table names.

2. Single Delete Command. $variables for the first triplet. You could use column names in all but the first triplet.

3. Ok, so you will convert the SELECT to a DELETE, after testing. Left Joins. Required for the Single Delete but not otherwise. That's identical to (2) with the WHERE replaced with JOIN>

I've already recommended (1), but you are more likely to go with (3).