Relational databases - there has to be more right?

Question

I really enjoy database design and the whole concept of managing data semantically and all the logic that comes with it.

My knowledge level when it comes to databases is

Answer 1

Disclaimer: not an expert in database design.

Some of the performance issues can be handled either by:

1. denormalizing your database, so to reduce the number of tables to join
2. adding indexes
3. filtering should be done so that you first remove the largest of the non matching data, then you cherry pick the next condition on the reduced set. It's better to go from 100 values -> 10 matching first condition -> 1 matching first and second condition than 100 values -> 80 matching second condition -> 1 matching first and second condition. Seems trivial, but it's important to keep in mind.

4. divide et impera is the motto for scalability. If you have something that scales in a non-linear way, say O(N^2) it makes sense to keep N as low as you can, and you should partition your data set into smaller sets, assuming they are independent and you can work out the partitioning. An example of this is sharding, typically used to keep databases of users in large social websites. (NB: an example, I would not implement it this way) Instead of having a huge database with all the users, they have 26 servers (one for each letter of the alphabet), then they put all the nicknames with the same first letter in the same server. This has the following advantages:

   a. you balance the load on different machines
   b. if one machine crashes, you make the site unaccessible only to a subset of your users, not to all of them
   c. you preselect the search with a highly discriminating criterium (the first letter), then perform the second search (the username)
   d. you reduce the number of entries each database has.