How to handle massive storage of records in database for user authorization purposes?

Question

I am using Ruby on Rails 3.2.2 and MySQL. I would like to know if it is \"advisable\" / \"desirable\" to store in a database table related to a class all records related to

Answer 1

Shard the ArticleUserAuthorization table by user_id. The principle is to reduce the effective dataset size on the access path. Some data will be accessed more frequently than others, also it be be accessed in a particular way. On that path the size of the resultset should be small. Here we do that by having a shard. Also, optimize that path more by maybe having an index if it is a read workload, cache it etc

This particular shard is useful if you want all the articles authorized by a user.
If you want to query by article as well, then duplicate the table and shard by article_id as well. When we have this second sharding scheme, we have denormalized the data. The data is now duplicated and the application would need to do extra work to maintain data-consistency. Writes also will be slower, use a queue for writes

Problem with sharding is that queries across shards is ineffectve, you will need a separate reporting database. Pick a sharding scheme and think about recomputing shards.

For truly massive databases, you would want to split it across physical machines. eg. one or more machines per user's articles.

some nosql suggestions are:

1. relationships are graphs. so look at graph databases. particularly
   https://github.com/twitter/flockdb
2. redis, by storing the relationship in a list.
3. column-oriented database like hbase. can treat it like a sparse nested hash

all this depends on the size of your database and the types of queries

EDIT: modified answer. the question previously had 'had_one' relationships Also added nosql suggestions 1 & 2