sharding

I have been looking for some articles or guidence on best practice for id generation (for the federated/primary key) for Azure Federated databases and haven't found anything compelling. Federated tables don't support identity columns, so it seems to me that the only practical type of id is a GUID, as trying to centrally create and use a BigInt creates a single point of failure in the app. My chief concern is the performance implications of using GUIDs over BigInts (particularly for indexing the tables). Is there any recommended/best practice (or existing libraries) to create unique BigInts for

We're a small startup with a write-heavy SAAS app and are (finally!) getting to the point where our usage is presenting scaling issues. We have a small team, so we really appreciate being able to offload sysadmin to Heroku and RDS. While Heroku is (mostly) fine, we have a couple problems with RDS: Scaling. This is the biggest concern. We currently run an XL RDS instance. We'll be able to get by for a while longer with straightforward optimizations, but unless we make some major structural changes to our app, we'll hit a bottleneck at some point. Also, the downtime for changing instance size

What's the best way to deal with a sharded database in Rails? Should the sharding be handled at the application layer, the active record layer, the database driver layer, a proxy layer, or something else altogether? What are the pros and cons of each? John Topley FiveRuns have a gem named DataFabric that does application-level sharding and master/slave replication. It might be worth checking out. I assume with shards we're talking about horizontal partitioning and not vertical partitioning ( here are the differences on Wikipedia ). First off, stretch vertical partitioning as far as you can

After reading the official documentation for the MongoDB sharding architecture I have not found out why you need to have one or three config servers, and not another number. The MongoDB documentation on Config Servers says: "If one or two config servers become unavailable, the cluster’s metadata becomes read only. You can still read and write data from the shards, but no chunk migrations or splits will occur until all three servers are available." Hence the reflection: one server is equivalent to a single point of failure, but with two servers we have the same behavior as three, right? So, why

I'm working on a web app that is somewhere between an email service and a social network. I feel it has the potential to grow really big in the future, so I'm concerned about scalability. Instead of using one centralized MySQL/InnoDB database and then partitioning it when that time comes, I've decided to create a separate SQLite database for each active user: one active user per 'shard'. That way backing up the database would be as easy as copying each user's small database file to a remote location once a day. Scaling up will be as easy as adding extra hard disks to store the new files. When

I am trying to grasp the different concepts of Database Partitioning and this is what I understood of it: Horizontal Partitioning/Sharding : Splitting a table into different table that will contain a subset of the rows that were in the initial table (an example that I have seen a lot if splitting a Users table by Continent, like a sub table for North America, another one for Europe, etc...). Each partition being in a different physical location (understand 'machine'). As I understood it, Horizontal Partitioning and Sharding are the exact same thing(?). Vertical Partitioning : From what I