Application-level JOIN with WHERE and ORDER BY on N postgresql shards

Question

I have a postgresql cluster with different tables residing within different shards (different physical postgresql servers). EG:

shard A + user_group (user_group_id, user_group_name)

shard B + user (user_id, user_group_id (NULL), user_name)

shard C + comment (comment_id, user_id, comment_content)

I need to run queries that if all 3 tables where on the same shard, it would look something like:

SELECT comment_id, comment_content FROM comment INNER JOIN user ON comment.user_id = user.user_id LEFT JOIN user_group ON user.user_group_id = user_group.user_group_id WHERE user_group_id > 10 AND user_name LIKE 'foo%' ORDER BY user_group_name ASC, user_name ASC, comment_id ASC

How would such a query be implemented if the 3 tables are residing within 3 different physical postgresql shards?

I've read about references that one would have to 'do the join in the application layer' but I am not sure how to go about this. Some of the complexities include: 1. The cardinality of the different tables is unknown (or can change over time), so from the application layer (EG: php, python, etc...), we would not know if we should first go query user_group, get all the user groups, then query user, get all the users, etc..., or first query comment, get all the comments, then filter the retrieved comments by users, then filter by user groups, etc...

I am looking for a generic way to translate sql into application-level joins, and the above schema is only a hypothetical example.

Answer 1

You need to look into the db_link contrib.

Answer 2

Usually data divides at shards in a way which allow to avoid cross-server JOINS at all. Because this operation is difficult and expensive. If your example is hypothetical I would recommend divide all data by user_id field or user_group_id.

For example shard A will contain all tables with information from users which user_id % 3 = 0, shard B - which user_id % 3 = 1, shard C - which user_id % 3 = 2. So most of needed JOINS will be inside one shard. For some complex cross-server queries you may have common NO-SQL storage like memcached or Redis which will have copies of needed data from all shards (of course it's not full copy of all tables). Such storages can be easily replicated on as much servers as you need. This is how highload projects works.