Distributed system design using only C

Question

I have the work of implementing a distributed system of nodes (like p2p nodes) each of these nodes (lets say A,B,C and D) perform certain functions and need to interact with each other for various operations such as synchronize operations and other things like 15 A nodes interact with a group of 5 B nodes to get into the least loaded node's queue and get a token number and thereafter wait for C to redirect them to a free node D and so on.

I am a bit lost as to how should i go about the design:

1. The protocol that i have thought of is to encapsulate a struct of the type of operation and other things to be sent. Also, this is done using an acknowledgment scheme so i can be sure the other side got the message.

2. How do i go about the distributed mutual exclusion aspect as I do not have a central server. I am guessing each node replicates the data, but this sounds a bit too expensive(not to mention stupid).

3. What is the basic design methodology followed while implementing p2p systems, ie how do i implement the program such that it is blocked on a receive but also can send further updates et al and at the same time get information from others about the 'state' of the whole system.

4. How do i ensure total ordering of requests?

Also, what are the other issues that I may need to look at/face. I also would appreciate if you could point me towards some good online resources on implementing p2p and distributed systems.

Thanks!!

Answer 1

I won't try to give a "whole" answer (because the question is way too large & vague anyways) but I could point you towards an interesting piece of the puzzle:

1. You could use a Message Queuing system (e.g. AMQP RabbitMQ: there is an experimental C binding available) to implement reliable message delivery between your nodes.

2. Mutual Exclusion: you can use a protocol such as Paxos

Answer 2

The secret to prevent blocking is your end points must all be written as servers with threads for "protocol" processing that are separate from the threads for data processing.

As for line protocol, I've become enamored with JSON for line protocol. It is human readable. It is streamable without need for length bytes! It is easily extensible and mostly immune to protocol version change.

Answer 3

Yet another piece of software worth a look could be KadC!

http://kadc.sourceforge.net