Distributed system design using only C

Question

I have the work of implementing a distributed system of nodes (like p2p nodes) each of these nodes (lets say A,B,C and D) perform certain functions and need to interact with

Answer 1

I won't try to give a "whole" answer (because the question is way too large & vague anyways) but I could point you towards an interesting piece of the puzzle:

1. You could use a Message Queuing system (e.g. AMQP RabbitMQ: there is an experimental C binding available) to implement reliable message delivery between your nodes.

2. Mutual Exclusion: you can use a protocol such as Paxos