This is kind of more generic question, isn\'t language-specific. More about idea and algorithm to use.
The system is as follows:
It registers small loans bet
Only if someone owes more than 2 people, whom also owe to the same set, can you reduce the number of transactions from the simple set.
That is, the simple set is just find each balance and repay it. That's no more than N! transactions.
If A owes B and C, and some subset of B C owe each other, so B owes C, then instead of: A -> B, A -> C (3 transactions). You'd use: A -> B, B -> C (2 transactions).
So in other words you are building a directed graph and you want to trim vertices on order to maximize path length and minimize total edges.
Sorry, I don't have an algorithm for you.