Best way to implement global counters for highly concurrent applications?

Question

What is the best way to implement global counters for a highly concurrent application? In my case I may have 10K-20K go routines performing \"work\", and I want to count th

Answer 1

The other answer using sync/atomic is suited for things like page counters, but not for submitting unique identifiers to an external API. To do that, you need an "increment-and-return" operation, which can only be implemented as a CAS loop.

Here's a CAS loop around an int32 to generate unique message IDs:

import "sync/atomic"

type UniqueID struct {
    counter int32
}

func (c *UniqueID) Get() int32 {
    for {
        val := atomic.LoadInt32(&c.counter)
        if atomic.CompareAndSwapInt32(&c.counter, val, val+1) {
            return val
        }
    }
}

To use it, simply do:

requestID := client.msgID.Get()
form.Set("id", requestID)

This has an advantage over channels in that it doesn't require as many extra idle resources - existing goroutines are used as they ask for IDs rather than using one goroutine for every counter your program needs.

TODO: Benchmark against channels. I'm going to guess that channels are worse in the no-contention case and better in the high-contention case, as they have queuing while this code simply spins attempting to win the race.