High-concurrency counters without sharding

Question

This question concerns two implementations of counters which are intended to scale without sharding (with a tradeoff that they might under-count in some situations):

Answer 1

Going to datastore is likely to be more expensive than going through memcache. Else memcache wouldn't be all that useful in the first place :-)

I'd recommend the first option.

If you have a reasonable request rate, you can actually implement it even simpler:

1) update the value in memcache
2) if the returned updated value is evenly divisible by N
2.1) add N to the datastore counter
2.2) decrement memcache by N

This assumes you can set a long enough timeout on your memcache to live between successive events, but if events are so sparse that your memcache times out, chances are you wouldn't need a "high concurrency" counter :-)

For larger sites, relying on a single memcache to do things like count total page hits may get you in trouble; in that case, you really do want to shard your memcaches, and update a random counter instance; the aggregation of counters will happen by the database update.

When using memcache, though, beware that some client APIs will assume that a one second timeout means the value isn't there. If the TCP SYN packet to the memcache instance gets dropped, this means that your request will erroneously assume the data isn't there. (Similar problems can happen with UDP for memcache)

Answer 2

Memcache gets flushed, you lose your counter. OUCH. Using a mysql database or a NOSQL solution will resolve that problem with a possible performance hit. (Redis, Tokyotyrant, MongoDB etc...) may not have that performance hit.

Keep in mind, you may want to do 2 actions:

1. keep a memcache counter just for the high performance reasons.
2. keep a log, and then get more accurate metrics from that.