golang + redis concurrency scheduler performance issue

Question

I write a simple concurrency scheduler, but it seems to have a performance issue at a high level concurrency.

Here is the code (scheduler + concurrent rate limiter test)

Answer 1

My test result shows, when the routine number increases, the execution time per routine per take function increases nearly exponentially.

It should be a problem of redis, here is reply from redis library community:

The problem is what you suspected the pool connection lock, which if your requests are small / quick will pushing the serialisation of your requests.

You should note that redis is single threaded so you should be able to obtain peak performance with just a single connection. This isn't quite true due to the round trip delays from client to server but in this type of use case a limited number of processors is likely the best approach.

I have some ideas on how we could improve pool.Get() / conn.Close() but in your case tuning the number of routines would be the best approach.