ServiceStack.Redis.Sentinel Usage

Question

I'm running a licensed version of ServiceStack and trying to get a sentinel cluster setup on Google Cloud Compute.

The cluster is basically GCE's click-to-deploy redis solution - 3 servers. Here is the code i'm using to initialize...

var hosts = Settings.Redis.Host.Split(';');
var sentinel = new ServiceStack.Redis.RedisSentinel(hosts, "master");
redis = sentinel.Setup();

container.Register<IRedisClientsManager>(redis);
container.Register<ICacheClient>(redis.GetCacheClient());

The client works fine - but once i shut down one of the redis instances everything craps the bed. The client complains about not being able to connect to the missing instance. Additionally, even when i bring the instance back up - it is in READ ONLY mode, so everything still fails. There doesn't seem to be a way to recover once you are in this state...

Am i doing something wrong? Is there some reason that RedisSentinal client doesn't figure out who the new master is? I feed it all 3 host IP addresses...

Answer 1

You should only be supplying the host of the Redis Sentinel Server to RedisSentinel as it gets the active list of other master/slave redis servers from the Sentinel host.

Some changes to RedisSentinel were recently added in the latest v4.0.37 that's now available on MyGet which includes extra logging and callbacks of Redis Sentinel events. The new v4.0.37 API looks like:

var sentinel = new RedisSentinel(sentinelHost, masterName);

Starting the RedisSentinel will connect to the Sentinel Host and return a pre-configured RedisClientManager (i.e. redis connection pool) with the active

var redisManager = sentinel.Start();

Which you can then register in the IOC with:

container.Register<IRedisClientsManager>(redisManager);

The RedisSentinel should then listen to master/slave changes from the Sentinel hosts and failover the redisManager accordingly. The existing connections in the pool are then disposed and replaced with a new pool for the newly configured hosts. Any active connections outside of the pool they'll throw connection exceptions if used again, the next time the RedisClient is retrieved from the pool it will be configured with the new hosts.

Callbacks and Logging

Here's an example of how you can use the new callbacks to introspect the RedisServer events:

var sentinel = new RedisSentinel(sentinelHost, masterName)
{
    OnFailover = manager => 
    {
        "Redis Managers were Failed Over to new hosts".Print();
    },
    OnWorkerError = ex =>
    {
        "Worker error: {0}".Print(ex);
    },
    OnSentinelMessageReceived = (channel, msg) =>
    {
        "Received '{0}' on channel '{1}' from Sentinel".Print(channel, msg);
    },                
};

Logging of these events can also be enabled by configuring Logging in ServiceStack:

LogManager.LogFactory = new ConsoleLogFactory(debugEnabled:false);

There's also an additional explicit FailoverToSentinelHosts() that can be used to force RedisSentinel to re-lookup and failover to the latest master/slave hosts, e.g:

var sentinelInfo = sentinel.FailoverToSentinelHosts();

The new hosts are available in the returned sentinelInfo:

"Failed over to read/write: {0}, read-only: {1}".Print(
    sentinelInfo.RedisMasters, sentinelInfo.RedisSlaves);