How to Xtrim (trim redis stream) in the safe way

问题

Redis Xtrim is to keep a number of records and remove the older ones.

suppose I am having 100 records, I want to keep the newest 10 ones. (suppose java/scala API)

redis = new Jedis(...)
redis.xlen("mystream") // returns 100
// new records arrive stream at here
redis.xtrim("mystream", 10)

however, new records arrive between the execution of xlen and xtrim. So for example now there are 105 records, it will keep 95 - 105, however I want to keep start from 90. So it also trim 90 - 95, which is bad.

Any ideas?

回答1:

Option 1: If you always want to keep only the 10 newest ones, use:

XADD mystream MAXLEN 10 * field value ...

By moving trimming to the addition, you keep it always at 10. You can forget about separate logic to trim.

In Jedis,

xadd(String key, StreamEntryID id, Map<String,String> hash, long maxLen, boolean approximateLength)

Option 2: Optimistic, use a transaction.

WATCH mystream 
XLEN mystream
MULTI
XTRIM mystream MAXLEN 10
EXEC

Here, if the stream has been modified, the transaction will fail, EXEC returns null, you then retry.

Jedis transaction example

Option 3:

Use a Lua Script to adjust your XTRIM target number. With this, you can execute the XLEN, do your logic to decide on the trimming you want, and then do XTRIM, all Redis-server-side, atomically, so no chance for your stream changing in-between.

Here some eval examples with Jedis.

Update:

A Lua Script like the following should do the trick: atomically evaluate the current number of myStreams entries and then call XTRIM accordingly.

EVAL "local streamLen = redis.call('XLEN', KEYS[1]) \n if streamLen >= tonumber(ARGV[1]) then \n return redis.call('XTRIM', KEYS[1], 'MAXLEN', streamLen - tonumber(ARGV[1])) \n else return redis.error_reply(KEYS[1]..' has less than '..ARGV[1]..' items') end" 1 myStream 90

Let's take a look at the Lua script:

local streamLen = redis.call('XLEN', KEYS[1])
if streamLen >= tonumber(ARGV[1]) then 
    return redis.call('XTRIM', KEYS[1], 'MAXLEN', streamLen - tonumber(ARGV[1]))
else
    return redis.error_reply(KEYS[1]..' has less than '..ARGV[1]..' items')
end

I posted more detail on a separate question on How to delete or remove a given number of entries from a Redis stream?

来源：https://stackoverflow.com/questions/59404054/how-to-xtrim-trim-redis-stream-in-the-safe-way

标签

Redis