Redis scan count: How to force SCAN to return all keys matching a pattern?

Question

I am trying to find out values stored in a list of keys which match a pattern from redis. I tried using SCAN so that later on i can use MGET to get

Answer 1

Just going to put this here for anyone interested in how to do it using the python redis library:

import redis
redis_server = redis.StrictRedis(host=settings.redis_ip, port=6379, db=0)
mid_results = []
cur, results = redis_server.scan(0,'foo:bar:*',1000)
mid_results += results

while cur != 0:
    cur, results = redis_server.scan(cur,'foo:bar:*',1000)
    mid_results += results

final_uniq_results = set(mid_results)

It took me a few days to figure this out, but basically each scan will return a tuple.

Examples:

(cursor, results_list)

(5433L, [... keys here ...])
(3244L, [... keys here, maybe ...])
(6543L, [... keys here, duplicates maybe too ...])
(0L, [... last items here ...])

• Keep scanning cursor until it returns to 0.
• There is a guarantee it will return to 0.
• Even if the scan returns an empty results_list between scans.
• However, as noted by @Josh in the comments, SCAN is not guaranteed to terminate under a race condition where inserts are happening at the same time.

I had a hard time figuring out what the cursor number was and why I would randomly get an empty list, or repeated items, but even though I knew I had just put items in.

After reading:

• https://github.com/antirez/redis/blob/unstable/src/dict.c#L772-L855

It made more sense, but still there is some deep programming magic and compromises happening to iterate the sets.