Redis — best way to store a large map (dictionary)

Question

What I need to do is to store a one-to-one mapping. The dataset consists of a large number of key-value pairs of the same kind (10M+). For example, one could use a single in

Answer 1

Yes, as Itamar Haber says, you should look at this redis memory optimization guide. But you should also keep in mind a few more things:

1. Prefer HSET to KEYS. Redis consumes a lot of memory just on key space management. In simple (and rough) terms, 1 HSET with 1,000,000 keys consumes up to 10x less memory than 1,000,000 keys with one value each.
2. Keep HSET size less then hash-max-zipmap-entries and valid hash-max-zipmap-value if memory is the main target. Be sure to understand what hash-max-zipmap-entries and hash-max-zipmap-value mean. Also, take some time to read about ziplist.
3. You actually do not want to handle hash-max-zipmap-entries with 10M+ keys; instead, you should break one HSET into multiple slots. For example, you set hash-max-zipmap-entries as 10,000. So to store 10M+ keys you need 1000+ HSET keys with 10,000 each. As a rough rule of thumb: crc32(key) % maxHsets.
4. Read about strings in redis and use a KEY name (in HSET) length based on real memory management for this structure. In simple terms, keeping key length under 7 bytes, you spend 16 bytes per key, but an 8-byte key spends 48 bytes each. Why? Read about simple dynamic strings.

It may be useful to read about:

• Redis Memory Optimization (from sripathikrishnan)
• Comments about internal ziplist structure.
• Storing hundreds of millions of simple key-value pairs in Redis (Instagram)