What's the simplest way to get a dump of all memcached keys into a file?

Question

This is from just a single memcached server with around 20M keys (no expiry) and around 2G of data.

What\'s the easiest way to get a dump of all the key/value pairs

Answer 1

memccat

Here is the script which I'm using to dump all the objects into corresponding files:

while read -r key; do
    [ -f "$key" ] || echo "get $key" | nc localhost 11211 > "$key.dump";
done < <(memcdump --server localhost)

It uses memcdump command which should be part of memcached utils.

For compressed objects, see: How to dump a compressed object for given key from Memcache?

memcdump

To dump a list of keys from a server, use memcdump/memdump tool, e.g.

memcdump --servers=localhost | tee my_keys.lst

To print the value of one item, use netcat:

echo "get 13456_-cache-some_object" | nc localhost 11211

To dump all objects into the screen via memcdump/memdump and netcat:

memcdump --servers=localhost | xargs -L1 -I% sh -c 'echo "get %" | nc localhost 11211'

memcached-tool

In the recent version of memcached there is also memcached-tool command, e.g.

memcached-tool localhost:11211 dump | less # dumps keys and values

Answer 2

I used this bash script

#!/bin/sh
MESSAGE=`memdump --servers="127.0.0.1"`
while read -r line; do
    echo $line
    VALUE=`echo "get $line" | nc 127.0.0.1 11211`
    echo $VALUE
done <<< "$MESSAGE"

just replace IP / port if necessary

Answer 3

Bash

Using Bash and save into the file:

exec {memcache}<>/dev/tcp/localhost/11211
printf "stats items\nquit\n" >&${memcache}
cat <&${memcache} > myfile.txt

Related: Writing a Redis client in pure bash (it's Redis, but very similar approach)

Answer 4

There is a hardcoded limit of 2MB for the dump of a slab. Unless you rewrite the do_item_cachedump, you will not be able to get all keys out.

Answer 5

disclaimer: I don't know what I"m doing, just sounded like an interesting problem.

Did you see this article? "How to Dump Keys from Memcache" by Lars Windolf.

From the article:

Memcache itself provide the means to peak into the data. The protocol provides commands to peak into the data that is organized by slabs (categories of data of a given size range. There are some significant limitations though:

• You can only dump keys per slab class (keys with roughly the same content size)
• You can only dump one page per slab class (1MB of data)
• This is an unofficial feature that might be removed anytime.

Effectively, it requires some knowledge of how memcache stores data in memory (which I don't). You need to find each 'slab', then you can dump the keys for that slab, and then ultimately, the values for those keys.

There is a tools section in the article which uses various languages to dump at least the keys, but only the perl script dumps both keys and values.