Redis scan count: How to force SCAN to return all keys matching a pattern?

37
votes

I am trying to find out values stored in a list of keys which match a pattern from redis. I tried using SCAN so that later on i can use MGET to get all the values, The problem is: 

SCAN 0 MATCH "foo:bar:*" COUNT 1000

does not return any value whereas

SCAN 0 MATCH "foo:bar:*" COUNT 10000

returns the desired keys. How do i force SCAN to look through all the existing keys? Do I have to look into lua for this?

3
redis
Forcing SCAN to do the entire keyspace in one go is the equivalent to running KEYS. Note that SCAN was introduced exactly for that purpose - not running KEYS. - Itamar Haber
@ItamarHaber does SCAN also block the the keyspace (for one iteration) like KEYS command does for the complete iteration? - DarthSpeedious
Yes - almost all operations are blocking. - Itamar Haber
Is there any recommendation for maximum value of count parameter? - Adam Szecowka
"Since these commands allow for incremental iteration, returning only a small number of elements per call, they can be used in production without the downside of commands like KEYS or SMEMBERS that may block the server for a long time (even several seconds) when called against big collections of keys or elements." - ives

3 Answers

37
votes

With the code below you will scan the 1000 first object from cursor 0

SCAN 0 MATCH "foo:bar:*" COUNT 1000

In result, you will get a new cursor to recall 

SCAN YOUR_NEW_CURSOR MATCH "foo:bar:*" COUNT 1000

To scan 1000 next object. Then when you increase COUNT from 1000 to 10000 and retrieve data you scan more keys then in your case match more keys.

To scan the entire list you need to recall SCAN until the cursor give in response return zero (i.e entire scan)

Use INFO command to get your amount of keys like

db0:keys=YOUR_AMOUNT_OF_KEYS,expires=0,avg_ttl=0

Then call 

SCAN 0 MATCH "foo:bar:*" COUNT YOUR_AMOUNT_OF_KEYS
26
votes

Just going to put this here for anyone interested in how to do it using the python redis library:

import redis
redis_server = redis.StrictRedis(host=settings.redis_ip, port=6379, db=0)
mid_results = []
cur, results = redis_server.scan(0,'foo:bar:*',1000)
mid_results += results

while cur != 0:
    cur, results = redis_server.scan(cur,'foo:bar:*',1000)
    mid_results += results

final_uniq_results = set(mid_results)

It took me a few days to figure this out, but basically each scan will return a tuple.

Examples:

(cursor, results_list)

(5433L, [... keys here ...])
(3244L, [... keys here, maybe ...])
(6543L, [... keys here, duplicates maybe too ...])
(0L, [... last items here ...])
  • Keep scanning cursor until it returns to 0.
  • There is a guarantee it will return to 0.
  • Even if the scan returns an empty results_list between scans.
  • However, as noted by @Josh in the comments, SCAN is not guaranteed to terminate under a race condition where inserts are happening at the same time.

I had a hard time figuring out what the cursor number was and why I would randomly get an empty list, or repeated items, but even though I knew I had just put items in.

After reading:

It made more sense, but still there is some deep programming magic and compromises happening to iterate the sets.

3
votes

Reminder this is a trivial task if you're using the scan_iter method on the redis python library:

from redis import StrictRedis

redis = StrictRedis.from_url(REDIS_URI)

keys = []
for key in redis.scan_iter('foo:bar:*', 1000):
    keys.append(key)

In the end, keys will contain all the keys you would get by applying @khanou 's method.

This is also more efficient than doing shell scripts, since those spawn a new client on each iteration of the loop.