Redis: Memory Optimization

Question

I have around 256 keys. Against each key I have to store a large number of non-repitative integers. Following are the top 7 keys with number of total values (entries) against each key. Each value is a unique integer with large value.

Key     No. of integers (values) in the list
Key 1   3394967
Key 2   3385081
Key 3   2172866
Key 4   2171779
Key 5   1776702
Key 6   1772936
Key 7   1748858

By default Redis consumes lot of memory in storing this data. I read that changing following parameters can result in memory usage reduction highly.

list-max-zipmap-entries 512 
list-max-zipmap-value 64 

Can anyone please explain me these above configuration commands (are 512 and 64 bytes?) and what changes I can make in the above configuration settings for my case to achieve memory usage reduction?

What should be kept in mind while selecting the values for entries and value in above command?

Answer 1

list-max-mipmap-entries 512: list-max-zipmap-value 64

If the number of entries in a List exceeds 512, or if the size of any given element in the list > 64 bytes, Redis will switch to a less-efficient in-memory storage structure. More specifically, below those thresholds it will use a ziplist, and above it will use a linked list.

So in your case, you would need to use an entries value of > 1748858 to see any change (and then only in keys 8-end). Also note that for Redis to re-encode them to the smaller object size you would also need to make the change in the config and restart Redis as it doesn't re-encode down automatically.

To verify a given key is using a ziplist vs. linked list, use the OBJECTcommand.

For more details, see Redis Memory Optimization

Answer 2

IMO you can't achieve redis' memory optimization. In your case the entries in each list/set is around 3 million. In order to achieve memory optimization if you give the value of list-max-zipmap-entries as 3 million.

Redis doc says,

This operation is very fast for small values, but if you change the setting in order to use specially encoded values for much larger aggregate types the suggestion is to run some benchmark and test to check the conversion time.

As per this encoding and decoding will take more time/CPU for that huge number. So it is better to run a benchmark test and then decide.

One alternative suggestion, if you only look up this sets to see whether a key is available or not. then you can change the Structure to a bucket kind of a thing.

For example a value 123456 set to key1 can be stored like this

Sadd key1:bucket:123 456

123 = 123456/1000 456 = 123456%1000

Note this won't work if you want to retrieve all the values for key1. In that case you would be looping through 1000 of sets. similarly for total size of key1 you have to loop through 1000 keys.

But the memory will be reduced to about 10 times.