Lua execution in Redis - Atomicity & Scale?

2
votes

I have some keys in Redis <K,V> where V is a comma separated string of sub strings V: <v1,v2,v3,...,vn>.

Until now I was simply setting keys by using a transaction in Jedis (Redis API in Java). But a new requirement needs me to perform some operations on the existing keys. So I have two options:

  1. Retrieve Keys, Perform Operations, Set keys
  2. Code the operations in Lua and pass the file along with keys as arguments.

Option 2 seems faster as it will save a round trip but I have the following considerations:

  1. What if Lua execution fails in between at Redis, say after n keys. (May be an issue with script or Redis). Does Redis ensure consistency? In option 1, I could abort writing and prevent inconsistent state.
  2. Lua script as I understand takes keys as arguments. However I need to operate on 10 million keys. Is Lua fine with such a huge argument list?

My Solution

Since as said by the answer below none of them are a feasible choice. I wrote my own Redis command in C and compiled the source. So now I will call

redis-cli> MyCommand K V

For each <K,V> in pipeline. I followed the tutorial here - Hacking Redis. However it might not be adequate and require one to browse and understand some code by yourself!

1
luaredis

1 Answers

3
votes

wrt #1 - a Lua script is atomic but it does not provide this type of consistency assurance. In case a script fails, its updates will not be rolled back by Redis automatically. There is a proposal to improve this behavior but it is yet to be accepted into Redis (see Josiah Carlson's blog here: http://www.dr-josiah.com/2015/07/transactions-in-redis.html)

wrt #2 - I don't know if such a huge list will go smoothly or not but if you need it you're probably using it wrong ;)