Redis how to reduce lua copy-paste

Question

I'm writing some logic for redis inside lua and almost each of my scripts have something common, it would be really handy to move out this to the shared function but

redis can't use lua's require statement
officially you can't call other redis function(see: https://stackoverflow.com/a/22599862/1812225)

For example I have this snippet literally everywhere

local prefix = "/" .. type
if typeId then
    prefix = prefix .. "(" .. typeId .. ")"
end

I'm thinking about some post-processing before feeding scripts to redis but this seems like an over-kill...

What is the best practice to solve/reduce this problem?

Updated:

local registryKey = "/counters/set-" .. type
local updatedKey = "/counters/updated/set-" .. type
if typeId then
    redis.call("SAdd", updatedKey, name .. ":" .. typeId)
    redis.call("SAdd", registryKey, name .. ":" .. typeId)
else
    redis.call("SAdd", updatedKey, name)
    redis.call("SAdd", registryKey, name)
end

is another code sample and it can't be trivially moved to client-side as it invokes redis commands, and works as a part of transaction

Thanks!

Assume for a moment that it is possible to have this common snippet "shared" - would you really want to use it? Put differently, here's a mundane task that needs to be done a lot of times... Pushing it into a Lua script means you'll be using resources from your Redis database to perform something that could be done elsewhere. I'm not saying it is wrong, but as a rule of thumb I try to keep as this type of logic outside the database whenever possible. — Itamar Haber
Regarding this one snippet you are probably right @itamar-haber. But I have another case when I need to do thing with conditional logic which triggers redis commands over and over again... It can't be easily moved to client side — let4be
On the other hand I probably should just create another script with repetative steps and use them both inside a MULTI, i.e. MULTI EVALSHA EVALSHA ... This just seems non ideal from the point of bandwidth but shouldn't be a big issue — let4be
Each case is different :) basically the options are a) copy-paste b) use the undocumented f_<sha1> approach or c) "hack" around Redis' protection of global. I feel that the safest is (a) — Itamar Haber
@ItamarHaber two things. a) what is the undocumented f_<sha1> approach? and b) Why the trick used by redis-lua-debugger be used here? (github.com/RedisLabs/redis-lua-debugger) (of course, taking into account the limitation regarding slaves not running the script by themselves). — Ignacio

Itamar Haber Itamar Haber · Accepted Answer · 2015-08-06T18:06:05

"Hack" #1

After you SCRIPT LOAD something, you get back a sha1 hash that you can use with EVALSHA. The same sha1 value can be used to call that script from inside another script - simply call the function f_<sha1>. That said, there are some differences in how you pass the KEYS/ARGV structures when used that way.

Note that this is undocumented behavior, which means the behavior could change in a future version of Redis.

Credit for teaching me this goes to Dr. Josiah Carlson who, in turn, credits someone else (IIRC Fritzy). For more information check out his lua-call Python wrapper: https://github.com/josiahcarlson/lua-call

"Hack" #2

Redis sandboxes Lua and puts several restrictions on it in order to maintain sanity. You could go around some of these, e.g. access _G and define your utility function there so it will be available to all scripts (like I did with https://github.com/redislabs/redis-lua-debugger).

However, this is also pretty risky - besides potential replication issues, this usage is untested and could therefore lead to undefined behavior (I managed to crash quite a few instances with my little script ;)).

P.S.

Both hacks require additional administrative work to ensure that these "global" scripts are actually loaded before any other script calls them.

Redis how to reduce lua copy-paste

1 Answers