How to properly handle atomic batch processing?

Question

In my app, there's a batching mechanism. It (in theory) accumulates items in a redis list, then processes them all at once and cleans up (if necessary). Here's the code:

def with_private_key redis, name
  return unless redis.exists name

  # atomically rename to a random name, so that the batch isn't appended to.
  tmpname = "temp:#{SecureRandom.hex}"
  redis.rename name, tmpname

  # get batch elements
  batch = redis.lrange(tmpname, 0, -1).map{|b| JSON.parse(b) }

  begin
    # actually do something with the batch
    yield batch
  rescue => ex
    # revert and give up
    redis.rename tmpname, name
    raise
  else
    # if there was no exception - drop the temp key, it's not needed anymore
    redis.del tmpname
  end
end

Here's a small problem with this implementation: sometimes temp keys don't get deleted and remain in the database. When I was composing this question, I realized that reverse-renaming might fail, because new key has been created, with new batch.

So, how should I improve this algorithm?

Modifying batch in place doesn't work. LRANGE + LTRIM pair is not atomic.
Periodically check for temp keys and try to re-process them (naming formula has to be changed to include app_id but this is probably not important here).
Anything else? Maybe some Lua tricks (haven't work with Lua yet)?

EDIT:

I see that my theory about renaming is invalid. RENAME command:

Renames key to newkey. It returns an error when the source and destination names are the same, or when key does not exist. If newkey already exists it is overwritten.

akonsu akonsu · Accepted Answer · 2012-12-27T05:31:11

I would try

batch, _ = redis.multi do |r|
  r.lrange name, 0, -1
  r.ltrim name, 0, -1
end

There is a possibility of losing the batch if the client crashes after fetching the batch and trimming the list.

How to properly handle atomic batch processing?

EDIT:

1 Answers