Versioning gearman workers

2
votes

I'm converting a linear processing system to Gearman but I haven't actually deployed Gearman on a production environment.

We have a deployment chain in regards to pushing new versions of source code (Staging, Dark launches, A/B) but since Gearman workers are completely separate from the main application, how do we upgrade them ?

There are a couple of options that come to mind, none of them are perfect:

  1. Versioning the source code as normal and shutdown the environment to upgrade
  2. Versioning workers in their function registration names, ResizeFunction-v1.3
  3. Add version number into workers and attach them to jobs so that if a worker should not process it, it can quit and let another one handle it.

The Ideal scenario would allow me to handle updates of worker code without worrying that jobs from old client code will get picked up by incompatible workers, so smooth version upgrades.

I guess semantic versioning will apply here (v1.0.0 - Major.minor.patch, Major would break API compatibility, minor adds new features, and patch fixes bugs)

1
phpversion-controlversioninggearman
Do you have to have 'legacy' workers running, or is it just about having them upgraded without a hassle? - Wrikken
I would have to still run the workers with old code, so they can process jobs from the 'not-yet-upgraded' servers - Yarek T
Hm, that's not something I already had a solution for :). But I'd version the 'entire gearman server'. meaning a new version gets a new daemon & a new port. Anything using the new version knows the new portnumber. - Wrikken
Interesting solution. I didn't think of that. There is a downside though. Workers that have not changed will have to be started at both locations, and it makes running more than one version difficult (gearmand eats memory for queue and does not give it back) - Yarek T
Either that, or a heck of a lot of extra work/settings in every single version of your gearman-client code. Memory allowance can of course be tweaked. Running multiple servers shouldn't be that difficult, except you probably can't use your OS's distribution out of the box. For Debian & variants ('buntu..), I did like the ease of setting something like this up with supervisord. - Wrikken

1 Answers

0
votes

Unfortunately in your use case the best option would be versioning the function names, that way you can handle the maintenance of legacy functions as well as implementing the new versions of those same functions. What you can do is implement some logging in your worker code and monitor it and once the legacy worker no longer receives jobs for a pre-defined period of time you know you can retire that worker as legacy code. There is no need to version the gearman daemon itself as you are not altering anything in the gearman daemon itself.