How to resolve deserialization error in delayed job?

35
votes

I am trying to use DelayedJob and the job is failing, giving the following error in the database:

{Delayed::DeserializationError
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/serialization/active_record.rb:7:in `yaml_new'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/yaml.rb:133:in `transfer'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/yaml.rb:133:in `node_import'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/yaml.rb:133:in `load'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/yaml.rb:133:in `load'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/backend/base.rb:79:in `payload_object'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/backend/base.rb:87:in `invoke_job_without_newrelic_transaction_trace'
(eval):3:in `invoke_job'
/Library/Ruby/Gems/1.8/gems/newrelic_rpm-2.13.4/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:252:in `perform_action_with_newrelic_trace'
/Library/Ruby/Gems/1.8/gems/newrelic_rpm-2.13.4/lib/new_relic/agent/method_tracer.rb:141:in `trace_execution_scoped'
/Library/Ruby/Gems/1.8/gems/newrelic_rpm-2.13.4/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:247:in `perform_action_with_newrelic_trace'
(eval):2:in `invoke_job'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:120:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:62:in `timeout'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:120:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:119:in `run'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:177:in `reserve_and_run_one_job'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:104:in `work_off'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:103:in `times'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:103:in `work_off'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:78:in `start'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:77:in `start'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:74:in `loop'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/worker.rb:74:in `start'
/Library/Ruby/Gems/1.8/gems/delayed_job-2.1.3/lib/delayed/tasks.rb:9
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
/Library/Ruby/Gems/1.8/gems/rake-0.8.7/bin/rake:31
/usr/bin/rake:19:in `load'
/usr/bin/rake:19

Not sure where to start in diagnosing this. This has never happened before and I have used delayed job before to serialize model objects without any issues. Why this time?

Thanks in advance!

7
ruby-on-railsdelayed-jobdeserialization
I've been having this problem as well. It doesn't seem to happen every time, just sometimes. – Tim Sullivan
Can you catch the error and print out the value it's trying to deserialize? It appears YAML is choking on your data somewhere. – Dominic
I'm not sure where to catch the error, as this seems to be occurring within DJ's code. – Tim Sullivan
Maybe inspect your database and see what the stored value is on the failed job? – Dominic
When the job fails, DJ deletes the record. Is there a way to shut that off? – Tim Sullivan

7 Answers

69
votes

It's not really a deserialization error, it's an ActiveRecord record-not-found error on a simple Model.find(id) query.

If you want to know the details, log them in the delayed_job-2.1.3/lib/delayed/serialization/active_record.rb file, in the rescue statement, just before delayed-job stupidly raises the DeserializationError and throws the useful information away.

12
votes

Michiel is right. Look at your handler field for objects like "!ruby/ActiveRecord:YourClassName"

Then check if objects can be retrieved via the primary key

From the console you can also test this out by doing something like:

# first job in your delayed queue
YAML.load(Delayed::Backend::ActiveRecord::Job.first.handler)
3
votes

I believe this occurs when you run a job against an unsaved or deleted AR object since the deserialization for AR loads the record by id. An exception should probably be thrown if you attempt to delay a method on an unsaved AR object.

1
votes

There's also a documented bug with DJ when the params passed into the handler field in the DB are longer than a standard TEXT column:

https://github.com/collectiveidea/delayed_job/issues/491

If this happens to be your problem, changing the column to a MEDIUMINT should fix the issue.

I did this in a migration like so:

change_column :delayed_jobs, :handler, :text, :limit => 16777215
ActiveRecord::Base.connection.execute("DELETE FROM delayed_jobs WHERE LENGTH(handler) >= 65535")

You can check to see if it's an issue with a simple DB query: 

SELECT * FROM delayed_jobs WHERE LENGTH(handler) >= 65535
1
votes

If anyone wants to make delayed_job just finish the job as a no-op you can monkey patch with this code in the initializer:

https://gist.github.com/spilliton/8494752

0
votes

Today, I also suffered through this error and after doing hectic analysis found:

  1. delayed_job converts methods & parameters to YAML format and stores it into database
  2. It can be found using select * from delayed_jobs;
  3. deserialization error occurs when delayed_job is not able to deserialize it.

Probable causes can be:

  1. args["xyz"] used before calling delayed_job & inside worker using it as args[:xyz]
  2. Sometimes extra arguments passed along with object to delayed_job that time delayed_job fails to build object as it is indifferent access.

I Hope this will help!

0
votes

Sometimes when we upgrade libs delayed jobs still keep old references.

Try to find the id of delayed_job in logs and play to parse its handler to ruby to find the wrong reference

j = DelayedJob.find(XXX)
data = YAML.load_dj(j.handler)
data.to_ruby

I made a pull request to help with this problem.

Meanwhile you can use this lines

# config/initializers/delayed_job.rb

# Monkey patch to use old class references
module Psych

  class << self; attr_accessor :old_class_references end
  @old_class_references = {}

  class ClassLoader
    private

    def find klassname
      klassname = ::Psych.old_class_references[klassname] || klassname
      @cache[klassname] ||= resolve(klassname)
    end
  end

  module Visitors
    class ToRuby < Psych::Visitors::Visitor
      def revive klass, node
        if klass.is_a? String
          klassname = ::Psych.old_class_references[klass] || klass
          klass = Kernel.const_get(klassname) rescue klassname
        end
        s = register(node, klass.allocate)
        init_with(s, revive_hash({}, node), node)
      end
    end
  end
end

# Add all old dependencies (hash keys) pointing to new references (hash values)
Psych.old_class_references = {
  'ActiveRecord::AttributeSet' => 'ActiveModel::AttributeSet'
  # ...
}