Unable to insert json arrays from Logstash to mongoDB

0
votes

I'm using Logstash 1.5.2
trying to insert datas in MongoDB with logstash-output-mongodb
It works well only when my parsed logs doesn't contain arrays, when the JSON contains array it generates an exception.

Failed to send event to MongoDB
undefined methodbson_type
NoMethodError: undefined method `error_code' for #NoMethodError:0x39794292

@cancelled=false>, :exception=>#<NoMethodError: undefined methodbson_type' for "2015-08-24T12:26:19.664Z":LogStash::Timestamp>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:44:in to_bson'", "org/jruby/RubyHash.java:1341:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:43:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/encodable.rb:57:inencode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:42:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/array.rb:49:into_bson'", "org/jruby/RubyArray.java:1613:in each'", "org/jruby/RubyEnumerable.java:978:ineach_with_index'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/array.rb:46:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/encodable.rb:57:inencode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/array.rb:45:in to_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:46:into_bson'", "org/jruby/RubyHash.java:1341:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:43:into_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/encodable.rb:57:in encode_with_placeholder_and_null'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.1-java/lib/bson/hash.rb:42:into_bson'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/serializers.rb:155:in serialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/message.rb:153:inserialize_fields'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/message.rb:141:inserialize_fields'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/protocol/message.rb:70:in serialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connection.rb:123:inwrite'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connection.rb:122:inwrite'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:66:in dispatch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/loggable.rb:44:inlog'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/loggable.rb:67:in log_debug'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connectable.rb:65:indispatch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/executable.rb:35:in execute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/connection_pool.rb:99:inwith_connection'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/server/context.rb:63:in with_connection'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/executable.rb:34:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:72:in execute_write_command'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/operation/write/insert.rb:62:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/collection.rb:190:in insert_many'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/mongo-2.0.6/lib/mongo/collection.rb:175:ininsert_one'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-mongodb-0.1.4/lib/logstash/outputs/mongodb.rb:56:in receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/outputs/base.rb:88:inhandle'", "(eval):513:in output_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:243:inoutputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:165:in start_outputs'"], :level=>:warn} NoMethodError: undefined methoderror_code' for # receive at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-mongodb-0.1.4/lib/logstash/outputs/mongodb.rb:60 handle at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/outputs/base.rb:88 output_func at (eval):513 outputworker at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipelinestrong text.rb:243 start_outputs at /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/pipeline.rb:165 `

Any idea to solve this issue please ? Thanks !

1
rubyjsonmongodblogstashbson
The timestamp is in every type i store in MongoDB, and it works for most of them, the problem occurs only for 2 types which contain arrays. That's why i think it's not related to the timestamp – Fares
Bug reports for the logstash mongodb plugin and for logstash itself have been created. Seems to be a known issue. github.com/logstash-plugins/logstash-output-mongodb/issues/9 github.com/elastic/logstash/issues/3482 – Kristinn Örn Sigurðsson

1 Answers

3
votes

I have this same error when logstash mongoldb output configuration has 

isodate => true

I have no arrays in my data. Just Nginx access-log data. When I output my data with 

stdout { codec => rubydebug }

@timestamp looks like this

{
"@timestamp" => "2015-09-16T18:49:11.347Z",
}

Main issues is that when I output this to mongodb, the data is inserted as string with quotes escaped like this: 

"@timestamp": "\"2015-09-16T18:14:06.178Z\"",

I'm using logstash 1.5.4 and mongodb 3.0.6, and the error I'm getting is

:exception=>#<NoMethodError: undefined method `bson_type' for "2015-09-16T18:33:52.474Z":LogStash::Timestamp>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bson-3.2.4-java/lib/bson/hash.rb:44:in `to_bson'", [...]

I haven't found any solution for this yet.