Cast or convert mongodb _id object to string in Logstash pipeline

1
votes

I am working on creating a pipeline to get data from MongoDB to ElasticSearch using Logstash. I am using dbschema mongodb jdbc drivers. I am able to connect to database using driver but I am facing issue with _id . As in MongoDB its of type object So I am getting issue with converter. Here is error I am getting.

Exception when executing JDBC query {:exception=>#<Sequel::DatabaseError: Java::OrgLogstash::MissingConverterException: Missing Converter handling for full class name=org.bson.types.ObjectId, simple name=ObjectId>}

My pipeline is as below :

input{
jdbc{
jdbc_driver_library => "C:/logstash-6.1.0/logstash-6.1.0/bin/driver/mongo/dbschema/mongojdbc1.2.jar"
jdbc_driver_class => "Java::com.dbschema.MongoJdbcDriver"
jdbc_connection_string => "jdbc:mongodb://abc.com:27017/test"
jdbc_user => ""
statement => "db.getCollection('Employee').find({})"
codec => json
}
}

output {
elasticsearch {
hosts => 'http://localhost:9200'
index => 'mongodbschema'
codec => json   
}
stdout { codec => rubydebug }
}

Is there any way I can convert/cast or do something in filter to change datatype of _id from object to string

3
mongodbelasticsearchlogstash
Have you found solution ? - Shantaram Tupe

3 Answers

1
votes

try 

statement => "db.getCollection('Employee').find({ },{'_id': false})"
0
votes

Try this code in filter

filter {
mutate {
remove_field => [ "_id" ]
}}
0
votes

Since in elasticsearch _id is a reserved field (you can pass _id to index API in PUT operation to update a document), you must apply a filter that mutates the mongodb _id, for instance:

filter {
  mutate {
    rename => [ "_id", "mongo_id" ]
  }
}