Connect Logstash with RethinkDB on IBM BlueMix

Question

I try and connect Logstash with RethinkDB on IBM BlueMix. I have provisioned the RethinkDB-Service from BlueMix (with is from Compose I believe) and also provisioned a Virtual Machine (CentOS). Then I installed Logstash (using the yum package) and the plugin from here (https://github.com/wayann/logstash-input-rethinker). So far so good. Now I struggle to get a connection from Logstash to RethinkDB. I have sent host and port and auth_key in a similar config file (https://github.com/wayann/logstash-input-rethinker/blob/master/rethinker.conf) and started Logstash with bin/logstash -f rethinker.conf However, Logstash is unable to connect to RethinkDB. Has anyone experience using Logstash with RethinkDB on Bluemix? I have sent host and port according to the credentials available on the BlueMix dashboard (same as the admin UI) but I'm unsure what to use for auth_key. admin-password didn't work. BlueMix runs RethinkDB version 2.3.5

Any hint highly apprechiated

Edit1, Error message:

05:46:12.049 [[main]<rethinker] ERROR logstash.pipeline - A plugin 
had an unrecoverable error. Will restart this plugin.
 Plugin: <LogStash::Inputs::Rethinker host=>"sl-eu-lon-2-
portal.3.dblayer.com", port=>15216, auth_key=>"********", 
watch_dbs=>["test", "MyDB"], watch_tables=>["MyTable"], 
backfill=>"true", id=>"f1c07e0332787a22600c0835d2aa6ad61ca9b22b-1", 
enable_metric=>true, codec=><LogStash::Codecs::JSONLines 
id=>"json_lines_e24c0744-3983-4001-9629-d93e266c5ffb", 
enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, 
squash=>true, user=>"admin">
 Error: Connection closed by server.

Edit2, config file:

input { 
  rethinker {
    host => 'sl-eu-lon-2-portal.3.dblayer.com'
    port => 15216
    auth_key => '*****'
    watch_dbs => ['test','MyDB']
    watch_tables => ['MyTable']
    backfill => true
  }
}

output { 
  stdout { 
    codec => json_lines 
  } 
}

Edit3:

Ok, it was my error. I need to use the 'username', 'password' and 'ca_cert' parameters and not 'auth_key'. Using this configuration connects to the rethinkdb instance:

input { 
  rethinker {
    host => 'sl-eu-lon-2-portal.3.dblayer.com'
    port => 15216
    watch_dbs => ['MyDB']
    watch_tables => ['MyTable']
    backfill => true
    user => 'admin'
    password => '******'
    ca_certs => '<<cert-string>>'
  }
}

output { 
  stdout { 
    codec => json_lines 
  } 
}

Now, however, I get the following error:

03:54:37.252 [[main]<rethinker] ERROR logstash.pipeline - A plugin 
had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Rethinker host=>"sl-eu-lon-2-
portal.3.dblayer.com", port=>15216, watch_dbs=>["MyDB"], 
watch_tables=>["MyTable"], backfill=>"true", user=>"admin", 
password=>"*****", ca_certs=>"**cert-string**", 
id=>"46aa34ee0917060057d4e9a0c657ee327df730ee-1", 
enable_metric=>true, codec=><LogStash::Codecs::JSONLines 
id=>"json_lines_c43cdbc3-1646-4e96-aadc-834e727949a6", 
enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, 
squash=>true>
  Error: No message available

How would I resolved this Error: No message available error? Logstash version is 5.5.2

Yes, I have added it as Edit1. Forgot about that earlier :blush: — Thor_Bux
OK so, your error is connection closed by server. This means that you are actually connecting to a third party which is responding but rejecting your connection. It looks like an high level error, such as an authentication or authorization error. Do you have any log access on the rethinkdb side? I'd expect to get the cause of the problem there. — whites11

Thor_Bux Thor_Bux · Accepted Answer · 2017-08-27T23:58:50

I finally figured it out, thanks a lot for your help @whites11.

I needed to set the correct settings in the configuration file. That includes user, password and ca-cert. For ca-cert it is important that this is the path to the certificate file and not the certificate content as a string.

My setting looks like this now:

input { 
  rethinker {
    host => 'sl-eu-lon-2-portal.3.dblayer.com'
    port => 15216
    watch_dbs => ['MyDB']
    watch_tables => ['MyTable']
    backfill => true
    user => 'admin'
    password => '******'
    ca_certs => '<<path-to-cert-file>>'
  }
}

output { 
  stdout { 
    codec => json_lines 
  } 
}

Connect Logstash with RethinkDB on IBM BlueMix

Edit1, Error message:

Edit2, config file:

Edit3:

1 Answers