2
votes

I need to get data from a PostgreSQL DB and index it into Elasticsearch. https://www.elastic.co/blog/logstash-jdbc-input-plugin

When I run /opt/logstash-2.3.3/bin/logstash -v -f es_table.logstash.conf
I receive the following error:

Pipeline aborted due to error 
{:exception=>#<LogStash::ConfigurationError: org.postgres.Driver not loaded. 
Are you sure you've included the correct jdbc driver in :jdbc_driver_library?>, :backtrace=>["/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.0.2/lib/logstash/plugin_mixins/jdbc.rb:156:in `prepare_jdbc_connection'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.0.2/lib/logstash/plugin_mixins/jdbc.rb:148:in `prepare_jdbc_connection'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-input-jdbc-3.0.2/lib/logstash/inputs/jdbc.rb:167:in `register'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:330:in `start_inputs'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:329:in `start_inputs'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:180:in `start_workers'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash-2.3.3/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.3-java/lib/logstash/agent.rb:473:in `start_pipeline'"], :level=>:error}

Here is a piece of my Logstash configuration:

input {
   jdbc {
      jdbc_user => 'user'
      jdbc_driver_class => 'org.postgresql.Driver'
      jdbc_connection_string => 'jdbc:postgresql://1.1.1.1:5432/db'
      lowercase_column_names => false
      clean_run => false
      jdbc_driver_library => '/usr/share/java/postgresql-jdbc4.jar'
      jdbc_password => 'pass'
      jdbc_validate_connection => true
      jdbc_page_size => 1000
      jdbc_paging_enabled => true
      statement => 'SELECT * FROM "table"'
      type => 'table'
   }
...

The jdbc4 driver exists. I tried jdbc3 too without success.

ls /usr/share/java | grep postgresql-jdbc
postgresql-jdbc3-9.2.jar
postgresql-jdbc3.jar
postgresql-jdbc4-9.2.jar
postgresql-jdbc4.jar

The Driver class is inside:

jar tf /usr/share/java/postgresql-jdbc4.jar | grep -i driver
org/postgresql/Driver$1.class
org/postgresql/Driver$ConnectThread.class
org/postgresql/Driver.class
org/postgresql/util/PSQLDriverVersion.class
META-INF/services/java.sql.Driver

The port 5432 is open:

telnet 192.168.109.108 5432
Trying 192.168.109.108...
Connected to 192.168.109.108.
Escape character is '^]'.

Authentication to the DB works.

2

2 Answers

4
votes

The problem was that I made a mistake in the driver name.

I wrote jdbc_driver_class => 'org.postgres.Driver'

And the correct name is jdbc_driver_class => 'org.postgresql.Driver'

0
votes

I resolved this issue by following the workaround suggested in this issue

Reason:

This is a known problem that we have with the modules changes in JDK 9 (Jigsaw). The classloaders have seen some changes and a work around we added before to some driver loading is now failing. The jdbc input has the same failing in JDK 11 (9+). We are working on a fix.

Workaround that worked for me:

An "extreme" work around is to copy the driver file to /logstash-core/lib/jars/ directory. These jar get added to the correct JDK classpath as logstash is started via java.