Issue connecting to elastic search from logstash in input

Question

I want to import data from oracle and would like to pass one of the params of the imported data to elastic search to fetch some other details.

For ex:- If I have an Employee Id which I get from oracle db for say 100 rows , I want to pass all these 100 employee ids to elastic search and get the emp name and salary.

I am able to retrieve the data from oracle now but unable to connect to elastic search. Also I am not sure what will be a better approach to do this.
I am using log stash 2.3.3 and the elastic search log stash filter plugin.

 input {
       jdbc {
        jdbc_connection_string => "jdbc:oracle:thin:@<dbhost>:<port>:<sid>"
        # The user we wish to execute our statement as
        jdbc_user => “user"
        jdbc_password => “pass"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => “<path>"
        # The name of the driver class for oracle
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        # our query
        statement => "SELECT empId, desg from Employee"
    }
    elasticsearch {
      hosts => "https://xx.corp.com:9200"
        index => “empdetails”
    }
   }
   output {
    stdout { codec => json_lines }
   }

I am getting the below error due to elastic search.

A plugin had an unrecoverable error. Will restart this plugin.

Plugin: ["https://xx.corp.com:9200"], index=>"empdetails ", query=>”empId:’1001'", codec=>"UTF-8">, scan=>true, size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>

Error: [401] {:level=>:error}

I could get the above error fixed and now I am able to get the data from elastic search. But can anyone still give me any pointer on how I can pass the data from jdbc and fetch corresponding data from elastic search. I am very new to log stash , hence any info is greatly appreciated. — LookingForSolution
Updated the code as below - elasticsearch { ssl => true hosts => "xx.corp.com:9200" index => "empdetails" query => '{ "query": { "match": { "empId": "1001" } } }' user => "admin" password => "admin" } — LookingForSolution

Val Val · Accepted Answer · 2016-09-07T10:56:58

You need to use the elasticsearch filter and not the elasticsearch input

input {
   jdbc {
    jdbc_connection_string => "jdbc:oracle:thin:@<dbhost>:<port>:<sid>"
    # The user we wish to execute our statement as
    jdbc_user => “user"
    jdbc_password => “pass"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => “<path>"
    # The name of the driver class for oracle
    jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
    # our query
    statement => "SELECT empId, desg from Employee"
   }
}
filter {
  elasticsearch {
    hosts => ["xx.corp.com:9200"]
    query => "empId:%{empId}"
    user => "admin"
    password => "admin"
    sort => "empName:desc"
    fields => {
      "empName" => "empName" 
      "salary" => "salary" 
    }
  }
}
output {
  stdout { codec => json_lines }
}

As a result, each record fetched via JDBC will be enriched by the corresponding data found in ES.

Issue connecting to elastic search from logstash in input

1 Answers