Incorrect document_id for Logstash elastic search output

Question

I'm using Logstash to read json messages from Solace queue and write it to elastic Search.I'm using the doc_as_upsert => true along with the document_id parameters in the output.This is how my logstash configuration looks like

logstash.conf

input

    {
        jms {
            include_header => false
            include_properties => false
            include_body => true
            use_jms_timestamp => false
            destination => 'SpringBatchTestQueue'
            pub_sub => false
            jndi_name => '/JMS/CF/MDM'
            jndi_context => {
             'java.naming.factory.initial' => 'com.solacesystems.jndi.SolJNDIInitialContextFactory'
             'java.naming.security.principal' => 'EDM_Test_User@NovartisDevVPN'
             'java.naming.provider.url' => 'tcp://localhost:55555'
             'java.naming.security.credentials' => 'EDM_Test_User'
            }
            require_jars=> ['/app/elasticsearch/jms/commons-lang-2.6.jar',
                            '/app/elasticsearch/jms/sol-jms-10.10.0.jar',
                            '/app/elasticsearch/jms/geronimo-jms_1.1_spec-1.1.1.jar']
        }
    }
    output
    {
        elasticsearch
        {
            hosts => ["https://glchbs-sd220240.eu.novartis.net:9200/"]
            index => "test-%{+YYYY.MM.dd}"
            document_id => "%{customerId}"
            doc_as_upsert => true
            ssl => true
            ssl_certificate_verification => true
            cacert => "/app/elasticsearch/config/ssl/Novartis_Silver_Three_Chain.pem"
        }
    }

Json Message:

{
    "customerId": "N-CA-Z9II2YJ1YJ",
    "name": "Alan Birch",
    "customerRecordType": "Health Care Professional",
    "country": "CA",
    "language": "EN",
    "privacyLawStatus": false,
    "salutation": "Mr.",
    "firstName": "Alan",
    "lastName": "Birch",
    "customerType": "Non Prescriber",
    "hcpType": "Pharmacist Assistant",
    "isMedicalExpert": false,
    "customerAddresses": [
        {
            "addressType": "Primary Address",
            "addressLine1": "4001 Leslie Street"
        },
        {
            "addressType": "Other",
            "addressLine1": "3004 Center St"
        }
    ],
    "meansOfContact": [
        {
            "type": "Email1",
            "value": "[email protected]",
            "status": "Active"
        },
        {
            "type": "Email2",
            "value": "[email protected]",
            "status": "Active"
        }
    ],
    "specialities": [
        {
            "specialtyType": "Primary Specialty",
            "specialty": "Pharmacy Technician",
            "status": "Active"
        }
    ]
}

As you can see, I'm trying to use the customerId field of the JSON message as the document id for elasticsearch. But this is what a document inserted into Elasticsearch looks like:

As you can see document_id field should be mapped to customerId field but this is not case..Document is being inserted as %{customerId}

How to fix this?Appreciate your help

Badger Badger · Accepted Answer · 2021-04-21T16:45:07

That is telling you that the [customerId] field does not exist on that event. If the [message] field is JSON then you should add a json filter to parse it. That will create the [customerId] field, which you can then use as the document_id.

json { source => "message" }

Incorrect document_id for Logstash elastic search output

1 Answers