Extract XML data with Logstash and Xpath

Question

I have this sample XML data:

<root>
    <actors>
        <actor id="1" name="Christian Bale"></actor>
        <actor id="2" name="Liam Neeson"></actor>
        <actor id="3" name="Michael Caine"></actor>
    </actors>   
</root>

and this is my logstash config to extract the data

input
{
    file
        {
            path => "D:/data.xml"
            start_position => "beginning"
            sincedb_path => "NUL"
            exclude => "*.gz"
            type => "xml"
            codec => multiline {
                    pattern => "<?xml " 
                    negate => "true"
                    what => "previous"
                }
        }
}

filter {

    xml{
        source => "message"
        store_xml => true target => "id"
        target => "root"
        xpath => [
            "/root/actors/actor/text()", "actor"            
        ]
    }    
}

output{

elasticsearch{
        hosts => ["http://localhost:9200/"]
        index => "actor"
    }

    stdout
    {
        codec => rubydebug
    }
}

When I run this config, what I get is below as print screen

But what I need is actor index as defined with column created based on actor which is id and name.

This is my log when I run the config:

"Using bundled JDK: "" OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/C:/Users/CHEEWE~1.NGA/AppData/Local/Temp/jruby-11656/jruby15503754749915308062jopenssl.jar) to field java.security.MessageDigest.provider WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Sending Logstash logs to D:/logstash/logs which is now configured via log4j2.properties [2020-12-07T17:54:43,527][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [mswin32-x86_64]"} [2020-12-07T17:54:43,843][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-12-07T17:54:45,899][INFO ][org.reflections.Reflections] Reflections took 43 ms to scan 1 urls, producing 23 keys and 47 values [2020-12-07T17:54:47,229][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [2020-12-07T17:54:47,482][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"} [2020-12-07T17:54:47,544][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7} [2020-12-07T17:54:47,551][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7} [2020-12-07T17:54:47,618][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]} [2020-12-07T17:54:47,689][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled} [2020-12-07T17:54:47,786][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [2020-12-07T17:54:47,846][INFO ][logstash.outputs.elasticsearch][main] Creating rollover alias <logstash-{now/d}-000001> [2020-12-07T17:54:47,964][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["D:/logstash/bin/logstash-simple.conf"], :thread=>"#<Thread:0x78c4a90f run>"} [2020-12-07T17:54:49,256][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.29} [2020-12-07T17:54:49,347][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"} The stdin plugin is now waiting for input: [2020-12-07T17:54:49,446][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2020-12-07T17:54:49,757][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Badger Badger · Accepted Answer · 2020-12-07T13:56:47

If both elasticsearch and logstash are running recent versions then ILM is enabled by default. In that case the value of the index option is ignored and the default index name is logstash-{now/d}-00001. If you want to set the index name using the index option then set the ilm_enabled option to false.

Extract XML data with Logstash and Xpath

1 Answers