I am trying to develop a new logstash plugin in my development Elasticsearch stack environment.
I started with generating the skeleton for my new plug-in using the logstash-plugin generate utility. Once I had this in place I then was able to build my plug-in using the gem build utility which ran without error and created a gem for me. I then ran the logstash-plugin install utility which successfully installed my plug-in.
The entry in my logstash.gem looks like this--
gem "logstash-output-s3-4.3.3", "0.1.0", :path => "./vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.3"
I then took the code from the actual logstash plug-in that I'm hoping to modify and copied it into my new plug-in directory and then created a new logstash.conf file to test with using my new plug-in and started the logstash service. It ran as expected moving data from the servers that were being watched in the appropriate s3 buckets.
logstash.conf output section--
output {
s3{
region => "us-east-1"
bucket => "xxxx"
prefix => "Dev/xxxx/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "xxxx"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
s3{
region => "us-east-1"
bucket => "xxxx"
prefix => "Dev/xxxx/%{+YYYY}/%{+MM}/%{+dd}/"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "xxxx"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
}
My next step was and this is where I am having issues was to try and create a new property for the plug-in called file_prefix--
s3{
region => "us-east-1"
bucket => "xxxx"
prefix => "Dev/xxxx/xxxx/%{+YYYY}/%{+MM}/%{+dd}/"
file_prefix => "test"
server_side_encryption => true
server_side_encryption_algorithm => "aws:kms"
ssekms_key_id => "arn:aws:kms:xxxx"
validate_credentials_on_root_bucket => false
codec => "json_lines"
size_file => 1024000
time_file => 5
rotation_strategy => "size_and_time"
temporary_directory => "../../LogstashS3OutputData/temp/"
canned_acl => "private"
}
I modified the following files in my solution \vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\outputs\s3_spec.rb
added "file_prefix" => file_prefix at line 15 \vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\spec\supports\helpers.rb
added let(:file_prefix) {} at line line 12 added "file_prefix" => file_prefix, at line 19 \vendor\bundle\jruby\2.5.0\gems\logstash-output-s3-4.3.3\lib\logstash\outputs\s3.rb
added config :file_prefix, :validate => :string right after the statement config :prefix, :validate => :string, :default => '' at line 154 added @file_prefix = file_prefix at line 226 Now when I start the logstash service I get this error message Unknown setting 'file_prefix' for s3
I've searched and I've tried everything I can think but I can't get past this hurdle.
TIA, Bill Youngman
