Parse and forward jenkins build logs to logstash/ elastricsearch

Question

Forward Jenkins build logs to logstash/elasticsearch using filebeat

Currently I am running filebeat as service on jenkins server and able to pass the log to logstash/ elasticsearch.

filebeat:
  prospectors:
    - paths:
        - "/var/jenkins_home/jobs/*/builds/*/log"

output:
  logstash:
    hosts: ["172.22.0.4:5044"]
    index: "jenkins"


Jenkins log file:
Started by user ha:////4Pt==anonymous
Building in workspace /var/jenkins_home/jobs/aaa/workspace
[workspace] $ /bin/sh -xe /tmp/jenkins2734923241755751652.sh
+ maven install
/tmp/jenkins2734923241755751652.sh: 2: 
/tmp/jenkins2734923241755751652.sh: maven: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE

How do I send entire build log as single document. Currently it taking each log line as one row in elastricsearch.
How do I add"Finished" as field in elastricsearch document.

justkind justkind · Accepted Answer · 2019-08-27T18:24:28

Not sure if you want to do this, but maybe look into implementing a multiline pattern: https://www.elastic.co/guide/en/beats/filebeat/master/multiline-examples.html

It will combine log lines based on how you configure it, mostly used for exceptions and stack traces but you can probably set one up for your Jenkins build logs.

From your build log example, with a beginning line "Started by", you can use the following pattern and a value of 'true' for the multiline.negate: ^Started by

Here's an example on the go playground, sounds like you want the multiline pattern to "match" everything but the first line so that it combines all lines into the top one which is how I have it setup: https://play.golang.org/p/W1on5dxNJTq

For the "Finished" field it can be done easily in Logstash using the mutate plugin which has an add_field option: https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-add_field

Note that the add_field option is a common option and can be done in multiple places including input plugins such as the beat input, a grok filter, etc

Parse and forward jenkins build logs to logstash/ elastricsearch

1 Answers