Goal is: for each line in the log, there should be a document in elastic containing the 'message' (text after time stamp). Each document should also contain fields for the the project name, plan name, and build #. <--this is where I'm getting stuck
example log structure in the beginning (atlassian bamboo build logs):
simple 01-Jan-2016 14:26:01 Build TestProj - Framework Code - Build #25 (TST-FC-25) started building on agent .NET Core 2
simple 01-Jan-2016 14:26:01 .NET-related builds, tests and publishing.
I have a Grok to get and create the fields I want - build name, build number, and project name (and have them as fields in Kibana):
%{NOTSPACE:log_entrytype}%{SPACE}(?<timestamp>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])-\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b-(?>\d\d){1,2}\s*(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))%{SPACE}Build%{SPACE}%{DATA:BamProjName}%{SPACE}-%{SPACE}%{DATA:BamBuildName}%{SPACE}-%{SPACE}Build%{SPACE}#%{NUMBER:BamBuildNum}
However I need these fields available in every record/entry in Kibana. With this other Grok, I can extract the other lines of the log into a log_message field:
grok { [
"message", "%{NOTSPACE:log_entrytype}%{SPACE}(?<timestamp>(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])-\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b-(?>\d\d){1,2}\s*(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))%{SPACE}%{GREEDYDATA:log_message}"
]
}
So do I need to somehow combine these two pattern matches into one, using the 'optional': ()? syntax as described here?: link
Is my end goal achievable with logstash and the Grok plugin alone? Can I handle this with some type of variable construct within logstash? add_field?
***NOTE: using filebeat for shipping logs, and elastic does not recommend the multiline codec , so I'm curious what my other options are
