Why won't Splunk recognize these fields?

Question

I am new to Splunk, forgive me if I need to provide more info.

I am generating logs that track metrics of a few websites with the end goal of sending me alerts when a value changes.

I am forwarding the logs to a Splunk Indexer. My log is in the following format:

fetchTime: 2018-12-02T18:33:56.621Z
fooVersion: 3.2.1
requestedUrl: https://cats.com/
finalUrl: https://cats.com/
accessibilityScore: 0.70
fetchTime: 2018-12-02T18:34:50.345Z
fooVersion: 3.2.1
requestedUrl: https://example.com/
finalUrl: https://example.com/
accessibilityScore: 0.90
fetchTime: 2018-12-03T18:35:50.750Z
fooVersion: 3.2.1
requestedUrl: https://cats.com/
finalUrl: https://cats.com/
accessibilityScore: 0.72
fetchTime: 2018-12-03T18:36:06.868Z
FooVersion: 3.2.1
requestedUrl: https://example.com/
finalUrl: https://example.com/
accessibilityScore: 0.88

The events show up in Splunk as I hoped:

But I expected that Splunk could easily identify them as "Interesting Fields".

Is there something wrong with my log format? Should I make a custom Field extractor since it does not identify them? Or is something not configured correctly in Splunk?

Thanks in advance.

RichG RichG · Accepted Answer · 2018-12-03T23:48:25

They're not listed as "Interesting Fields" because no fields were extracted. Splunk expects to find fields in key=value format. If the data is not in that format (like yours) then you need settings in the props.conf file to tell Splunk how to interpret your data. Start with the following:

[mysourcetype]
TIME_PREFIX = fetchTime: 
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^fetchTime
TRANSFORM-fields = mysourcetypefields

In transforms.conf add:

[mysourcetypefields]
REGEX = ([^:]+):\s(\S+)
FORMAT = $1::$2

Why won't Splunk recognize these fields?

1 Answers