How to make variables that are linked by a common correlation ID show on the same row of a Splunk table?

Question

I am logging the time it takes to make external service calls from various clients within my application. These come as various events in Splunk that I search for and then extract the time the call took. For a single execution of the application several clients will be used and there is a unique correlation ID that links them all together.

Imagine an event in Splunk might look like this:

<RandomStuff, client1 time1: 3.2 , Random Stuff, correlation id: 250>

and then another event would be:

<RandomStuff, client2 time2: 2.7 , Random Stuff, correlation id: 250>

So, the client name (client2 time2 for example) will be different but the correlation ID is the same for a particular execution.

I do a Splunk search for time1, time2, timeN and then extract the time and correlation ID:

<my search> 
| rex "time1: (?<t1>.*)" 
| rex "time2: (?<t2>.*)" 
| rex "time3: (?<t3>.*)" 
| rex "correlation_id: (?<corId>.*)

This grabs all the relevant events and extracts the times (if they exist -- because one Splunk event will only have one of the three clients in it, there are three different events for time1, time2, and time3).

I then add | table t1 t2 t3 corId to the end of the search and I get a table that looks like this:

What I want, however, is something that looks like this:

Anyone have any ideas how to join t1, t2, and t3 together in the same row with the matching correlation ID?

I'm not very good with Splunk yet, so I think what might be happening is that when I do rex time1 but it is the event with rex time2 it is setting time1 as empty/null for that for that particular event. So it's actually displaying what I am telling it to. But since these are different event logs that I am trying to combine into one row with the linking correlation ID I'm not sure how to do that.

RichG RichG · Accepted Answer · 2019-07-12T00:38:00

I think what might be happening is that when I do rex time1 but it is the event with rex time2 it is setting time1 as empty/null for that for that particular event. So it's actually displaying what I am telling it to.

This is correct. We can use the stats command to combine the rows.

<my search> 
| rex "time1: (?<t1>.*)" 
| rex "time2: (?<t2>.*)" 
| rex "time3: (?<t3>.*)" 
| rex "correlation_id: (?<corId>.*)"
| stats values(t1) as t1, values(t2) as t2, values(t3) as t3 by corId
| table t1 t2 t3 corId

How to make variables that are linked by a common correlation ID show on the same row of a Splunk table?

1 Answers