Splunk map without losing fields

Question

I've got 2 logs that I need to join together in order to get all of the information I need. summary.log has most of the fields I need (there's a good number of them) along with an indicator of whether or not this event applies to me. location.log has exactly 1 field I need along with a unique identifier in order to tie the logs together.

At first I thought to use a join command as the name implies but the resulting fields of the first search can't be used in a subsearch (which join uses). Then I discovered the map command which allows exactly that, however the map has a side affect of deleting all fields that didn't come from the map just now. Later I thought maybe I could use collect or outputlookup to store the fields temporarily. But I need multiple people to be able to run this query without race conditions so a lookup is out of the question. collect testmode=true might work (or it might not) but only existing indexes can be used and I'm not going to create an empty index and make sure it stays empty. After a bit of online searching I couldn't find any other commands that didn't use a subsearch (therefore set has the same problem as join). So I could come up with only 3 options:

Option 1:

source=summary.log | xmlkv | search transactionFailed=true
| join transactionId max=0 [search
source=summary.log | xmlkv | search transactionFailed=true
| map search="search source=location.log \"[$transactionId$]\" | eval transactionId=\"$transactionId$\""
| rex "(?<locationId>\w+)$" | fields transactionId locationId]
| table *

Option 2:

source=summary.log | xmlkv | search transactionFailed=true
| map search="search source=location.log \"[$transactionId$]\" | eval transactionId=\"$transactionId$\"
| eval every=\"$every$\" | eval single=\"$single$\" | eval field=\"$field$\" | eval I=\"$I$\" | eval need=\"$need$\""
| rex "(?<locationId>\w+)$" | table *

Option 3:

source=location.log | rex "\[(?<transactionId>.+?)\]" | rex "(?<locationId>\w+)$"
| map search="source=summary.log \"$transactionId$\" | eval locationId=\"$locationId$\""
maxsearches=2147483647
| xmlkv | search transactionFailed=true | table *

Option 1 does EXACTLY the same search twice (which is why I looked into storing its results with collect or outputlookup), but a problem is that this is just a simplified mock query, the actual query is complicated and repeating the large thing is non-DRY and probably bad for performance.

Option 2 requires a list of every single field which works, but is messy (I think it's about 10 fields). It also isn't flexible but that isn't a requirement since the fields won't be changing any time soon.

Option 3 will never finish, like ever. The vast majority of transactions are not failures and the map command is very slow. No I don't need more than a few thousand, the maxsearches above was just to highlight the problem of the syntactically cleanest approach.

All of them are bad. Option 3 is not an option. Options 1 is slow and messy. Options 2 is messy but otherwise fine making it the best. I've spent time searching through splunk documentation but there might be a relevant command I missed. I didn't find any other people's questions to be the same as my problem.

What surprises me is that this seems like a fairly normal use case for the map command yet there is no option to keep all non-internal fields.

Bottom line: Is there any way to perform such a search in a clean and reasonable way?

I'm not at work right now but I think the version of Splunk is 6.1.8 (it isn't Splunk Mint).

Answer 1

Splunk only supports piping things together like SQL but we want something more like PL/SQL so we're using a tool in a way it wasn't designed for. Meaning that without a major change to Splunk there just isn't a good way to do this.

The best solution currently is to have the app calculate everything of interest and log that. In my example the summery.log should include the locationId. After being in prod for a while it isn't hard to know all of the important fields and to put those in each applicable log (preprod you'll just have to guess based on QA). Ideally you should have a log for each purpose: performance.log, security.log, errors.log, businessMetrics.log (eg "how many successes per day in New York?") each with all of the fields needed in a nice key value pair format. It is OK for the same field to be in multiple logs and for a single event to trigger more than 1 log. If all goes well you could drop Splunk for something cheaper and better like Elasticsearch (you may have to run a script to convert old logs into JSON).

So the "solution" to this Splunk problem is to not use Splunk... Yeah that's a bad indication for Splunk. For Splunk to really shine it needs to support arbitrary JavaScript (or another language) with predefined functions for accessing the logs. But even then: why do you need extremely complex searches? If these fields all exist in the logs then you might as well put them in easy to reach places. So even then Splunk would only be useful for answering business questions involving a non-standard field or debugging an issue that can't be solved with the existing fields (both examples using rex, xmlkv, or spath to parse).