Splunk log data optimization

Question

I am new to Splunk and I wish to optimize the log data files (do a lossless compression) that I will add to splunk. Since the data has to be textual (not binary or any other format), I cannot go for huffman coding etc, and don't know where to start from.

Any help/idea would be great.

halr9000 halr9000 · Accepted Answer · 2015-07-17T20:29:57

According to Monitor files and directories:

Splunk Enterprise decompresses archive files before it indexes them. It can handle these common archive file types: tar, gz, bz2, tar.gz, tgz, tbz, tbz2, zip, and z.

I suggest using any of the above compression methods, and then configure Splunk to monitor the files by filename or directory spec using the UI or props.conf. If for some reason you need to use a different compression algorithm, you can do so and then instruct Splunk to use a special unarchive_cmd during the index pipeline. You can read more about that by looking at props.conf.spec. Here is a relevant portion:

unarchive_cmd = <string>
* Only called if invalid_cause is set to "archive".
* This field is only valid on [source::<source>] stanzas.
* <string> specifies the shell command to run to extract an archived source.
* Must be a shell command that takes input on stdin and produces output on stdout.
* Use _auto for Splunk's automatic handling of archive files (tar, tar.gz, tgz, tbz, tbz2, zip)
* This setting applies at input time, when data is first read by Splunk. 
  The setting is used on a Splunk system that has configured inputs acquiring the data.
* Defaults to empty.

Splunk log data optimization

1 Answers