Splunk forwarder not getting configured through command line

Question

I am very new to Splunk and have been trying to understand it. I am trying to automate the Splunk forwarder configuration through Ansible but before that I want to try manually through command line.

I have installed Splunk forwarder on my Red Hat machine (localhost) and I can access Splunk through localhost:8000. Now when I am trying following command to add forward-server (indexer) and monitor(data input), i cant see anything on UI.

Below are the commands:

splunk add forward-server localhost:9997 -auth admin:changeme
splunk add monitor /var/log

My understanding is it should log a separate entry on UI under settings->datainputs for /var/log, right?

Also, I enabled port 9997 by using following command:

splunk enable listen 9997 -auth admin:changeme

My questions are:

1. Is my understanding right of separate entry should get created on UI?
2. Can I use localhost as both Splunk forwarder and indexer which is what I am doing here?
3. What am I missing as a part of configuration?

Here is my contents of my inputs.conf file:

[monitor:///var/log]
index=test_index
[splunktcp://9997]
disabled = 0

Ideally splunktcp should also get create automatically once I enable the port but it didn't get created and I added it manually.

Answer 1

You don't need to setup the forwarding bit in a Splunk all-in-one box (Search Head + Indexer) if all you want is ingesting data local to that box

When you're interested in data from a remote server You need to install a Splunk universal forwarder on it, which directories/files/scripts output/etc and set it up to forward all data you monitor there to the indexer(s) (every forwarder will share that outputs.conf configuration and as such you should set it through the deployment server, not locally)

Answer 2

You need an outputs.conf which is pointing to your indexer(s) and have to restart splunkd after making those changes. You also need to enable listening on your indexer(s).

You do not enable listening on the UF as you described in your question unless its an intermediate forwarder

Answer 3

It seems like you didn't installed Splunk forwarder but Splunk EP single instance. In forwarder, you wouldn't have UI.

so if it Universal forwarder, you will be pointing it to your deployment server first, which can deploy outputs.conf file into forwarder to tell, where are your indexers are. From that point on, whether you have monitors or not, forwarder should be starting its internal logs to Splunk indexer.

if it is a single instance as I thought, then you can just add from ui. for ansible automation, i guess we can think of 2 strategies,

1. automation via Splunk configuration files
2. automation via Splunk cli commands (as you have tried in the question)

one key thing to note, you do monitoring automation with Splunk deployment server in modern systems. i can think of an exception only in systems like containers which you may want to preconfigure each container to make them start sending immediately.

Assuming, your automation will take place in clustered environment, you may want to read little more in Splunk architecture, checking on the commands you have put:

splunk add forward-server localhost:9997 -auth admin:changeme # this is telling universal forwarder where to forward data (generally an indexer, sometimes HF) More here, https://docs.splunk.com/Documentation/Forwarder/7.1.2/Forwarder/Configuretheuniversalforwarder

splunk add monitor /var/log # this is locally done on universal forwarder, since index not defined, if anything successful to monitor, it should be going to index=main in splunk side. in forwarder, you would find the trace of your command in /opt/splunkforwarder/etc/system/local/inputs.conf splunk enable listen 9997 -auth admin:changeme # this simply tells splunk software to listen incoming traffic. 9997 is default data port for Splunk https://docs.splunk.com/Documentation/Forwarder/8.1.1/Forwarder/Enableareceiver

now questions:

1. it depends :) . in a single instance, you can configure them in UI, or .conf files or via CLI. in a multi clustered environment, you would use other splunk instances such as deployment server, license server, cluster master, search head deployer.
2. I believe you can, however, you need to have 2 seperate installs, one for splunk indexer-search head (single instance) , and one for splunk universal forwarder.

Following this you will need configure ports in respect to splunk architecture. such that, splunk enable listen 9997 -auth admin:changeme will be on indexer vs splunk add forward-server localhost:9997 -auth admin:changeme splunk add monitor /var/log on forwarder.

3. I believe Architecture. There are several deployment architectures for Splunk. They are not super complex. I personally have a mix of architectures in my work. Secondly, UI is there, however my first personal choice of configuration is .conf files, then cli commands (mostly troubleshooting times since you can do other tests at the same time) , then UI.