Unable to integrate Splunk with Jenkins

Question

I have installed Jenkins on AWS EC2 instance and it is running fine. I have installed Splunk on same EC2 instance.

My Jenkins URL:-
http://10.x.x.x:8080

And my Splunk URL:-
http://10.x.x.x:8000

I have installed Splunk plugin on Jenkins as well and trying to configure Splunk with Jenkins such that data from Jenkins will act as source to Splunk. I tried a lot but it is not working. On Splunk, I created an 'HTTP Event Collector' token and trying to configure on Jenkins using this token.

On Jenkins -> Manage Jenkins -> Configure System, under 'Splunk for Jenkins Configuration',

**enable checkbox -> checked, 
indexer hostname -> host IP address of Splunk (which is same as Jenkins)
HTTP Input Port -> 8088
HTTP Input Token -> <<< Token generated at Splunk side >>>
SSL Enabled -> Checked (tried with uncheck as well)
Jenkins Master Hostname -> Host name of Jenkins (same as Splunk)**

When I click on 'Test connection', nothing happens and fails. Please assist to clear my following doubts :- Q1) I googled and found HTTP input port for Splunk is 8088 then what is 8000. I can access to Splunk console using 8000. Then it should be 8000 only right instead of 8088?

Q2) Since both Splunk and Jenkins are running on different ports though they have same host name then I believe there should not be any connectivity issue. Please confirm.

Q3) Am I missing anything to confure Splunk with Jenkins finally.

Answer 1

Port 8000 is for Splunk GUI/web/login page URL.

Port 8089 is for Splunk daemon's port#

Port 8088 / any other that you can enter, if for HEC (HTTP Event Collector).

As ports are different, you'd be fine.

What port and protocol does the HTTP Event Collector receive data on and how can I change that?

The HTTP Event Collector receives data over HTTPS on TCP port 8088 by default. You can change the port as well as disable HTTPS by clicking on the Global Settings button at the top of the HTTP Event Collector management page.

When you configured your New Token (to get the token and HTTP Input Connector port (either default or any user defined)) i.e. when you did this: Splunk > Settings > Data Inputs > HTTP Event Collector > Global Settings or New Token, did you select "ENABLE" (usually has a blue color boundary line if selected) button for "All Tokens"?

http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

If your HEC (HTTP Event Collector) is configured correctly, from the host Linux $ prompt, you should be able to CURL like this successfully. Note: It's https below and port 8088 if that's what ENABLED in HEC configuration and 10.x.x.x if your IP for your machine and assuming B5A79AAD-D822-46CC-80D1-819F80D7BFB0 is your Token (generated by Splunk).

curl -k https://10.x.x.x:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'

{"text": "Success", "code": 0}

Have you looked into installing Jenkins Add-On (within Splunk) which works with "Splunk plugin" (available in Jenkins)? This is one another way to get Jenkins metrics into Splunk (indexes provided by this Jenkins Add-on/App), you should check that too as it provides out of the box dashboards etc.

Splunk Plugin (in Jenkins) works together with a Splunk App for Jenkins that provides out-of-the-box dashboards and search capabilities to enable organizations to run a high performing Jenkins cluster and bring operational intelligence into the software development life cycle.

Run ps -eAf|grep -i splunk and see what ports show up especially the port for splunkd (daemon) service, it's usually 8089. Try that and see if that helps if there's no 8088 in the output of the above command. netstat -tulpn |grep 8088 should show something about 8088 if that's listening on the host machine.

See the full doc on configuring Splunk plugin (available for/in Jenkins) here: https://wiki.jenkins.io/display/JENKINS/Splunk+Plugin+for+Jenkins Make sure you configure Steps 1 and 2 at least. 4th step is Optional.

and How to configure Jenkins app/add-on (within Splunk) here: https://splunkbase.splunk.com/app/3332/

Until in Jenkins main / global configuration (under Splunk for Jenkins configuration) you see Connection verified by clicking "Test Connection", you can't push Jenkins data (JSON blobs) to Splunk indexes (available via Jenkins Add-on/App in Splunk).