Unable to send metrics to Splunk HEC

Question

I have a HEC input set up on my Splunk v. 8.1.1 server and I am trying to send metrics to it, ie.:

curl -k https://$SPLUNK_HOST:$HEC_PORT/services/collector/raw -H "Authorization: Splunk $HEC_TOKEN" \
-d "
{'time': 1610123044, 'fields': {'metric_name': 'kernel.all.load', '_value': 2.8499999046325684, 'instance_id': 1, 'instance_name': '1 minute'}}
{'time': 1610123044, 'fields': {'metric_name': 'kernel.all.load', '_value': 3.8299999237060547, 'instance_id': 5, 'instance_name': '5 minute'}}
{'time': 1610123044, 'fields': {'metric_name': 'kernel.all.load', '_value': 3.6700000762939453, 'instance_id': 15, 'instance_name': '15 minute'}}"

(Note: line breaks within the quotes added for clarity)

I get a positive response from the server every time:

{'text': 'Success', 'code': 0}

But no data is saved in the index. I have the default index set for the HEC input. If I delete or disable this index I get the message on the main page in Splunk, ie.:

Received event for unconfigured/disabled/deleted index=pcp_hec with source="source::http:PCP via HEC" host="host::localhost:32926" sourcetype="sourcetype::httpevent". So far received events from 1 missing index(es).

When I restore the index back the message disappears, but still no data is saved under the index. I cannot figure out what is wrong in my case, because the official documentation is very brief on this subject. I found two threads on this forum (1, 2) and a few similar ones elsewhere, but the answers only contained the same example from the documentation. I tried to include the metadata from the examples, but that did not solve the problem. Nowhere does it say what the perflog sourcetype actually is. I also tried log2metrics_json for the sourcetype, but it did not help either.

Adam Marshall Adam Marshall · Accepted Answer · 2021-02-23T01:57:22

It looks like you’re using the raw endpoint but sending a json object. Change your endpoint to just /services/collector and try again.

From the examples page, in the Raw example section, it shows that data for the raw endpoint is just a simple string.

Otherwise, even though your event was accepted by the API it doesn’t mean it was indexed yet (it returns the Success right away so your client isn’t hung waiting for the indexer). After an event is accepted, there are multiple reasons it might not be indexed including an outage or some other system failure. This is where Indexer Acknowledgement comes in.

Indexer Acknowledgement can be enabled for each token. If enabled, along with the “Success” status, an int ackId will be sent. Once you have the ackId (or ids from multiple requests from the same client) you can check their status with the endpoint /services/collector/ack:

curl -k https://<host>:<port>/services/collector/ack?channel=FE0ECFAD-13D5-401B-847D-77833BD77131 -H "Authorization: Splunk {token}" -d "{"acks":[0,1,2,3]}"

This will return something like:

{"acks": {"0": true, "1": false, "2": true, "3": false}}

Where true means it has been indexed and false means it hasn’t (yet). For each that hasn’t, the docs suggest you continue polling until either they have been indexed or you reach a timeout (that you configure) before you send the event again.

In the request there’s a query parameter channel that’s required in addition to indexer acknowledgment being enabled on your token. This is any unique string to keep track of your client’s events and statuses. Instead of a query parameter you can also pass the value in a X-Splunk-Request-Channel header.

Make sure to read the Client Behavior section at the bottom of the docs too. It has some helpful best practices I didn’t mention here.

Unable to send metrics to Splunk HEC

1 Answers