How can I provide metrics to Splunk via HTTP?

Question

I have been reading through Splunk Enterprise documentation and it appears I can provide metrics in JSON format over HTTP/HTTPS: https://docs.splunk.com/Documentation/Splunk/8.1.1/Metrics/GetMetricsInOther#Get_metrics_in_from_clients_over_HTTP_or_HTTPS

However I can't see a reference what exactly this JSON format looks like, beyond one example. I'm also not clear from the docs if Splunk can be configured to poll this endpoint on my process, or if I must push the data to Splunk.

RichG RichG · Accepted Answer · 2021-01-04T18:05:56

Splunk's HEC interface is receive-only. It does not poll.

Any time you find a Splunk documentation page that is unclear, submit feedback on it. Splunk's Docs team is great about updating the documents in response to feedback.

Let's look at the example payload from the documentation.

{
  "time": 1486683865,
  "source": "metrics",
  "sourcetype": "perflog",
  "host": "host_1.splunk.com",
  "fields": {
    "region": "us-west-1",
    "datacenter": "dc2",
    "rack": "63",
    "os": "Ubuntu16.10",
    "arch": "x64",
    "team": "LON",
    "service": "6",
    "service_version": "0",
    "service_environment": "test",
    "path": "/dev/sda1",
    "fstype": "ext3",
    "metric_name:cpu.usr": 11.12,
    "metric_name:cpu.sys": 12.23,
    "metric_name:cpu.idle": 13.34
  }
}

The time field is in *nix epoch form and says when the metric was collected. The source field identifies this as a metric. The value is free-text. The sourcetype field tells Splunk how to parse the payload. Your system may have a different source type configured for metrics. The host field identifies the server that generated the metrics. This is free-text. The fields section is where the metrics data goes. The measurements themselves are noted by the "metric_name:" prefix. The name of the metric is free-text. Splunk will treats dots within the metric name as a hierarchy separator.

Everything does not not begin with "metric_name:" is a dimension rather than a metric. Dimensions describe metrics and are optional.

How can I provide metrics to Splunk via HTTP?

1 Answers