Outcold Solutions is sponsoring Splunk .conf26 - see you there!

Splunk HTTP Event Collector

Configure HTTP Event Collector secure connection

Splunk HEC ships with self-signed certificates by default. Collectord gives you a few configuration options for how to trust them.

Configure trusted SSL connection to the self-signed certificate

To trust Splunk’s self-signed certificate properly (instead of disabling validation with insecure = true), copy the server CA certificate from $SPLUNK_HOME/etc/auth/cacert.pem onto the host and point Collectord at it.

The configuration below accepts the license, points at your HEC URL, and tells Collectord to trust cacert.pem while verifying the server name SplunkServerDefaultCert (the name baked into Splunk’s default self-signed certificate):

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
caPath = /opt/collectorforlinux/etc/cacert.pem
caName = SplunkServerDefaultCert

Place cacert.pem at /opt/collectorforlinux/etc/cacert.pem and restart the collectorforlinux service.

HTTP Event Collector incorrect index behavior

When Collectord forwards an event to an index that the HEC token isn’t allowed to write to, HEC rejects the payload. The incorrectIndexBehavior setting controls how Collectord handles these rejections:

  • RedirectToDefault - the default. Forwards rejected events to the HEC token’s default index.
  • Drop - drops rejected events.
  • Retry - keeps retrying. Be careful: a single rejected pipeline (process stats, for example) can stall every other pipeline on the host.

Set the behavior in the output configuration:

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
incorrectIndexBehavior = Drop

Using a proxy for HTTP Event Collector

If Collectord has to reach HEC through a proxy, set proxyUrl. For an SSL connection through the proxy, also include the proxy’s CA certificate:

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
proxyUrl = http://proxy.example:4321
caPath = /opt/collectorforlinux/etc/proxie-ca.pem

Using multiple HTTP Event Collector endpoints for load balancing and failover

When you have several HEC endpoints fronting the same indexer cluster, Collectord can spread traffic across them and fail over automatically.

Three URL-selection algorithms are available:

  • random - pick a random URL on first selection and after each failure (connection error or HTTP status >= 500).
  • round-robin - start with the first URL and advance on each failure.
  • random-with-round-robin - pick a random URL on first selection, then advance round-robin on each failure.

random-with-round-robin is the default.

[general]
acceptLicense = true

[output.splunk]
urls.0 = https://hec1.example.com:8088/services/collector/event/1.0
urls.1 = https://hec2.example.com:8088/services/collector/event/1.0
urls.2 = https://hec3.example.com:8088/services/collector/event/1.0

urlSelection = random-with-round-robin

token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

Enable indexer acknowledgement

HEC offers Indexer acknowledgment, which confirms not just that HEC accepted a payload but that the indexer wrote it. It costs throughput - sometimes a lot - so enable it only when you need delivery guarantees. You have to enable it on both the HEC token and in the Collectord output:

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
ackUrl = https://hec.example.com:8088/services/collector/ack
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
ackEnabled = true
ackTimeout = 3m

Client certificates for collector

If your HEC endpoint requires mutual TLS, place the client certificate and key on the host and point Collectord at them:

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
clientCertPath = /opt/collectorforlinux/etc/client-cert.pem
clientKeyPath = /opt/collectorforlinux/etc/client-cert.key

Support for multiple Splunk clusters

To forward data from the same host to more than one Splunk cluster, declare additional [output.splunk::<name>] sections. The example below adds a prod1 output:

[output.splunk::prod1]
url = https://prod1.hec.example.com:8088/services/collector/event/1.0
token = AF420832-F61B-480F-86B3-CCB5D37F7D0D

Anything not set on the named output is inherited from the default output.splunk block.