Outcold Solutions is sponsoring Splunk .conf26 - see you there!

# Monitoring Kubernetes

## Installation

This guide walks you through installing Monitoring Kubernetes end-to-end: configuring the Splunk app and HTTP Event Collector, preparing the cluster, and deploying Collectord to forward metadata-enriched container logs, host logs, and metrics. A typical install takes under 10 minutes. If you don’t have a license yet, you can [request a 30-day evaluation](/content/trial/request/index.html).

## Splunk configuration

### Install the Monitoring Kubernetes application

Install the latest version of [Monitoring Kubernetes](https://splunkbase.splunk.com/app/3743/) from Splunkbase on your Search Heads only.

If you’re using a dedicated index that isn’t searchable by default, update the `macro_kubernetes_base` macro to include it:

```text
macro_kubernetes_base = (index=kubernetes)
```

### Enable HTTP Event Collector in Splunk

Collectord forwards data to Splunk over the HTTP Event Collector (HEC). If HEC isn’t enabled yet, follow Splunk’s guide to [Configure the Splunk HTTP Event Collector for use with additional technologies](https://dev.splunk.com/view/event-collector/SP-CAAAE6M).

Once HEC is enabled, you need two pieces of information for the rest of this guide: the HEC endpoint URL and an HEC token. You can verify both with `curl`:

```bash
$ curl -k https://hec.example.com:8088/services/collector/event/1.0 \
       -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
       -d '{"event": "hello world"}'
{"text": "Success", "code": 0}
```

> Don’t set a source type on the token - Collectord assigns its own source types per datatype, and the Splunk app expects them.
>  
> `-k` skips certificate validation; use it only for self-signed certificates.
>  
> Splunk Cloud uses a different HEC URL than Splunk Web - see [Send data to HTTP Event Collector on Splunk Cloud instances](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_Event_Collector_on_Splunk_Cloud_instances).

If your indexes aren’t searchable by default, see [Splunk Indexes](/content/docs/monitoring-kubernetes/splunk-indexes/index.html) for how to configure them on both the Splunk side and inside Collectord.

## Install Collectord for Kubernetes

### Pre-requirements

Collectord works out of the box with [CRI-O](https://cri-o.io/), [Containerd](https://containerd.io/), and [Docker](https://www.docker.com/) as runtime engines.

The most important thing to get right before you install is container log rotation. Some Kubernetes providers ship aggressive defaults - AKS, for example, rotates across just 5 files of 10 MiB. Those files are your safety buffer for any disruption between Collectord and Splunk HEC - a connectivity issue, an HEC outage, a misconfiguration. The math is straightforward: a container writing 10 MiB per hour gives you a 5-hour buffer before the oldest log is overwritten; at 10 MiB per minute, that buffer shrinks to 5 minutes.

Check your provider’s defaults and raise them if they’re too tight. Rotation is normally controlled through the KubeletConfiguration (or kubelet command-line flags). At minimum, aim for 5 files of 128 MiB - tune from there based on how much log volume your pods produce:

```yaml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
...
containerLogMaxSize: 128Mi
containerLogMaxFiles: 5
```

### Installation

If you prefer Helm, see the [collectord-splunk-kubernetes Helm chart](https://github.com/outcoldsolutions/collectord-helm-charts/blob/master/charts/collectord-splunk-kubernetes).

Download the latest [collectorforkubernetes.yaml](/content/docs/monitoring-kubernetes/reference/index.html). The manifest creates the `collectorforkubernetes` namespace and deploys every workload it needs.

Open the file and edit it to:

- Set the Splunk HEC URL, token, and (if needed) certificate options.
- [Review and accept the license agreement](/content/legal/license-agreement/index.html) and paste in your license key.
- Optionally, name the cluster - useful when you’re monitoring more than one and want to filter by cluster in the app.

```ini
[general]

acceptLicense = false

license =

fields.kubernetes_cluster = -

...

# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url =

# Splunk HTTP Event Collector Token
token =

# Allow invalid SSL server certificate
insecure = false

# Path to CA certificate
caPath =

# CA Name to verify
caName =
```

A filled-in example:

```ini
[general]

acceptLicense = true

license = ...

fields.kubernetes_cluster = development

...

# Splunk output
[output.splunk]

# Splunk HTTP Event Collector url
url = https://hec.example.com:8088/services/collector/event/1.0

# Splunk HTTP Event Collector Token
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

# Allow invalid SSL server certificate
insecure = true
```

> If you’re deploying onto a cluster that’s been running for a while and has a lot of historical logs on disk, Collectord will start by forwarding all of them - which can spike both your network and your Splunk indexing. Use the `[general]` settings `thruputPerSecond` to cap throughput and `tooOldEvents` to skip events older than a given age.

Apply the manifest:

```bash
$ kubectl apply -f ./collectorforkubernetes.yaml
```

Check that the workloads came up:

```bash
$ kubectl get all --namespace collectorforkubernetes
```

If the pods aren’t running, see [Troubleshooting](/content/docs/monitoring-kubernetes/troubleshooting/index.html).

Once the images are pulled and the pods are `Running`, open the Monitoring Kubernetes app in Splunk - dashboards should start populating within a minute or two.

By default, Collectord forwards container logs, host logs (including syslog), and metrics for hosts, pods, containers, and processes.

## Next steps

- Review the predefined [alerts](/content/docs/monitoring-kubernetes/alerts/index.html) and enable the ones relevant to your environment.
- If something looks off, work through the [troubleshooting](/content/docs/monitoring-kubernetes/troubleshooting/index.html) checks.
- Enable [Audit Logs](/content/docs/monitoring-kubernetes/audit/index.html). Kubernetes doesn’t write API server audit logs out of the box; you’ll need to enable them on the API server before Collectord can forward them.
- Verify [Prometheus Metrics](/content/docs/monitoring-kubernetes/prometheus/index.html). The default configuration covers most clusters, but if Control Plane metrics are missing in the dashboards, double-check that the Prometheus endpoints are reachable from Collectord.
- For better indexing performance, split logs and metrics into separate indexes. See [Splunk Indexes](/content/docs/monitoring-kubernetes/splunk-indexes/index.html) for the recommended layout and how to wire it up on both sides.
- For search-time field extraction on container logs, see [Splunk fields extraction for container logs](/content/docs/monitoring-kubernetes/splunk-fields-extraction/index.html).
- For per-pod control over log forwarding - application logs, custom log file paths, multi-line patterns, field extraction, index/source/sourcetype overrides, dropping noisy lines, and hashing sensitive values - see [annotations](/content/docs/monitoring-kubernetes/annotations/index.html).