Outcold Solutions is sponsoring Splunk .conf26 - see you there!

# Monitoring Kubernetes

## Configuration

This page is the reference for the Collectord deployment manifest and the configuration file it ships with - what objects get created in your cluster, how to override settings safely, and a few patterns (secrets, EC2 metadata, dynamic indexes) you’ll likely reach for after the initial install.

## collectorforkubernetes.yaml

- [collectorforkubernetes.yaml](/content/docs/monitoring-kubernetes/reference/index.html)

> Looking for an older release? Manifests for previous Collectord versions live at [github.com/outcoldsolutions/collectord-configurations](https://github.com/outcoldsolutions/collectord-configurations).

## Created Kubernetes Objects

Applying `collectorforkubernetes.yaml` creates the following objects - everything is scoped to the `collectorforkubernetes` namespace, and the cluster-level grants are read-only:

- `Namespace` `collectorforkubernetes`.
- `ClusterRole` `collectorforkubernetes` with `get`, `list`, and `watch` on the deployment objects Collectord needs to enrich logs and metrics with Kubernetes metadata.
- `ServiceAccount` `collectorforkubernetes` - the identity Collectord uses to talk to the Kubernetes API.
- `ClusterRoleBinding` `collectorforkubernetes` binds the service account to the cluster role.
- `ConfigMap` `collectorforkubernetes` carries the Collectord configuration files.
- `DaemonSet` `collectorforkubernetes` runs Collectord on non-master nodes.
- `DaemonSet` `collectorforkubernetes-master` runs Collectord on master nodes.
- `Deployment` `collectorforkubernetes-addon` is a single Collectord pod that forwards cluster-wide data (events, cluster metrics) once per cluster - not once per node.

The `collectorforkubernetes.yaml` file itself is heavily commented; read it alongside this page for the full picture of what each input collects.

## Collectord configuration

The `ConfigMap` `collectorforkubernetes` holds Collectord’s configuration as `ini` files, with every default value visible inline.

You can override any value through environment variables using this format:

```text
COLLECTOR__{ANY_NAME}={section}__{key}={value}
```

Environment variables are the quickest way to try things out and debug - but for anything you’re going to keep, edit the configuration file based on the defaults shipped in `collectorforkubernetes.yaml`. It’s easier to review and easier to diff.

### Using secrets to manage configurations

When you don’t want the HEC token or license key sitting in plain text in the manifest, store them in a Kubernetes secret and project them as environment variables. The example below sets up the Splunk HEC token and the license that way.

First, make sure the `collectorforkubernetes` namespace exists - create it if it doesn’t:

```bash
kubectl create namespace collectorforkubernetes
```

Create the secret with both values:

```bash
kubectl create secret generic collectorforkubernetes \
    --namespace collectorforkubernetes \
    --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
    --from-literal=license="general__license="
```

In the YAML manifest, find the env block on each workload (two DaemonSets and one Deployment) and add:

```yaml
env:
- name: COLLECTOR__SPLUNK_TOKEN
  valueFrom:
    secretKeyRef:
      name: collectorforkubernetes
      key: splunk-token
- name: COLLECTOR__LICENSE
  valueFrom:
    secretKeyRef:
      name: collectorforkubernetes
      key: license
```

Then apply the manifest as you normally would in the [installation guide](/content/docs/monitoring-kubernetes/installation/index.html).

## Attaching EC2 Metadata

If you’re running on EC2 and want every event to carry instance-level context - instance ID, instance type, availability zone - point Collectord at the EC2 metadata service. Each entry maps a field name to a path from [Instance Metadata and User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html):

```ini
# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
```

For example, to attach `ec2_instance_id` and `ec2_instance_type` to every event:

```ini
[general]
...
ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
...
```

## Placeholders in indexes and sources

Static index names work for small clusters, but once you have multiple teams sharing a cluster you’ll often want logs to land in per-team or per-namespace indexes. Collectord lets you use any metadata field as a placeholder in `index`, `source`, and `sourcetype`.

For example, to route each namespace’s logs to its own index:

```ini
[input.files]
index = oc_{{kubernetes_namespace}}
```

Or to build a structured `source` path that mirrors the workload hierarchy - useful when a single index holds logs from many namespaces and you want Splunk’s source-based filtering to do the heavy lifting:

```ini
[input.files]
source = /{{kubernetes_namespace}}/{{::coalesce(kubernetes_daemonset_name, kubernetes_deployment_name, kubernetes_statefulset_name, kubernetes_cronjob_name, kubernetes_job_name, kubernetes_replicaset_name, kubernetes_pod_name)}}/{{kubernetes_pod_name}}/{{kubernetes_container_name}}
```

The `coalesce` function picks the first non-empty workload name, so a pod owned by a Deployment shows the Deployment name, a standalone pod shows its own name, and so on.

## License server endpoint (static IPs for egress-restricted networks)

Collectord checks in with the Outcold-hosted license server on a schedule and reports telemetry to the same host. The default endpoint - `license.outcold.solutions` - sits behind AWS API Gateway and CloudFront, which means it resolves to a large, frequently changing range of IP addresses. That’s fine for most clusters, but it’s a problem in environments where outbound traffic has to traverse a firewall with a strict allow-list.

Available since Collectord version `26.04.2`

Starting in 26.04.2, we also publish `license-static.outcold.solutions`, a parallel endpoint backed by AWS Global Accelerator. It serves the same license and telemetry traffic from two stable anycast IPv4 addresses, so the allow-list stays small and doesn’t drift over time:

- `166.117.80.67`
- `99.83.183.50`

The original `license.outcold.solutions` endpoint is unchanged - if your cluster can already reach it, there’s nothing you need to do. Only switch over if you need the static IPs.

To use the static-IP endpoint, override the two URLs under `[general]` in `001-general.conf` (or via env var):

```ini
[general]
licenseEndpoint = https://license-static.outcold.solutions/license/
telemetryEndpoint = https://license-static.outcold.solutions/telemetry/
```

The same change as environment variables on each workload (two DaemonSets and one Deployment):

```yaml
env:
- name: COLLECTOR__LICENSEENDPOINT
  value: "general__licenseEndpoint=https://license-static.outcold.solutions/license/"
- name: COLLECTOR__TELEMETRYENDPOINT
  value: "general__telemetryEndpoint=https://license-static.outcold.solutions/telemetry/"
```

After that, allow-list the two IPs above for outbound `tcp/443` and you’re done. The static-IP endpoint runs the same backend as the default one, so the license behaviour is identical - the only differences are the hostname and the front door.

## FIPS-compliant images

Since version `26.04`, we publish FIPS 140-compliant images alongside the regular images, with a `-fips` suffix:

```
outcoldsolutions/collectorforkubernetes:26.04.3-fips
```

By default, these images use FIPS-certified cryptographic algorithms but still allow fallback to non-approved algorithms when needed. To strictly enforce FIPS 140 mode, set the environment variable `GODEBUG=fips140=only` - Collectord will crash if any code attempts to use a non-FIPS-140-compliant algorithm.

Collectord logs its FIPS state at startup and includes it in the output of `collectord diag` and `collectord verify`.

See [FIPS 140-3 Compliance](https://go.dev/doc/security/fips140) for details on the Go implementation.