Outcold Solutions is sponsoring Splunk .conf26 - see you there!

# Monitoring OpenShift

## Configuration

This page is the reference for the Collectord deployment manifest. It covers the two image variants we ship (the standard image on Docker Hub and the Red Hat–certified image on `registry.connect.redhat.com`), the OpenShift objects that get created, and the configuration patterns you’ll most often need - secrets, EC2 metadata, and dynamic indexes.

## Using images on docker.io (hub.docker.com)

The standard images are built on `scratch` \- minimal, no shell, nothing to update beyond Collectord itself. Use these unless your environment specifically requires Red Hat–certified containers.

### Current most recent version of OpenShift

- [collectorforopenshift.yaml](/content/docs/monitoring-openshift/reference/index.html)

> Looking for an older release? Manifests for previous Collectord versions live at [github.com/outcoldsolutions/collectord-configurations](https://github.com/outcoldsolutions/collectord-configurations).

## Using certified images on registry.connect.redhat.com

If your environment requires Red Hat–certified containers - for compliance, support contracts, or internal policy - use the RHEL-based image instead. It’s published at [outcoldsolutions/collectorforopenshift](https://access.redhat.com/containers/?tab=overview&platform=docker#/registry.connect.redhat.com/outcoldsolutions/collectorforopenshift) and requires a pull secret to download.

### Current most recent version of OpenShift

- [collectorforopenshift.yaml](/content/docs/monitoring-openshift/reference-rhel/index.html)

> Manifests for previous releases live at [github.com/outcoldsolutions/collectord-configurations](https://github.com/outcoldsolutions/collectord-configurations).

### registry.connect.redhat.com authentication

> `registry.connect.redhat.com` is not the same as `registry.access.redhat.com`. The latter hosts Red Hat’s own images and works with OpenShift out of the box; the former hosts certified partner images and requires authentication.

To pull from `registry.connect.redhat.com` you need to give OpenShift credentials it can use. The general procedure is documented in [Allowing Pods to Reference Images from Other Secured Registries](https://docs.openshift.com/container-platform/3.11/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries); the steps below walk through it specifically for Collectord.

After applying the manifest, switch to the namespace where Collectord runs (default `collectorforopenshift`):

```bash
$ oc project collectorforopenshift
```

On Linux, log in with Docker - the resulting `~/.docker/config.json` is what we’ll convert into a secret:

```bash
$ docker login registry.connect.redhat.com
Username: [redhat-username]
Password: [redhat-user-password]
Login Succeeded
```

> Log in with your Red Hat **username**, not your email. Both will succeed at the login step, but only the username form lets you actually pull images.

Create the pull secret from the Docker config file:

```bash
$ oc --namespace collectorforopenshift secrets new rhcc .dockerconfigjson=$HOME/.docker/config.json
```

> On macOS, Docker Desktop stores credentials in the Keychain instead of `config.json`, so the file-based approach above won’t work. Create the secret directly from the command line:
> `oc secrets --namespace collectorforopenshift new-dockercfg rhcc --docker-server=registry.connect.redhat.com --docker-username=<user_name> --docker-password=<password> --docker-email=<email>`.  
> Make sure your password doesn’t end up in shell history - run `export HISTFILE=/dev/null` first, or see [Execute command without keeping it in history](https://stackoverflow.com/questions/8473121/execute-command-without-keeping-it-in-history).

Link the `rhcc` secret to the `collectorforopenshift` service account so the kubelet uses it for image pulls:

```bash
$ oc --namespace collectorforopenshift secrets link collectorforopenshift rhcc --for=pull
```

If pods were already created before you linked the secret, delete them so the scheduler recreates them with the right pull credentials:

```bash
$ oc delete --namespace collectorforopenshift pods --all
```

## Created OpenShift Objects

Applying `collectorforopenshift.yaml` creates the following objects - everything is scoped to the `collectorforopenshift` project, and the cluster-level grants are read-only:

- `Project` `collectorforopenshift`.
- `ClusterRole` `collectorforopenshift` with `get`, `list`, and `watch` on the deployment objects Collectord needs to enrich logs and metrics with OpenShift metadata.
- `ServiceAccount` `collectorforopenshift` \- the identity Collectord uses to talk to the OpenShift API.
- `ClusterRoleBinding` `collectorforopenshift` binds the service account to the cluster role.
- `ConfigMap` `collectorforopenshift` carries the Collectord configuration file.
- `DaemonSet` `collectorforopenshift` runs Collectord on non-master nodes.
- `DaemonSet` `collectorforopenshift-master` runs Collectord on master nodes.
- `Deployment` `collectorforopenshift-addon` is a single Collectord pod that forwards cluster-wide data (events, cluster metrics) once per cluster - not once per node.

The `collectorforopenshift.yaml` file itself is heavily commented; read it alongside this page for the full picture of what each input collects.

## Collectord configuration

The `ConfigMap` `collectorforopenshift` holds Collectord’s configuration as `ini` files, with every default value visible inline.

You can override any value through environment variables using this format:

```text
COLLECTOR__{ANY_NAME}={section}__{key}={value}
```

Environment variables are the quickest way to try things out and debug - but for anything you’re going to keep, edit the configuration file based on the defaults shipped in `collectorforopenshift.yaml`. It’s easier to review and easier to diff.

### Using secrets to manage configurations

When you don’t want the HEC token or license key sitting in plain text in the manifest, store them in an OpenShift secret and project them as environment variables. The example below sets up the Splunk HEC token and the license that way.

First, make sure the `collectorforopenshift` namespace exists - create it if it doesn’t:

```bash
$ oc create namespace collectorforopenshift
```

Create the secret with both values:

```bash
$ oc create secret generic collectorforopenshift \
    --namespace collectorforopenshift \
    --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
    --from-literal=license="general__license="
```

In the YAML manifest, find the env block on each workload (two DaemonSets and one Deployment) and add:

```yaml
  env:
  - name: COLLECTOR__SPLUNK_TOKEN
    valueFrom:
      secretKeyRef:
        name: collectorforopenshift
        key: splunk-token
  - name: COLLECTOR__LICENSE
    valueFrom:
      secretKeyRef:
        name: collectorforopenshift
        key: license
```

Then apply the manifest as you normally would in the [installation guide](/content/docs/monitoring-openshift/installation/index.html).

## Attaching EC2 Metadata

If you’re running OpenShift on EC2 and want every event to carry instance-level context - instance ID, instance type, availability zone - point Collectord at the EC2 metadata service. Each entry maps a field name to a path from [Instance Metadata and User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html):

```ini
# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
```

For example, to attach `ec2_instance_id` and `ec2_instance_type` to every event:

```ini
[general]
...
ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
...
```

## Placeholders in indexes and sources

Static index names work for small clusters, but once you have multiple teams sharing a cluster you’ll often want logs to land in per-team or per-namespace indexes. Collectord lets you use any metadata field as a placeholder in `index`, `source`, and `sourcetype`.

For example, to route each namespace’s logs to its own index:

```ini
[input.files]
index = oc_{{openshift_namespace}}
```

Or to build a structured `source` path that mirrors the workload hierarchy - useful when a single index holds logs from many namespaces and you want Splunk’s source-based filtering to do the heavy lifting:

```ini
[input.files]
source = /{{openshift_namespace}}/{{::coalesce(openshift_daemonset_name, openshift_deployment_name, openshift_statefulset_name, openshift_cronjob_name, openshift_job_name, openshift_replicaset_name, openshift_pod_name)}}/{{openshift_pod_name}}/{{openshift_container_name}}
```

The `coalesce` function picks the first non-empty workload name, so a pod owned by a Deployment shows the Deployment name, a standalone pod shows its own name, and so on.

## License server endpoint (static IPs for egress-restricted networks)

Collectord checks in with the Outcold-hosted license server on a schedule and reports telemetry to the same host. The default endpoint - `license.outcold.solutions` \- sits behind AWS API Gateway and CloudFront, which means it resolves to a large, frequently changing range of IP addresses. That’s fine for most clusters, but it’s a problem in environments where outbound traffic has to traverse a firewall with a strict allow-list.

Available since Collectord version `26.04.2`

Starting in 26.04.2, we also publish `license-static.outcold.solutions`, a parallel endpoint backed by AWS Global Accelerator. It serves the same license and telemetry traffic from two stable anycast IPv4 addresses, so the allow-list stays small and doesn’t drift over time:

- `166.117.80.67`
- `99.83.183.50`

The original `license.outcold.solutions` endpoint is unchanged - if your cluster can already reach it, there’s nothing you need to do. Only switch over if you need the static IPs.

To use the static-IP endpoint, override the two URLs under `[general]` in the ConfigMap (or via env var):

```ini
[general]
licenseEndpoint = https://license-static.outcold.solutions/license/
telemetryEndpoint = https://license-static.outcold.solutions/telemetry/
```

The same change as environment variables on each workload (two DaemonSets and one Deployment):

```yaml
env:
- name: COLLECTOR__LICENSEENDPOINT
  value: "general__licenseEndpoint=https://license-static.outcold.solutions/license/"
- name: COLLECTOR__TELEMETRYENDPOINT
  value: "general__telemetryEndpoint=https://license-static.outcold.solutions/telemetry/"
```

After that, allow-list the two IPs above for outbound `tcp/443` and you’re done. The static-IP endpoint runs the same backend as the default one, so the license behaviour is identical - the only differences are the hostname and the front door.

## FIPS-compliant images

Since version `26.04`, we publish FIPS 140-compliant images alongside the regular images, with a `-fips` suffix:

```
outcoldsolutions/collectorforopenshift:26.04.3-fips
```

By default, these images use FIPS-certified cryptographic algorithms but still allow fallback to non-approved algorithms when needed. To strictly enforce FIPS 140 mode, set the environment variable `GODEBUG=fips140=only` \- Collectord will crash if any code attempts to use a non-FIPS-140-compliant algorithm.

Collectord logs its FIPS state at startup and includes it in the output of `collectord diag` and `collectord verify`.

See [FIPS 140-3 Compliance](https://go.dev/doc/security/fips140) for details on the Go implementation.