Outcold Solutions is sponsoring Splunk .conf26 - see you there! # Monitoring OpenShift ## Installation This guide walks you through installing Monitoring OpenShift end-to-end: configuring the Splunk app and HTTP Event Collector, preparing the cluster, and deploying Collectord to forward metadata-enriched container logs, host logs, and metrics. A typical install takes under 10 minutes. If you don’t have a license yet, you can [request a 30-day evaluation](/content/trial/request/index.html). ## Splunk configuration ### Install the Monitoring OpenShift application Install the latest version of [Monitoring OpenShift](https://splunkbase.splunk.com/app/3836/) from Splunkbase on your Search Heads only. If you’re using a dedicated index that isn’t searchable by default, update the `macro_openshift_base` macro to include it: ```text macro_openshift_base = (index=openshift) ``` ### Enable HTTP Event Collector in Splunk Collectord forwards data to Splunk over the HTTP Event Collector (HEC). If HEC isn’t enabled yet, follow Splunk’s guide to [Configure the Splunk HTTP Event Collector for use with additional technologies](https://dev.splunk.com/view/event-collector/SP-CAAAE6M). Once HEC is enabled, you need two pieces of information for the rest of this guide: the HEC endpoint URL and an HEC token. You can verify both with `curl`: ```bash $ curl -k https://hec.example.com:8088/services/collector/event/1.0 \ -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \ -d '{"event": "hello world"}' {"text": "Success", "code": 0} ``` > Don’t set a source type on the token - Collectord assigns its own source types per datatype, and the Splunk app expects them. > `-k` skips certificate validation; use it only for self-signed certificates. > Splunk Cloud uses a different HEC URL than Splunk Web - see [Send data to HTTP Event Collector on Splunk Cloud instances](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_Event_Collector_on_Splunk_Cloud_instances). If your indexes aren’t searchable by default, see [Splunk Indexes](/content/docs/monitoring-openshift/splunk-indexes/index.html) for how to configure them on both the Splunk side and inside Collectord. ## OpenShift preparation OpenShift 4.x rotates each container’s logs across at most 5 files of 50 MiB. Those files are your safety buffer for any disruption between Collectord and Splunk HEC - a connectivity issue, an HEC outage, a misconfiguration. The math is straightforward: a container writing 50 MiB per hour gives you a 5-hour buffer before the oldest log is overwritten; at 50 MiB per minute, that buffer shrinks to 5 minutes. If you increase the rotation - say, 5 files of 500 MiB on a node running ~20 containers - plan for the disk: that’s roughly 50 GiB per node just for container logs. Follow Red Hat’s [Modification of log rotation of CRI-O in Openshift 4](https://access.redhat.com/solutions/4924281) to adjust rotation. ## Installation If you prefer Helm, see the [collectord-splunk-openshift Helm chart](https://github.com/outcoldsolutions/collectord-helm-charts/blob/master/charts/collectord-splunk-openshift). Make sure you’re logged in as a user with admin rights: ```bash $ oc login -u kubeadmin ``` Download the latest [collectorforopenshift.yaml](/content/docs/monitoring-openshift/reference/index.html) - or [collectorforopenshift-rhel.yaml](/content/docs/monitoring-openshift/reference-rhel/index.html) if you’re using the RHEL-certified image. The manifest creates the `collectorforopenshift` namespace and deploys every workload it needs. Open the file and edit it to: - Set the Splunk HEC URL, token, and (if needed) certificate options. - [Review and accept the license agreement](/content/legal/license-agreement/index.html) and paste in your license key. - Optionally, name the cluster - useful when you’re monitoring more than one and want to filter by cluster in the app. ```ini [general] acceptLicense = false license = fields.openshift_cluster = - ... # Splunk output [output.splunk] # Splunk HTTP Event Collector url url = # Splunk HTTP Event Collector Token token = # Allow invalid SSL server certificate insecure = false # Path to CA certificate caPath = # CA Name to verify caName = ``` A filled-in example: ```ini [general] acceptLicense = true license = ... fields.openshift_cluster = development ... # Splunk output [output.splunk] # Splunk HTTP Event Collector url url = https://hec.example.com:8088/services/collector/event/1.0 # Splunk HTTP Event Collector Token token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0 # Allow invalid SSL server certificate insecure = true ``` > If you’re deploying onto a cluster that’s been running for a while and has a lot of historical logs on disk, Collectord will start by forwarding all of them - which can spike both your network and your Splunk indexing. Use the `[general]` settings `thruputPerSecond` to cap throughput and `tooOldEvents` to skip events older than a given age. Apply the manifest: ```bash $ oc apply -f ./collectorforopenshift.yaml ``` If you’re pulling the Red Hat certified image from `registry.connect.redhat.com`, you also need to configure the pull secret - see the [Configuration Reference](/content/docs/monitoring-openshift/configuration/index.html). Check that the workloads came up: ```bash $ oc get all --namespace collectorforopenshift ``` If the pods aren’t running, see [Troubleshooting](/content/docs/monitoring-openshift/troubleshooting/index.html). Once the images are pulled and the pods are `Running`, open the Monitoring OpenShift app in Splunk - dashboards should start populating within a minute or two. By default, Collectord forwards container logs, host logs (including syslog), and metrics for hosts, pods, containers, and processes. ## Next steps - [Integrate the OpenShift Web Console with the Monitoring OpenShift application](/content/blog/2018-10-26-openshift-webconsole-integration/index.html). - Review the predefined [alerts](/content/docs/monitoring-openshift/alerts/index.html) and enable the ones relevant to your environment. - If something looks off, work through the [troubleshooting](/content/docs/monitoring-openshift/troubleshooting/index.html) checks. - Enable [Audit Logs](/content/docs/monitoring-openshift/audit/index.html). OpenShift doesn’t write API server audit logs out of the box; you’ll need to enable them on the API server before Collectord can forward them. - Verify [Prometheus Metrics](/content/docs/monitoring-openshift/prometheus/index.html). The default configuration covers most clusters, but if Control Plane metrics are missing in the dashboards, double-check that the Prometheus endpoints are reachable from Collectord. - For better indexing performance, split logs and metrics into separate indexes. See [Splunk Indexes](/content/docs/monitoring-openshift/splunk-indexes/index.html) for the recommended layout and how to wire it up on both sides. - For search-time field extraction on container logs, see [Splunk fields extraction for container logs](/content/docs/monitoring-openshift/splunk-fields-extraction/index.html). - For per-pod control over log forwarding - custom log file paths, multi-line patterns, field extraction, index/source/sourcetype overrides, dropping noisy lines, and hashing sensitive values - see [annotations](/content/docs/monitoring-openshift/annotations/index.html).