Troubleshooting

On OpenShift, swap kubectl for oc and use the collectorforopenshift-syslog namespace.

Verify configuration

When something looks off, the first thing to do is run collectord verify from inside a Collectord pod. It checks the configuration end-to-end - license, syslog output, container runtime, file inputs - and reports each item as OK or FAILED.

Start by listing the Collectord pods:

$ kubectl get pods -n collectorforkubernetes-syslog
NAME                                           READY     STATUS    RESTARTS   AGE
collectorforkubernetes-syslog-addon-857fccb8b9-t9qgq   1/1       Running   1          1h
collectorforkubernetes-syslog-master-bwmwr             1/1       Running   0          1h
collectorforkubernetes-syslog-xbnaa                    1/1       Running   0          1h

Collectord runs as three workloads - a DaemonSet on master nodes (collectorforkubernetes-syslog-master), a DaemonSet on the rest of the nodes (collectorforkubernetes-syslog), and a single Deployment add-on (collectorforkubernetes-syslog-addon). Run verify against one pod from each so every code path is exercised:

$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-addon-857fccb8b9-t9qgq -- /collectord verify
$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord verify
$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-xbnaa -- /collectord verify

Each command produces output similar to:

Version = 5.2.176
Build date = 181012
Environment = kubernetes

General:
+ conf: OK
+ db: OK
+ db-meta: OK
+ instanceID: OK
  instanceID = 2LEKCFD4KT4MUBIAQSUG7GRSAG
+ license load: OK
  trial
+ license expiration: OK
  license expires 2018-11-12 15:51:18.200772266 -0500 EST
+ license connection: OK

Kubernetes configuration:
+ api: OK
+ pod cgroup: OK
  pods = 18
+ container cgroup: OK
  containers = 39
+ volumes root: OK
+ runtime: OK
  docker

Docker configuration:
+ connect: OK
  containers = 43
+ path: OK
+ cgroup: OK
  containers = 40
+ files: OK

CRI-O configuration:
- ignored: OK
  kubernetes uses other container runtime

File Inputs:
 x input(syslog): FAILED
  no matches
+ input(logs): OK
  path /rootfs/var/log/

Errors: 1

The total number of errors appears at the bottom. Not every failure is a real problem - some are expected on smaller or non-standard clusters. The example above is from minikube, where this failure is benign:

input(syslog) - minikube doesn’t persist syslog to disk, so those logs aren’t available.

If you fix a real configuration error, kubectl apply -f ./collectorforkubernetes-syslog.yaml won’t restart the running pods. Delete them so the workloads recreate them with the new config: kubectl delete pods --all -n collectorforkubernetes-syslog.

Describe command

When the same setting can be defined on a pod, its workload, the namespace, a CRD Configuration, and the ConfigMap, it can be hard to track which value Collectord is actually using. The collectord describe command resolves a pod’s effective configuration and prints every field. Run it from inside any Collectord pod:

kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres

Available since Collectord version 26.04

Starting with version 26.04, describe also tags each resolved field with its origin in square brackets:

[pod] - the value comes from a pod annotation
[namespace] - the value comes from a namespace annotation
[configuration:<name>] - the value comes from a Collectord CRD Configuration resource (the <name> matches the resource name)

This makes it easy to trace which level of the configuration hierarchy is winning when the same syslog.collectord.io/ annotation is defined at multiple levels - for example, when a CRD-level default is being overridden by a pod-level annotation, or when a namespace annotation is unexpectedly routing logs to a different output:

$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-fqhmv -- /collectord describe --namespace webportal --pod audit-logger-774675c89c-rpfwx | grep '\['
logs-type [pod] = audit_logs
volume.1-logs-name [pod] = data
volume.1-logs-glob [pod] = *.log

Especially useful when debugging why a pod is routing to an unexpected output, using the wrong sourcetype, or picking up a field extraction you didn’t expect.

Collect diagnostic information

When you open a support case, attach a diagnostic bundle so we can reproduce the issue without a back-and-forth. The bundle includes performance, memory, and telemetry metrics, host Linux information, and the Collectord configuration - sensitive output settings are stripped out.

Run all four steps below.

1. Collect diagnostics information run following command

Pick any Collectord pod and run collectord diag. The command takes a few minutes:

kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord diag --stream 1>diag.tar.gz

You can extract the archive yourself to see exactly what’s in it - performance and memory profiles, basic telemetry metrics, host Linux info, and license metadata.

2. Collect logs

kubectl logs -n collectorforkubernetes-syslog --timestamps collectorforkubernetes-syslog-master-bwmwr  1>collectorforkubernetes.log 2>&1

3. Run verify

kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord verify > verify.log

4. Prepare tar archive

tar -czvf collectorforkubernetes-$(date +%s).tar.gz verify.log collectorforkubernetes.log diag.tar.gz