Outcold Solutions is sponsoring Splunk .conf26 - see you there! Syslog (QRadar) # Concepts A short orientation to how the syslog forwarder actually works. Read this once and the rest of the docs will make sense without piecing things together from installation, configuration, and annotations all at once. ## How it’s structured This is a forwarder-only product - there’s no app on the receiving end. The destination is your existing SIEM (QRadar, Microsoft Sentinel, or anything else that accepts RFC 5424 syslog). - **Collectord** \- the agent. Same binary as the Splunk and ElasticSearch products; only the output is different. Runs as two DaemonSets on every node (one for control-plane / master nodes, one for workers), plus a single Deployment add-on that watches the Kubernetes API for events. - **Syslog** \- the destination. Collectord opens a TCP or UDP connection to your syslog server and sends each event as a structured RFC 5424 message. Multiple endpoints can be configured for load-balancing and failover. The product ships in two flavors with parallel directory structure - one for Kubernetes (`collectorforkubernetes-syslog`) and one for OpenShift (`collectorforopenshift-syslog`, plus a Red Hat-certified variant). Both deploy into their own namespace so you can run them alongside any other Collectord instance on the same cluster. ## What Collectord forwards Collectord forwards logs and events; it does **not** forward metrics or Prometheus data through this integration. The syslog product is logs-focused. The categories you’ll see arrive at your syslog server: - **Logs** \- container `stdout` / `stderr`, application logs from files inside the container or on a mounted volume, host logs (syslog, journald). - **Audit logs** \- on OpenShift, `[input.files::audit-logs]` is enabled by default and tails the API server audit log path. On Kubernetes, you have to enable audit logging on the API server first and then add a `[input.files]` stanza pointing at the audit log path. - **Events** \- Kubernetes / OpenShift API events. Object watching (Pods, Deployments, etc.) is **not** enabled by default. You can opt in by adding `[input.kubernetes_watch::*]` stanzas to the addon ConfigMap if you want object specs to flow over syslog too - keep in mind that syslog isn’t an ideal transport for large structured payloads. ## Inputs, outputs, and pipes Internally, Collectord is a pipeline of three concepts: - An **input** discovers data - `[input.files]` walks `/var/log`, `[input.kubernetes_events]` watches the API server, `[input.app_logs]` picks up application log files exposed via annotations. - A **pipe** transforms events on the way through - replace, hash, extract, override, sample, throttle. - An **output** ships the result somewhere - `[output.syslog]` is the default; `devnull` exists for dropping data. Most of [annotations](/content/docs/syslog-kubernetes/annotations/index.html) is just “configure pipes and the destination output, scoped to one pod or namespace.” ## Annotation prefix This product sets `annotationsSubdomain = syslog`, so all annotations use the `syslog.collectord.io/` prefix instead of plain `collectord.io/`. This is so you can run a Splunk-output Collectord and a syslog-output Collectord on the same cluster without their annotations colliding - each only reads its own namespace. If you want an annotation to apply to **every** Collectord instance regardless of subdomain, use the `collectord.collectord.io/` prefix. ## Where configuration comes from The same setting can be specified in several places. The general layering, highest priority first: 1. **Pod annotations** \- most specific. 2. **Workload annotations** \- annotations on the Deployment / DeploymentConfig / StatefulSet / DaemonSet propagate to its pods. 3. **Namespace annotations** \- apply to every pod in the namespace. 4. **CRD `Configuration`** \- cluster-level defaults applied by selector. 5. **ConfigMap files** \- `001-general.conf` and the role-specific files (`002-daemonset.conf`, `003-daemonset-master.conf`, `004-addon.conf`) mounted into the relevant Collectord pods. The lowest-priority layer. A CRD `Configuration` can be marked with `force: true` to override pod, workload, and namespace annotations - see [Annotations → Forcing Cluster Level Annotations](/content/docs/syslog-kubernetes/annotations/#forcing-cluster-level-annotations/index.html). When the same setting is defined at multiple levels and the wrong value is winning, run `collectord describe` from inside a Collectord pod - see [Troubleshooting → Describe](/content/docs/syslog-kubernetes/troubleshooting/#describe-command/index.html). ## Auto-discovery vs explicit opt-in Some things Collectord finds on its own; others require you to point at them: | Picked up automatically | Requires annotation or config | | --- | --- | | Container `stdout` / `stderr` | Application log files inside containers | | Host logs (syslog, journald) | Files on a PVC shared by multiple pods | | Kubernetes / OpenShift events | API objects (Pods, Deployments, ConfigMaps, …) | | OpenShift audit logs (default `[input.files::audit-logs]`) | Custom field extractions on container logs | | | Kubernetes audit logs (must be enabled at the API server first, then add a `[input.files]` pointing at the audit log path) | If something you expect isn’t reaching your SIEM, the first question is usually: was this auto-discovered, or do I need to point Collectord at it? ## How data flows ```text 1Your container ─┐ 2 ├─► [input] ──► [pipe] ──► [output.syslog] ──► RFC 5424 messages ──► Your SIEM 3Your annotation ┘ ``` Collectord runs on every node, reads from local sources (filesystem, the API server) and pushes RFC 5424 messages over TCP or UDP to your syslog server. Anything beyond that - parsing, indexing, dashboards - is up to your SIEM. ## Where to go next - New cluster: [Installation](/content/docs/syslog-kubernetes/installation/index.html). - Tuning what gets forwarded for a specific workload: [Annotations](/content/docs/syslog-kubernetes/annotations/index.html). - Something looks wrong: [Troubleshooting](/content/docs/syslog-kubernetes/troubleshooting/index.html).