Outcold Solutions is sponsoring Splunk .conf26 - see you there!

Monitoring OpenShift

Audit logs

Learn how to configure and forward OpenShift audit logs for monitoring

OpenShift 4.x enables audit logging by default - no extra configuration needed. The rest of this page applies to OpenShift 3.x.

The Monitoring OpenShift app ships dashboards built specifically for API server audit data - who did what, against which resource, and how the API responded. On OpenShift 3.x, audit logs aren’t on by default, so you’ll need to enable them at the master before Collectord can forward them. For background, see the OpenShift Master and Node Configuration / Advanced Audit guide.

Enable auditing on each master by editing master-config.yaml:

sudo vi /etc/origin/master/master-config.yaml

The block below keeps audit data for 10 days, capped at 3 rotated files of 100 MB each - adjust to fit your retention policy and disk budget:

auditConfig:
  auditFilePath: "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log"
  enabled: true
  maximumFileRetentionDays: 10
  maximumFileSizeMegabytes: 100
  maximumRetainedFiles: 3
  policyFile: "/etc/origin/master/audit-policy.yaml"
  logFormat: json

Collectord forwards anything written under /var/lib/origin/openpaas-oscp-audit/ automatically - no Collectord-side changes are required.

Older versions of this guide pointed at /var/log. Starting with OpenShift 3.10, /var/log is no longer mapped into the master API container, which is why the path moved to /var/lib/origin/openpaas-oscp-audit/.

Next, create the policy file:

sudo vi /etc/origin/master/audit-policy.yaml

The example below silences the noisiest system traffic - kube-system, openshift-infra, the SDN, the monitoring stack, Collectord itself - logs secret and configmap mutations at the metadata level, and catches everything else at the request level. Use it as a starting point.

For a more exhaustive policy, see the audit profile used by GCE.

apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # Do not log from kube-system accounts
  - level: None
    userGroups:
    - system:serviceaccounts:kube-system
    - system:serviceaccounts:openshift-infra
    - system:serviceaccounts:kube-service-catalog
    - system:serviceaccounts:openshift-template-service-broker
    - system:serviceaccounts:openshift-sdn
    - system:serviceaccounts:openshift-node
    - system:serviceaccounts:openshift-ansible-service-broker
    - system:serviceaccounts:openshift-monitoring
  - level: None
    users:
    - system:apiserver
    - system:kube-scheduler
    - system:volume-scheduler
    - system:kube-controller-manager
    - system:node
    - system:openshift-master

# Do not log from collector
  - level: None
    users:
    - system:serviceaccount:collectorforopenshift:collectorforopenshift

# Don't log nodes communications
  - level: None
    userGroups:
    - system:nodes

# Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
    - /healthz*
    - /version
    - /swagger*

# Log configmap and secret changes in all namespaces at the metadata level.
  - level: Metadata
    resources:
    - resources: ["secrets", "configmaps"]

# A catch-all rule to log all other requests at the request level.
  - level: Request

Restart the master services so the new configuration takes effect:

sudo /usr/local/bin/master-restart api
sudo /usr/local/bin/master-restart controllers

On OpenShift 3.9 or older, restart with sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers instead.

The Splunk app finds audit events through the macro_openshift_audit_logs macro, which scopes searches to host logs containing the audit.k8s.io API group:

(`macro_openshift_host_logs` "audit.k8s.io")