Outcold Solutions is sponsoring Splunk .conf26 - see you there!

# Monitoring OpenShift

## Audit logs

Learn how to configure and forward OpenShift audit logs for monitoring

> OpenShift 4.x enables audit logging by default - no extra configuration needed. The rest of this page applies to OpenShift 3.x.

The Monitoring OpenShift app ships dashboards built specifically for API server audit data - who did what, against which resource, and how the API responded. On OpenShift 3.x, audit logs aren’t on by default, so you’ll need to enable them at the master before Collectord can forward them. For background, see the OpenShift [Master and Node Configuration / Advanced Audit](https://docs.openshift.com/container-platform/3.11/install_config/master_node_configuration.html#master-node-config-advanced-audit) guide.

Enable auditing on each master by editing `master-config.yaml`:

```bash
sudo vi /etc/origin/master/master-config.yaml
```

The block below keeps audit data for 10 days, capped at 3 rotated files of 100 MB each - adjust to fit your retention policy and disk budget:

```yaml
auditConfig:
  auditFilePath: "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log"
  enabled: true
  maximumFileRetentionDays: 10
  maximumFileSizeMegabytes: 100
  maximumRetainedFiles: 3
  policyFile: "/etc/origin/master/audit-policy.yaml"
  logFormat: json
```

Collectord forwards anything written under `/var/lib/origin/openpaas-oscp-audit/` automatically - no Collectord-side changes are required.

> Older versions of this guide pointed at `/var/log`. Starting with OpenShift 3.10, `/var/log` is no longer mapped into the master API container, which is why the path moved to `/var/lib/origin/openpaas-oscp-audit/`.

Next, create the policy file:

```bash
sudo vi /etc/origin/master/audit-policy.yaml
```

The example below silences the noisiest system traffic - kube-system, openshift-infra, the SDN, the monitoring stack, Collectord itself - logs secret and configmap mutations at the metadata level, and catches everything else at the request level. Use it as a starting point.

> For a more exhaustive policy, see the [audit profile used by GCE](https://github.com/kubernetes/kubernetes/blob/0ef45b4fcf7697ea94b96d1a2fe1d9bffb692f3a/cluster/gce/gci/configure-helper.sh#L722-L862).

```yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # Do not log from kube-system accounts
  - level: None
    userGroups:
    - system:serviceaccounts:kube-system
    - system:serviceaccounts:openshift-infra
    - system:serviceaccounts:kube-service-catalog
    - system:serviceaccounts:openshift-template-service-broker
    - system:serviceaccounts:openshift-sdn
    - system:serviceaccounts:openshift-node
    - system:serviceaccounts:openshift-ansible-service-broker
    - system:serviceaccounts:openshift-monitoring
  - level: None
    users:
    - system:apiserver
    - system:kube-scheduler
    - system:volume-scheduler
    - system:kube-controller-manager
    - system:node
    - system:openshift-master

# Do not log from collector
  - level: None
    users:
    - system:serviceaccount:collectorforopenshift:collectorforopenshift

# Don't log nodes communications
  - level: None
    userGroups:
    - system:nodes

# Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
    - /healthz*
    - /version
    - /swagger*

# Log configmap and secret changes in all namespaces at the metadata level.
  - level: Metadata
    resources:
    - resources: ["secrets", "configmaps"]

# A catch-all rule to log all other requests at the request level.
  - level: Request
```

Restart the master services so the new configuration takes effect:

```bash
sudo /usr/local/bin/master-restart api
sudo /usr/local/bin/master-restart controllers
```

> On OpenShift 3.9 or older, restart with `sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers` instead.

The Splunk app finds audit events through the `macro_openshift_audit_logs` macro, which scopes searches to host logs containing the `audit.k8s.io` API group:

```text
(`macro_openshift_host_logs` "audit.k8s.io")
```