Outcold Solutions is sponsoring Splunk .conf26 - see you there!

# Monitoring Docker

## Splunk indexes

Out of the box, `collectorfordocker` writes everything to the default index of your HTTP Event Collector token. Each token has an allow-list of indexes it can write to, and one of those is treated as the default when the sender doesn’t pick one. The Monitoring Docker app, in turn, assumes those indexes are searchable by default for your Splunk role - `main` is the obvious example.

If you forward to an index that isn’t searchable by default for your role, the dashboards will come up empty even though the data is being indexed.

You have two ways to fix that. The first is to add the index to _Indexes searched by default_ for your role under _Settings_ \- _Access Control_ \- _Roles_:

The second is to update the search macros the app uses, so the dashboards search your indexes explicitly. You’ll find them in the Splunk Web UI under _Settings_ \- _Advanced search_ \- _Search macros_, or you can override `$SPLUNK_HOME/etc/apps/monitoring-docker/default/macros.conf` with a `local/macros.conf`:

Since version 5.10, every other macro inherits from a single base macro `macro_docker_base` \- so you typically only need to set the index list once:

```text
macro_docker_base = (index=docker_stats OR index=docker_logs)
```

If you want finer-grained control - for example, pinning a specific datatype to a specific index and sourcetype - override the individual macro instead:

```text
macro_docker_stats = (index=docker_stats sourcetype=docker_stats)
```

The macros worth knowing about:

- `macro_docker_events` \- all the docker events.
- `macro_docker_host_logs` \- host logs.
- `macro_docker_logs` \- container logs.
- `macro_docker_proc_stats` \- proc metrics.
- `macro_docker_net_stats` \- network metrics.
- `macro_docker_net_socket_table` \- network socket tables.
- `macro_docker_mount_stats` \- container runtime storage usage metrics.
- `macro_docker_stats` \- system and container metrics.

## Using dedicated indexes for different types of data

For most hosts, it’s worth splitting logs from metrics - they have different access patterns, different volumes, and different retention needs. A common layout is `docker_logs` for events, container logs, and host logs and `docker_stats` for proc and system metrics. You can go further and pin every datatype Collectord forwards to its own index if you want.

The big practical win: dedicated indexes let you set different retention policies for logs and metrics independently.

Override the destination index per input in the Collectord configuration - see [Collectord Configuration](/content/docs/monitoring-docker/configuration/index.html) for how to apply changes:

```ini
[input.system_stats]
index = docker_stats

[input.proc_stats]
index = docker_stats

[input.net_stats]
index = docker_stats

[input.net_socket_table]
index = docker_stats

[input.mount_stats]
index = docker_stats

[input.files]
index = docker_logs

[input.app_logs]
index = docker_logs

[input.files::syslog]
index = docker_logs

[input.files::logs]
index = docker_logs

[input.docker_events]
index = docker_logs
```