Outcold Solutions is sponsoring Splunk .conf26 - see you there!
Monitoring Linux
Installation
This guide walks you through installing Monitoring Linux end-to-end: configuring the Splunk app and HTTP Event Collector, then installing Collectord on your Linux host as a systemd service to forward host logs (syslog, journald), host metrics, and process metrics. A typical install takes under 10 minutes. If you don’t have a license yet, you can request a 30-day evaluation.
Install the Monitoring Linux application
Install Monitoring Linux from Splunkbase on your Search Heads only.
If you’re using a dedicated index that isn’t searchable by default, update the macro_linux_base macro to include it:
macro_linux_base = (index=linux)
Enable HTTP Event Collector in Splunk
Collectord forwards data to Splunk over the HTTP Event Collector (HEC). If HEC isn’t enabled yet, follow Splunk’s HTTP Event Collector walkthrough.
The minimum requirement is Splunk Enterprise or Splunk Cloud 6.5. If you’re managing Splunk Clusters older than 6.5, see our FAQ on setting up a Heavy Weight Forwarder in between.
Once HEC is enabled, you need two pieces of information for the rest of this guide: the HEC endpoint URL and an HEC token. You can verify both with curl:
$ curl -k https://hec.example.com:8088/services/collector/event/1.0 -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}
-kskips certificate validation; use it only for self-signed certificates.
Splunk Cloud uses a different HEC URL than Splunk Web - see Send data to HTTP Event Collector on Splunk Cloud instances.
Install Collectord for Linux
Download collectorforlinux.tar.gz and extract it into /opt/collectorforlinux. The archive contains builds for both amd64 and aarch64 architectures.
sudo curl -O /docs/monitoring-linux/builds/5.21.410/collectorforlinux.tar.gz -o /tmp/collectorforlinux.tar.gz
sudo mkdir -p /opt/collectorforlinux
sudo tar -xvf /tmp/collectorforlinux.tar.gz -C /opt/collectorforlinux
Open /opt/collectorforlinux/etc/002-user.conf with your editor:
sudo edit /opt/collectorforlinux/etc/002-user.conf
This file holds your overrides for the Collectord defaults. The full default configuration lives in /opt/collectorforlinux/etc/002-general.conf - refer to it when you need to know what options exist.
In 002-user.conf, set the Splunk HEC URL and token, review and accept the license agreement, and paste in your license key (request an evaluation key with this automated form). Naming the cluster is optional but useful when you’re monitoring more than one host group and want to filter by cluster in the app.
[general]
acceptLicense = true
license = ...
fields.linux_cluster = dev
[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
insecure = true
You can run collectorforlinux directly from the terminal to confirm the configuration is valid:
sudo /opt/collectorforlinux/bin/collectorforlinux
Install the collectorforlinux service with systemd
The package ships a systemd unit you can link with systemctl and run as a background daemon:
sudo systemctl link /opt/collectorforlinux/bin/collectorforlinux.service
sudo systemctl daemon-reload
sudo systemctl enable collectorforlinux
sudo systemctl start collectorforlinux
Tail the logs to confirm Collectord is running:
sudo journalctl -fu collectorforlinux
Next steps
- Review the predefined alerts and enable the ones relevant to your environment.
- If something looks off, work through the troubleshooting checks.
- Configure log forwarding from custom locations beyond
/var/logand journald.