Lookout is a small monitoring agent for Linux servers. It runs on the machine you care about, watches the basics, and sends an alert when something crosses a line or stops reporting.
It is meant for the kind of server where you want useful alerts without running a full Prometheus stack. One static binary, one YAML file, one systemd service.
Lookout collects from the local machine and evaluates alerts in-process.
| Area | What Lookout checks |
|---|---|
| Memory | Used memory percentage, based on /proc/meminfo and MemAvailable |
| Swap | Used swap percentage |
| Disk | Used percentage for configured mount points |
| Disk growth | Whether a disk is on pace to fill within a configured window |
| Load | 1-minute load average divided by CPU cores |
| CPU | CPU usage percentage from /proc/stat |
| Resource pressure | CPU contention and full memory/I/O stalls from Linux PSI |
| systemd | Whether named services are active |
| HTTP | Whether configured URLs return the expected status code |
| TCP | Whether configured TCP addresses accept a connection |
| Processes | Whether named processes are present in /proc |
| Docker | Optional container exit, OOM, restart-loop, and healthcheck alerts |
| Heartbeat | Optional dead-man's-switch ping to a service like healthchecks.io |
Alerts can go to Google Chat, Discord, Slack, Microsoft Teams, Telegram, PagerDuty, a generic webhook, or email. If no notifier is configured, alerts are printed to stdout, which means they land in the systemd journal when installed as a service.
On a Linux server with systemd:
curl -fsSL https://raw.githubusercontent.com/AmoabaKelvin/lookout/main/install.sh | sudo shThe installer:
- downloads the latest GitHub release for your CPU architecture
- verifies the release checksum
- installs the binary to
/usr/local/bin/lookout - creates an unprivileged
lookoutsystem user - writes
/etc/lookout/config.yamlif it does not already exist - installs and starts
lookout.service
To pin a specific release:
curl -fsSL https://raw.githubusercontent.com/AmoabaKelvin/lookout/main/install.sh | sudo VERSION=v1.0.0 shRe-running the installer upgrades the binary and keeps your existing config.
The installed config lives at:
/etc/lookout/config.yamlAfter changing it, restart the service:
sudo systemctl restart lookoutA minimal useful config might look like this:
collection_interval: 30s
state_file: /var/lib/lookout/state.json
metrics:
enabled: false
listen: "127.0.0.1:9100"
alerts:
renotify_after: 1h
stale_after: 90s
memory:
threshold: 85
resolve_below: 80
for: 2m
severity: warning
disk:
threshold: 85
resolve_below: 80
for: 2m
severity: warning
predict_full_within: 4h
mounts:
- /
load:
threshold: 1.5
resolve_below: 1.0
for: 2m
severity: warning
cpu:
threshold: 85
resolve_below: 80
for: 2m
severity: warning
pressure:
# Requires Linux PSI. Kept opt-in while thresholds are evaluated on real workloads.
enabled: false
cpu:
threshold: 20
resolve_below: 15
for: 2m
severity: warning
memory:
threshold: 5
resolve_below: 3
for: 1m
severity: critical
io:
threshold: 10
resolve_below: 5
for: 2m
severity: warning
swap:
threshold: 80
resolve_below: 75
for: 2m
severity: warning
systemd:
severity: critical
services:
- nginx
- postgresql
http:
severity: critical
checks:
- name: app
url: "https://example.com/health"
timeout: 5s
expected_status: 200
tcp:
severity: critical
checks:
- name: redis
address: "127.0.0.1:6379"
timeout: 5s
process:
severity: critical
names:
- nginx
notifiers:
slack:
webhook_url: "https://hooks.slack.com/services/XXX/YYY/ZZZ"
heartbeat:
interval: 60s
docker:
enabled: false
severity: critical
restart_threshold: 3
restart_window: 10mThe full example config is in
deploy/config.example.yaml.
For threshold checks, threshold is the point where an alert starts and
resolve_below is the point where it clears. Keeping those separate prevents
alerts from firing and resolving over and over when a value sits near the line.
The for field controls how long a value must stay over the threshold before
Lookout sends the first alert. Set it to 0s if you want immediate alerts.
renotify_after controls repeat notifications while a problem is still firing.
stale_after controls alerts for metrics that stop reporting. By default, it is
three times the collection interval.
Durations use Go-style strings such as 5s, 2m, and 1h.
Linux Pressure Stall Information (PSI) measures time in which work cannot make
progress because CPU, memory, or I/O is contended. This complements utilization:
a CPU can be fully utilized without delaying work, while memory reclaim can
stall an application even when host-wide MemAvailable still looks healthy.
Pressure alerts are opt-in. They require readable cpu, memory, and io
files under /proc/pressure (normally Linux 4.20 or newer with CONFIG_PSI
enabled; some systems also require the psi=1 kernel command-line option).
Lookout refuses to start if pressure alerts are enabled but those signals are
unavailable, rather than silently leaving the alerts inactive.
The configured thresholds are percentages of wall time lost between Lookout collection passes:
cpuuses PSIsome: at least one runnable task was waiting for CPU time.memoryuses PSIfull: all non-idle work was stalled on memory.iouses PSIfull: all non-idle work was stalled on I/O.
Lookout also exports the some and full kernel averages over 10, 60, and 300
seconds when the Prometheus endpoint is enabled. System-wide CPU full is not
used because the kernel defines it as unavailable and reports it as zero.
The example thresholds are starting points, not universal limits. Enable PSI on
one server first and compare pressure against application latency and throughput
before rolling the alert thresholds out broadly. A reproducible CPU and memory
disagreement study is described in
docs/psi-evaluation.md.
Lookout can expose the latest collected values as Prometheus text at
GET /metrics. This is disabled by default and keeps only the current in-memory
snapshot; it does not add storage, retention, dashboards, or a query layer.
metrics:
enabled: true
listen: "127.0.0.1:9100"Keep the listener on localhost or a private interface unless you place it behind a firewall or reverse proxy.
Severity can be warning or critical. It affects the label, color, and
PagerDuty severity used for the alert.
Lookout sends an alert to every notifier you configure. Set min_severity on a
notifier to limit what it receives: critical sends only critical alerts, and
warning (the default) sends everything. This lets you page only on criticals
while warnings go to a chat channel. A resolved alert keeps its rule's
severity, so it reaches the same channels as the alert that fired.
notifiers:
google_chat:
webhook_url: "https://chat.googleapis.com/v1/spaces/XXX/messages?key=...&token=..."
discord:
webhook_url: "https://discord.com/api/webhooks/XXX/YYY"
slack:
webhook_url: "https://hooks.slack.com/services/XXX/YYY/ZZZ"
teams:
webhook_url: "https://example.webhook.office.com/..."
telegram:
bot_token: "123456:ABC..."
chat_id: "987654321"
pagerduty:
integration_key: "pagerduty-events-api-v2-key"
min_severity: critical
webhook:
webhook_url: "https://example.com/hooks/lookout"
email:
host: "smtp.example.com"
port: 587
implicit_tls: false
username: ""
password: ""
from: "lookout@example.com"
to:
- "you@example.com"The generic webhook receives structured JSON with the alert status, severity, title, metric, value, threshold, unit, hostname, color, and rendered text.
Webhook URLs are redacted in logs so tokens in paths or query strings are not printed on send failures.
Docker monitoring is off by default.
docker:
enabled: true
severity: critical
restart_threshold: 3
restart_window: 10mWhen enabled, Lookout listens to Docker container events and alerts on:
- non-zero container exits
- containers killed by OOM
- restart loops
- unhealthy Docker healthchecks
Intentional stops are ignored. Quick restarts are debounced so a normal restart does not immediately look like a failed container.
The service user must be able to read Docker events. On many systems that means
adding it to the docker group and restarting the service:
sudo usermod -aG docker lookout
sudo systemctl restart lookoutLookout can ping a dead-man's-switch URL on a fixed interval:
heartbeat:
url: "https://hc-ping.com/your-uuid"
interval: 60sThis is useful for catching the cases where the whole server, network, or Lookout service disappears and therefore cannot send its own alert.
Check status:
sudo systemctl status lookoutWatch logs:
sudo journalctl -u lookout -fRestart after changing config:
sudo systemctl restart lookoutRun manually with a specific config:
sudo lookout --config /etc/lookout/config.yamlRemove the service and binary, but keep config and the service user:
curl -fsSL https://raw.githubusercontent.com/AmoabaKelvin/lookout/main/uninstall.sh | sudo shRemove the service, binary, config directory, and service user:
curl -fsSL https://raw.githubusercontent.com/AmoabaKelvin/lookout/main/uninstall.sh | sudo sh -s -- --purgeYou need Go installed.
Run the tests:
go test ./...Build a local binary:
go build -o lookout .Build release artifacts for Linux amd64 and arm64:
sh build.shArtifacts are written to dist/, along with checksums.txt.
Tagged releases are built by GitHub Actions when a tag like v1.0.0 is pushed.
Lookout is Linux-first. The collectors read Linux kernel files such as
/proc/meminfo, /proc/loadavg, /proc/stat, and /proc/mounts, and the
installer creates a systemd service.
The agent stores alert state under /var/lib/lookout by default. That state is
what lets it remember firing alerts across restarts and send clean recovery
messages instead of starting from scratch every time the service restarts.