manage/docs/observability-runbooks.md

Observability Runbooks

Operational playbooks for the Manage self-hosted observability stack (Prometheus, Grafana, Loki, Alertmanager).

Service Overview

Service	Compose name	Internal URL	Health check
Prometheus	prometheus	http://prometheus:9090	/-/healthy
Grafana	grafana	http://grafana:3000	/api/health
Loki	loki	http://loki:3100	/ready
Alloy	alloy	http://alloy:12345	/-/healthy
Alertmanager	alertmanager	http://alertmanager:9093	/-/healthy
Node Exporter	node-exporter	http://node-exporter:9100	/
Manage backend	backend	http://backend:8000	/api/health

---

Alert: BackupJobFailed

Severity: critical
Meaning: A backup job reported status=failed within the last hour.

Steps

1. Open Manage → Backups and identify the failed job/run.
2. Check the run output / logs for the failure reason.
3. Search Loki for {container="backend"} | json | message=~"(?i)backup" around the failure time.
4. If transient (network, lock file), retry the job.
5. If persistent, open a task to fix the backup script or credentials.

---

Alert: BackupJobStuck

Severity: warning
Meaning: No successful backup run has been recorded for a job in the last 24 hours.

Steps

1. Confirm the job is still scheduled and expected to run.
2. Check whether the backup scheduler/host is running.
3. Verify the job can still report success to POST /api/backups/reports.
4. Inspect Prometheus graph for manage_backup_runs_last_success_timestamp by job_name.
5. If the job was intentionally retired, remove or disable its reporting.

---

Alert: PrometheusTargetMissing

Severity: warning
Meaning: A Prometheus scrape target is down (up == 0) for more than 2 minutes.

Steps

1. Identify job and instance from the alert labels.
2. Check the container/process status:
   • docker compose ps <service>
   • docker compose logs --tail 100 <service>
3. Verify network reachability from the Prometheus container:
   • docker compose exec prometheus wget -qO- http://<instance>/
4. If the target is a remote Node Exporter:
   • Check the machine is reachable over SSH.
   • Verify Node Exporter is installed and running (systemctl status node_exporter).
   • Confirm the scrape host/port in Manage → Settings for that machine.
5. Restart if needed: docker compose restart <service>.

---

Alert: AlertmanagerDown

Severity: critical
Meaning: Prometheus cannot scrape Alertmanager; new alerts may not be delivered.

Steps

1. Check container status: docker compose ps alertmanager
2. Review logs: docker compose logs --tail 200 alertmanager
3. Validate config syntax:
   • docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml
4. Verify SMTP environment variables are present if using email receivers.
5. Restart: docker compose restart alertmanager

---

Alert: GrafanaDown

Severity: warning
Meaning: Grafana is unreachable; dashboards and iframe panels in Manage are unavailable.

Steps

1. Check container status and logs.
2. Verify the OAuth client configuration is correct (GF_AUTH_GENERIC_OAUTH_*).
3. If embedded panels are blank, confirm Grafana allow_embedding = true and cookie settings.
4. Restart: docker compose restart grafana

---

Routine Maintenance

Check overall health

cd /path/to/manage
docker compose ps
docker compose exec prometheus wget -qO- http://127.0.0.1:9090/-/healthy
docker compose exec grafana wget -qO- http://127.0.0.1:3000/api/health
docker compose exec loki wget -qO- http://127.0.0.1:3100/ready
docker compose exec alertmanager wget -qO- http://127.0.0.1:9093/-/healthy

Reload Prometheus after rule/config changes

Prometheus is started with --web.enable-lifecycle, so a SIGHUP or HTTP call reloads config:

curl -X POST http://localhost:9090/-/reload

Inspect logs

# All backend logs in Loki via Grafana Explore, or locally:
docker compose logs --tail 500 backend

# Specific service:
docker compose logs -f prometheus

Storage usage

docker system df -v
docker compose exec prometheus du -sh /prometheus
docker compose exec loki du -sh /loki
docker compose exec grafana du -sh /var/lib/grafana

---

Backup and Disaster Recovery

The observability data lives in named volumes:

• prometheus_data
• loki_data
• grafana_data
• alertmanager_data

Backup volumes

# Stop the stack to ensure consistency
docker compose down

# Back up each volume to a tarball
docker run --rm -v manage_prometheus_data:/data -v $(pwd)/backups:/backups alpine \
  tar czf /backups/prometheus-$(date +%F).tar.gz -C /data .
docker run --rm -v manage_loki_data:/data -v $(pwd)/backups:/backups alpine \
  tar czf /backups/loki-$(date +%F).tar.gz -C /data .
docker run --rm -v manage_grafana_data:/data -v $(pwd)/backups:/backups alpine \
  tar czf /backups/grafana-$(date +%F).tar.gz -C /data .
docker run --rm -v manage_alertmanager_data:/data -v $(pwd)/backups:/backups alpine \
  tar czf /backups/alertmanager-$(date +%F).tar.gz -C /data .

# Start the stack again
docker compose up -d

Replace manage_ with your actual Docker Compose project name if different.

Restore a volume

docker compose down
docker volume rm manage_prometheus_data
docker volume create manage_prometheus_data
docker run --rm -v manage_prometheus_data:/data -v $(pwd)/backups:/backups alpine \
  tar xzf /backups/prometheus-YYYY-MM-DD.tar.gz -C /data
docker compose up -d

---

Scaling Notes

• The current deploy.resources blocks are tuned for a small homelab. Raise memory limits if you monitor many machines or retain logs longer than 30 days.
• Loki is configured for single-node filesystem storage. For larger deployments, migrate to object storage (S3/GCS/MinIO) and a shared index.
• Prometheus remote-write or Thanos/Cortex can be added later for long-term metrics without changing application instrumentation.