manage/docs/observability-runbooks.md

Observability Runbooks

Operational playbooks for the standalone example observability stack (docker-compose.observability.yml) that can be deployed alongside Manage. Manage itself does not deploy these services; it connects to existing Grafana / Prometheus / Alertmanager instances. These runbooks cover operating the standalone stack shipped under monitoring/.

Service Overview

Service	Compose name	Internal URL	Health check
Prometheus	prometheus	http://prometheus:9090	/-/healthy
Grafana	grafana	http://grafana:3000	/api/health
Loki	loki	http://loki:3100	/ready
Alloy	alloy	http://alloy:12345	/-/healthy
Alertmanager	alertmanager	http://alertmanager:9093	/-/healthy
Node Exporter	node-exporter	http://node-exporter:9100	/
Manage backend	backend	http://backend:8000	/api/health

---

Alert: BackupJobFailed

Severity: critical
Meaning: A backup job reported status=failed within the last hour.

Steps

1. Open Manage → Backups and identify the failed job/run.
2. Check the run output / logs for the failure reason.
3. Search Loki for {container="backend"} | json | message=~"(?i)backup" around the failure time.
4. If transient (network, lock file), retry the job.
5. If persistent, open a task to fix the backup script or credentials.

---

Alert: BackupJobStuck

Severity: warning
Meaning: No successful backup run has been recorded for a job in the last 24 hours.

Steps

1. Confirm the job is still scheduled and expected to run.
2. Check whether the backup scheduler/host is running.
3. Verify the job can still report success to POST /api/backups/reports.
4. Inspect Prometheus graph for manage_backup_runs_last_success_timestamp by job_name.
5. If the job was intentionally retired, remove or disable its reporting.

---

Alert: PrometheusTargetMissing

Severity: warning
Meaning: A Prometheus scrape target is down (up == 0) for more than 2 minutes.

Steps

1. Identify job and instance from the alert labels.
2. Check the container/process status:
   • docker compose ps <service>
   • docker compose logs --tail 100 <service>
3. Verify network reachability from the Prometheus container:
   • docker compose exec prometheus wget -qO- http://<instance>/
4. If the target is a remote Node Exporter:
   • Check the machine is reachable over SSH.
   • Verify Node Exporter is installed and running (systemctl status node_exporter).
   • Confirm the scrape host/port in Manage → Settings for that machine.
5. Restart if needed: docker compose restart <service>.

---

Alert: AlertmanagerDown

Severity: critical
Meaning: Prometheus cannot scrape Alertmanager; new alerts may not be delivered.

Steps

1. Check container status: docker compose ps alertmanager
2. Review logs: docker compose logs --tail 200 alertmanager
3. Validate config syntax:
   • docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml
4. Verify SMTP environment variables are present if using email receivers.
5. Restart: docker compose restart alertmanager

---

Alert: GrafanaDown

Severity: warning
Meaning: Grafana is unreachable; dashboards and iframe panels in Manage are unavailable.

Steps

1. Check container status and logs.
2. Verify the OAuth client configuration is correct (GF_AUTH_GENERIC_OAUTH_*).
3. If embedded panels are blank, confirm Grafana allow_embedding = true and cookie settings.
4. Restart: docker compose restart grafana

---

Routine Maintenance

Check overall health

cd /path/to/manage
docker compose ps
docker compose exec prometheus wget -qO- http://127.0.0.1:9090/-/healthy
docker compose exec grafana wget -qO- http://127.0.0.1:3000/api/health
docker compose exec loki wget -qO- http://127.0.0.1:3100/ready
docker compose exec alertmanager wget -qO- http://127.0.0.1:9093/-/healthy

Reload Prometheus after rule/config changes

Prometheus is started with --web.enable-lifecycle, so a SIGHUP or HTTP call reloads config:

curl -X POST http://localhost:9090/-/reload

Inspect logs

# All backend logs in Loki via Grafana Explore, or locally:
docker compose logs --tail 500 backend

# Specific service:
docker compose logs -f prometheus

Storage usage

docker system df -v
docker compose exec prometheus du -sh /prometheus
docker compose exec loki du -sh /loki
docker compose exec grafana du -sh /var/lib/grafana

---

Backup and Disaster Recovery

The observability data lives on the host under OBSERVABILITY_DATA_ROOT (./observability-data by default). Subdirectories are created for each service:

• prometheus
• loki
• grafana
• alertmanager
• alloy

Backup data

# Stop the stack to ensure consistency
docker compose -f docker-compose.observability.yml down

# Back up the whole data directory
rsync -aP --delete "$OBSERVABILITY_DATA_ROOT" /mnt/backups/observability-data/

# Start the stack again
docker compose -f docker-compose.observability.yml up -d

Restore data

docker compose -f docker-compose.observability.yml down
rm -rf "$OBSERVABILITY_DATA_ROOT"
rsync -aP /mnt/backups/observability-data/ "$OBSERVABILITY_DATA_ROOT"
docker compose -f docker-compose.observability.yml up -d

---

Standalone Observability Stack

Run the observability services without the Manage backend or frontend:

cd /path/to/manage
# create an env file with at least the required variables
cat > .env.observability <<EOF
CERT_RESOLVER=myresolver

PROMETHEUS_ROOT=/var/lib/manage/observability/prometheus
LOKI_ROOT=/var/lib/manage/observability/loki
ALLOY_ROOT=/var/lib/manage/observability/alloy
GRAFANA_ROOT=/var/lib/manage/observability/grafana
ALERTMANAGER_ROOT=/var/lib/manage/observability/alertmanager

GRAFANA_APP_HOST=grafana.example.com
PROMETHEUS_APP_HOST=prometheus.example.com
ALERTMANAGER_APP_HOST=alertmanager.example.com
EOF

# prepare config directories from the repository defaults
mkdir -p "$PROMETHEUS_ROOT"/{config,data} "$LOKI_ROOT"/{config,data} "$ALLOY_ROOT"/{config,data} "$GRAFANA_ROOT"/{config,data} "$ALERTMANAGER_ROOT"/{config,data}
cp monitoring/prometheus/prometheus.standalone.yml "$PROMETHEUS_ROOT/config/prometheus.yml"
cp -r monitoring/prometheus/rules "$PROMETHEUS_ROOT/config/rules"
cp -r monitoring/prometheus/file-sd "$PROMETHEUS_ROOT/config/file-sd"
cp monitoring/loki/loki.yml "$LOKI_ROOT/config/loki.yml"
cp monitoring/alloy/config.alloy "$ALLOY_ROOT/config/config.alloy"
cp monitoring/grafana/grafana.ini "$GRAFANA_ROOT/config/grafana.ini"
cp -r monitoring/grafana/provisioning "$GRAFANA_ROOT/config/provisioning"
cp monitoring/alertmanager/alertmanager.yml "$ALERTMANAGER_ROOT/config/alertmanager.yml"

docker compose -f docker-compose.observability.yml --env-file .env.observability up -d

Each service root must contain config/ (read-only config files) and data/ (runtime state). If you do not use Traefik, set CERT_RESOLVER to any non-empty value and do not attach the services to a web network. The direct host ports still work without Traefik.

Reachable web UIs

Only three services expose a human-facing web interface. With Traefik they are served on their public hostnames; direct ports are still open on localhost for debugging.

Service	Has UI	Direct URL	Traefik hostname variable	Notes
Grafana	yes	http://localhost:3000	GRAFANA_APP_HOST	Dashboards, log explore, alert management
Prometheus	yes	http://localhost:9090	PROMETHEUS_APP_HOST	Query, targets, alerts, config status
Alertmanager	yes	http://localhost:9093	ALERTMANAGER_APP_HOST	Alerts, silences, routing status
Loki	no	http://localhost:3100	none	Log API only; browse logs through Grafana
Alloy	partial	http://localhost:12345	none	Agent debug UI for pipeline inspection
Node Exporter	no	http://localhost:9100	none	Metrics endpoint only (/metrics)

Grafana defaults to admin / admin. Datasources and dashboards are provisioned automatically.

Environment variables

Variable	Default	Purpose
PROMETHEUS_ROOT	required	Absolute host directory for Prometheus config and data. Must contain config/ and data/.
LOKI_ROOT	required	Absolute host directory for Loki config and data. Must contain config/ and data/.
ALLOY_ROOT	required	Absolute host directory for Alloy config and data. Must contain config/ and data/.
GRAFANA_ROOT	required	Absolute host directory for Grafana config and data. Must contain config/ and data/.
ALERTMANAGER_ROOT	required	Absolute host directory for Alertmanager config and data. Must contain config/ and data/.
CERT_RESOLVER	required	Traefik certificate resolver name (for example letsencrypt or cloudflare). Must be set before deploy.
TRAEFIK_ENTRYPOINT	websecure	Traefik entrypoint to use for the web UIs.
GRAFANA_APP_HOST	required	Public hostname for Grafana (for example grafana.example.com).
GRAFANA_APP_NAME	grafana	Traefik router/service name for Grafana.
GRAFANA_APP_PORT	3000	Internal port Traefik forwards to for Grafana.
PROMETHEUS_APP_HOST	required	Public hostname for Prometheus (for example prometheus.example.com).
PROMETHEUS_APP_NAME	prometheus	Traefik router/service name for Prometheus.
PROMETHEUS_APP_PORT	9090	Internal port Traefik forwards to for Prometheus.
ALERTMANAGER_APP_HOST	required	Public hostname for Alertmanager (for example alertmanager.example.com).
ALERTMANAGER_APP_NAME	alertmanager	Traefik router/service name for Alertmanager.
ALERTMANAGER_APP_PORT	9093	Internal port Traefik forwards to for Alertmanager.
PROMETHEUS_PORT	9090	Direct host port for Prometheus web UI and API.
LOKI_PORT	3100	Direct host port for Loki API.
ALLOY_PORT	12345	Direct host port for Alloy debug UI.
GRAFANA_PORT	3000	Direct host port for Grafana web UI.
ALERTMANAGER_PORT	9093	Direct host port for Alertmanager web UI.
NODE_EXPORTER_PORT	9100	Direct host port for Node Exporter metrics endpoint.
GRAFANA_ADMIN_USER	admin	Grafana admin username.
GRAFANA_ADMIN_PASSWORD	admin	Grafana admin password. Change this in production.
GF_AUTH_GENERIC_OAUTH_CLIENT_ID	empty	Generic OAuth client ID for Authentik or another provider.
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET	empty	Generic OAuth client secret.
GF_AUTH_GENERIC_OAUTH_AUTH_URL	empty	OAuth authorization endpoint.
GF_AUTH_GENERIC_OAUTH_TOKEN_URL	empty	OAuth token endpoint.
GF_AUTH_GENERIC_OAUTH_API_URL	empty	OAuth userinfo endpoint.
LOG_LEVEL	INFO	Grafana log level.
SMTP_HOST	smtp.example.com	SMTP host for Alertmanager email notifications.
SMTP_PORT	587	SMTP port for Alertmanager.
SMTP_USERNAME	empty	SMTP username.
SMTP_PASSWORD	empty	SMTP password.
SMTP_FROM_ADDRESS	no-reply@example.com	From address for alert emails.
ALERT_EMAIL_TO	admin@example.com	Default recipient for alert emails.

Prometheus and Alertmanager do not have authentication. When exposing them through Traefik, add a basic-auth middleware or restrict access by network.

To scrape a Manage backend from this standalone stack, edit monitoring/prometheus/prometheus.standalone.yml and add a static target for the backend's /metrics endpoint, or drop a file-SD JSON file into monitoring/prometheus/file-sd/.

Backing up standalone data

Because config and data are stored on the host under each service root, you can back them up with normal filesystem tools:

for svc in prometheus loki alloy grafana alertmanager; do
  rsync -aP --delete "/var/lib/manage/observability/$svc" "/mnt/backups/observability/$svc"
done

Stop the stack first if you need a consistent snapshot.

Scaling Notes

• The current deploy.resources blocks are tuned for a small homelab. Raise memory limits if you monitor many machines or retain logs longer than 30 days.
• Loki is configured for single-node filesystem storage. For larger deployments, migrate to object storage (S3/GCS/MinIO) and a shared index.
• Prometheus remote-write or Thanos/Cortex can be added later for long-term metrics without changing application instrumentation.