alex/manage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
45c295457a2851fd49eaea30e082b477c0f35641
manage/docs/observability-runbooks.md
T
Developer d4f95b64d4 chore(observability): externalize stack from root compose files 
Manage now connects to existing Grafana/Prometheus/Alertmanager instances
and never deploys its own stack.

- docker-compose.yml / docker-compose.dev.yml: removed prometheus, loki,
  alloy, grafana, alertmanager, node-exporter services, the monitoring
  network, and observability named volumes; they now ship only backend +
  frontend. Dev frontend now joins the web network so the Vite dev proxy
  can reach the backend.
- backend: alertmanager_url default is now empty; /api/monitoring/alerts
  and /alertmanager-status return graceful "not configured" responses
  when ALERTMANAGER_URL is unset. Added not-configured tests.
- docker-compose.observability.yml: kept as the optional standalone
  example; header clarifies Manage does not deploy it.
- Removed orphaned combined monitoring/prometheus/prometheus.yml
  (standalone stack uses prometheus.standalone.yml).
- Docs (README, REQUIREMENTS decision log, monitoring-logging-design,
  observability-runbooks, context.md, MIGRATION_PLAN, frontend/README,
  CHANGELOG) updated to the connect-to-existing model.

VITE_GRAFANA_URL / VITE_PROMETHEUS_URL remain as optional frontend
deep-link overrides. .env.example still needs a manual update (safety
policy blocks assistant edits): set ALERTMANAGER_URL empty/optional and
move standalone-only vars out of the root file.
2026-06-23 21:20:07 +00:00

294 lines
12 KiB
Markdown
Raw Blame History

# Observability Runbooks
Operational playbooks for the **standalone example observability stack**
(`docker-compose.observability.yml`) that can be deployed alongside Manage.
Manage itself does **not** deploy these services; it connects to existing
Grafana / Prometheus / Alertmanager instances. These runbooks cover operating
the standalone stack shipped under `monitoring/`.
## Service Overview
| Service | Compose name | Internal URL | Health check |
|---------|--------------|--------------|--------------|
| Prometheus | `prometheus` | `http://prometheus:9090` | `/-/healthy` |
| Grafana | `grafana` | `http://grafana:3000` | `/api/health` |
| Loki | `loki` | `http://loki:3100` | `/ready` |
| Alloy | `alloy` | `http://alloy:12345` | `/-/healthy` |
| Alertmanager | `alertmanager` | `http://alertmanager:9093` | `/-/healthy` |
| Node Exporter | `node-exporter` | `http://node-exporter:9100` | `/` |
| Manage backend | `backend` | `http://backend:8000` | `/api/health` |
---
## Alert: `BackupJobFailed`
**Severity**: critical
**Meaning**: A backup job reported `status=failed` within the last hour.
### Steps
1. Open **Manage → Backups** and identify the failed job/run.
2. Check the run output / logs for the failure reason.
3. Search Loki for `{container="backend"} | json | message=~"(?i)backup"` around the failure time.
4. If transient (network, lock file), retry the job.
5. If persistent, open a task to fix the backup script or credentials.
---
## Alert: `BackupJobStuck`
**Severity**: warning
**Meaning**: No successful backup run has been recorded for a job in the last 24 hours.
### Steps
1. Confirm the job is still scheduled and expected to run.
2. Check whether the backup scheduler/host is running.
3. Verify the job can still report success to `POST /api/backups/reports`.
4. Inspect Prometheus graph for `manage_backup_runs_last_success_timestamp` by `job_name`.
5. If the job was intentionally retired, remove or disable its reporting.
---
## Alert: `PrometheusTargetMissing`
**Severity**: warning
**Meaning**: A Prometheus scrape target is down (`up == 0`) for more than 2 minutes.
### Steps
1. Identify `job` and `instance` from the alert labels.
2. Check the container/process status:
- `docker compose ps <service>`
- `docker compose logs --tail 100 <service>`
3. Verify network reachability from the Prometheus container:
- `docker compose exec prometheus wget -qO- http://<instance>/`
4. If the target is a remote Node Exporter:
- Check the machine is reachable over SSH.
- Verify Node Exporter is installed and running (`systemctl status node_exporter`).
- Confirm the scrape host/port in Manage → Settings for that machine.
5. Restart if needed: `docker compose restart <service>`.
---
## Alert: `AlertmanagerDown`
**Severity**: critical
**Meaning**: Prometheus cannot scrape Alertmanager; new alerts may not be delivered.
### Steps
1. Check container status: `docker compose ps alertmanager`
2. Review logs: `docker compose logs --tail 200 alertmanager`
3. Validate config syntax:
- `docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml`
4. Verify SMTP environment variables are present if using email receivers.
5. Restart: `docker compose restart alertmanager`
---
## Alert: `GrafanaDown`
**Severity**: warning
**Meaning**: Grafana is unreachable; dashboards and iframe panels in Manage are unavailable.
### Steps
1. Check container status and logs.
2. Verify the OAuth client configuration is correct (`GF_AUTH_GENERIC_OAUTH_*`).
3. If embedded panels are blank, confirm Grafana `allow_embedding = true` and cookie settings.
4. Restart: `docker compose restart grafana`
---
## Routine Maintenance
### Check overall health
```bash
cd /path/to/manage
docker compose ps
docker compose exec prometheus wget -qO- http://127.0.0.1:9090/-/healthy
docker compose exec grafana wget -qO- http://127.0.0.1:3000/api/health
docker compose exec loki wget -qO- http://127.0.0.1:3100/ready
docker compose exec alertmanager wget -qO- http://127.0.0.1:9093/-/healthy
```
### Reload Prometheus after rule/config changes
Prometheus is started with `--web.enable-lifecycle`, so a SIGHUP or HTTP call reloads config:
```bash
curl -X POST http://localhost:9090/-/reload
```
### Inspect logs
```bash
# All backend logs in Loki via Grafana Explore, or locally:
docker compose logs --tail 500 backend
# Specific service:
docker compose logs -f prometheus
```
### Storage usage
```bash
docker system df -v
docker compose exec prometheus du -sh /prometheus
docker compose exec loki du -sh /loki
docker compose exec grafana du -sh /var/lib/grafana
```
---
## Backup and Disaster Recovery
The observability data lives on the host under `OBSERVABILITY_DATA_ROOT` (`./observability-data` by default). Subdirectories are created for each service:
- `prometheus`
- `loki`
- `grafana`
- `alertmanager`
- `alloy`
### Backup data
```bash
# Stop the stack to ensure consistency
docker compose -f docker-compose.observability.yml down
# Back up the whole data directory
rsync -aP --delete "$OBSERVABILITY_DATA_ROOT" /mnt/backups/observability-data/
# Start the stack again
docker compose -f docker-compose.observability.yml up -d
```
### Restore data
```bash
docker compose -f docker-compose.observability.yml down
rm -rf "$OBSERVABILITY_DATA_ROOT"
rsync -aP /mnt/backups/observability-data/ "$OBSERVABILITY_DATA_ROOT"
docker compose -f docker-compose.observability.yml up -d
```
---
## Standalone Observability Stack
Run the observability services without the Manage backend or frontend:
```bash
cd /path/to/manage
# create an env file with at least the required variables
cat > .env.observability <<EOF
CERT_RESOLVER=myresolver
PROMETHEUS_ROOT=/var/lib/manage/observability/prometheus
LOKI_ROOT=/var/lib/manage/observability/loki
ALLOY_ROOT=/var/lib/manage/observability/alloy
GRAFANA_ROOT=/var/lib/manage/observability/grafana
ALERTMANAGER_ROOT=/var/lib/manage/observability/alertmanager
GRAFANA_APP_HOST=grafana.example.com
PROMETHEUS_APP_HOST=prometheus.example.com
ALERTMANAGER_APP_HOST=alertmanager.example.com
EOF
# prepare config directories from the repository defaults
mkdir -p "$PROMETHEUS_ROOT"/{config,data} "$LOKI_ROOT"/{config,data} "$ALLOY_ROOT"/{config,data} "$GRAFANA_ROOT"/{config,data} "$ALERTMANAGER_ROOT"/{config,data}
cp monitoring/prometheus/prometheus.standalone.yml "$PROMETHEUS_ROOT/config/prometheus.yml"
cp -r monitoring/prometheus/rules "$PROMETHEUS_ROOT/config/rules"
cp -r monitoring/prometheus/file-sd "$PROMETHEUS_ROOT/config/file-sd"
cp monitoring/loki/loki.yml "$LOKI_ROOT/config/loki.yml"
cp monitoring/alloy/config.alloy "$ALLOY_ROOT/config/config.alloy"
cp monitoring/grafana/grafana.ini "$GRAFANA_ROOT/config/grafana.ini"
cp -r monitoring/grafana/provisioning "$GRAFANA_ROOT/config/provisioning"
cp monitoring/alertmanager/alertmanager.yml "$ALERTMANAGER_ROOT/config/alertmanager.yml"
docker compose -f docker-compose.observability.yml --env-file .env.observability up -d
```
Each service root must contain `config/` (read-only config files) and `data/` (runtime state). If you do not use Traefik, set `CERT_RESOLVER` to any non-empty value and do not attach the services to a `web` network. The direct host ports still work without Traefik.
### Reachable web UIs
Only three services expose a human-facing web interface. With Traefik they are served on their public hostnames; direct ports are still open on localhost for debugging.
| Service | Has UI | Direct URL | Traefik hostname variable | Notes |
|---------|--------|------------|---------------------------|-------|
| Grafana | yes | `http://localhost:3000` | `GRAFANA_APP_HOST` | Dashboards, log explore, alert management |
| Prometheus | yes | `http://localhost:9090` | `PROMETHEUS_APP_HOST` | Query, targets, alerts, config status |
| Alertmanager | yes | `http://localhost:9093` | `ALERTMANAGER_APP_HOST` | Alerts, silences, routing status |
| Loki | no | `http://localhost:3100` | none | Log API only; browse logs through Grafana |
| Alloy | partial | `http://localhost:12345` | none | Agent debug UI for pipeline inspection |
| Node Exporter | no | `http://localhost:9100` | none | Metrics endpoint only (`/metrics`) |
Grafana defaults to `admin` / `admin`. Datasources and dashboards are provisioned automatically.
### Environment variables
| Variable | Default | Purpose |
|----------|---------|---------|
| `PROMETHEUS_ROOT` | required | Absolute host directory for Prometheus config and data. Must contain `config/` and `data/`. |
| `LOKI_ROOT` | required | Absolute host directory for Loki config and data. Must contain `config/` and `data/`. |
| `ALLOY_ROOT` | required | Absolute host directory for Alloy config and data. Must contain `config/` and `data/`. |
| `GRAFANA_ROOT` | required | Absolute host directory for Grafana config and data. Must contain `config/` and `data/`. |
| `ALERTMANAGER_ROOT` | required | Absolute host directory for Alertmanager config and data. Must contain `config/` and `data/`. |
| `CERT_RESOLVER` | required | Traefik certificate resolver name (for example `letsencrypt` or `cloudflare`). Must be set before deploy. |
| `TRAEFIK_ENTRYPOINT` | `websecure` | Traefik entrypoint to use for the web UIs. |
| `GRAFANA_APP_HOST` | required | Public hostname for Grafana (for example `grafana.example.com`). |
| `GRAFANA_APP_NAME` | `grafana` | Traefik router/service name for Grafana. |
| `GRAFANA_APP_PORT` | `3000` | Internal port Traefik forwards to for Grafana. |
| `PROMETHEUS_APP_HOST` | required | Public hostname for Prometheus (for example `prometheus.example.com`). |
| `PROMETHEUS_APP_NAME` | `prometheus` | Traefik router/service name for Prometheus. |
| `PROMETHEUS_APP_PORT` | `9090` | Internal port Traefik forwards to for Prometheus. |
| `ALERTMANAGER_APP_HOST` | required | Public hostname for Alertmanager (for example `alertmanager.example.com`). |
| `ALERTMANAGER_APP_NAME` | `alertmanager` | Traefik router/service name for Alertmanager. |
| `ALERTMANAGER_APP_PORT` | `9093` | Internal port Traefik forwards to for Alertmanager. |
| `PROMETHEUS_PORT` | `9090` | Direct host port for Prometheus web UI and API. |
| `LOKI_PORT` | `3100` | Direct host port for Loki API. |
| `ALLOY_PORT` | `12345` | Direct host port for Alloy debug UI. |
| `GRAFANA_PORT` | `3000` | Direct host port for Grafana web UI. |
| `ALERTMANAGER_PORT` | `9093` | Direct host port for Alertmanager web UI. |
| `NODE_EXPORTER_PORT` | `9100` | Direct host port for Node Exporter metrics endpoint. |
| `GRAFANA_ADMIN_USER` | `admin` | Grafana admin username. |
| `GRAFANA_ADMIN_PASSWORD` | `admin` | Grafana admin password. Change this in production. |
| `GF_AUTH_GENERIC_OAUTH_CLIENT_ID` | empty | Generic OAuth client ID for Authentik or another provider. |
| `GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET` | empty | Generic OAuth client secret. |
| `GF_AUTH_GENERIC_OAUTH_AUTH_URL` | empty | OAuth authorization endpoint. |
| `GF_AUTH_GENERIC_OAUTH_TOKEN_URL` | empty | OAuth token endpoint. |
| `GF_AUTH_GENERIC_OAUTH_API_URL` | empty | OAuth userinfo endpoint. |
| `LOG_LEVEL` | `INFO` | Grafana log level. |
| `SMTP_HOST` | `smtp.example.com` | SMTP host for Alertmanager email notifications. |
| `SMTP_PORT` | `587` | SMTP port for Alertmanager. |
| `SMTP_USERNAME` | empty | SMTP username. |
| `SMTP_PASSWORD` | empty | SMTP password. |
| `SMTP_FROM_ADDRESS` | `no-reply@example.com` | From address for alert emails. |
| `ALERT_EMAIL_TO` | `admin@example.com` | Default recipient for alert emails. |
Prometheus and Alertmanager do not have authentication. When exposing them through Traefik, add a basic-auth middleware or restrict access by network.
To scrape a Manage backend from this standalone stack, edit `monitoring/prometheus/prometheus.standalone.yml` and add a static target for the backend's `/metrics` endpoint, or drop a file-SD JSON file into `monitoring/prometheus/file-sd/`.
### Backing up standalone data
Because config and data are stored on the host under each service root, you can back them up with normal filesystem tools:
```bash
for svc in prometheus loki alloy grafana alertmanager; do
rsync -aP --delete "/var/lib/manage/observability/$svc" "/mnt/backups/observability/$svc"
done
```
Stop the stack first if you need a consistent snapshot.
## Scaling Notes
- The current `deploy.resources` blocks are tuned for a small homelab. Raise memory limits if you monitor many machines or retain logs longer than 30 days.
- Loki is configured for single-node filesystem storage. For larger deployments, migrate to object storage (S3/GCS/MinIO) and a shared index.
- Prometheus remote-write or Thanos/Cortex can be added later for long-term metrics without changing application instrumentation.
Reference in New Issue View Git Blame Copy Permalink