feat(observability): add Prometheus/Grafana/Loki/Alertmanager/Alloy stack and remove legacy Monitoring UI
This commit is contained in:
Developer
@@ -0,0 +1,190 @@
|
||||
# Observability Runbooks
|
||||
|
||||
Operational playbooks for the Manage self-hosted observability stack (Prometheus, Grafana, Loki, Alertmanager).
|
||||
|
||||
## Service Overview
|
||||
|
||||
| Service | Compose name | Internal URL | Health check |
|
||||
|---------|--------------|--------------|--------------|
|
||||
| Prometheus | `prometheus` | `http://prometheus:9090` | `/-/healthy` |
|
||||
| Grafana | `grafana` | `http://grafana:3000` | `/api/health` |
|
||||
| Loki | `loki` | `http://loki:3100` | `/ready` |
|
||||
| Alloy | `alloy` | `http://alloy:12345` | `/-/healthy` |
|
||||
| Alertmanager | `alertmanager` | `http://alertmanager:9093` | `/-/healthy` |
|
||||
| Node Exporter | `node-exporter` | `http://node-exporter:9100` | `/` |
|
||||
| Manage backend | `backend` | `http://backend:8000` | `/api/health` |
|
||||
|
||||
---
|
||||
|
||||
## Alert: `BackupJobFailed`
|
||||
|
||||
**Severity**: critical
|
||||
**Meaning**: A backup job reported `status=failed` within the last hour.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Open **Manage → Backups** and identify the failed job/run.
|
||||
2. Check the run output / logs for the failure reason.
|
||||
3. Search Loki for `{container="backend"} | json | message=~"(?i)backup"` around the failure time.
|
||||
4. If transient (network, lock file), retry the job.
|
||||
5. If persistent, open a task to fix the backup script or credentials.
|
||||
|
||||
---
|
||||
|
||||
## Alert: `BackupJobStuck`
|
||||
|
||||
**Severity**: warning
|
||||
**Meaning**: No successful backup run has been recorded for a job in the last 24 hours.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Confirm the job is still scheduled and expected to run.
|
||||
2. Check whether the backup scheduler/host is running.
|
||||
3. Verify the job can still report success to `POST /api/backups/reports`.
|
||||
4. Inspect Prometheus graph for `manage_backup_runs_last_success_timestamp` by `job_name`.
|
||||
5. If the job was intentionally retired, remove or disable its reporting.
|
||||
|
||||
---
|
||||
|
||||
## Alert: `PrometheusTargetMissing`
|
||||
|
||||
**Severity**: warning
|
||||
**Meaning**: A Prometheus scrape target is down (`up == 0`) for more than 2 minutes.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Identify `job` and `instance` from the alert labels.
|
||||
2. Check the container/process status:
|
||||
- `docker compose ps <service>`
|
||||
- `docker compose logs --tail 100 <service>`
|
||||
3. Verify network reachability from the Prometheus container:
|
||||
- `docker compose exec prometheus wget -qO- http://<instance>/`
|
||||
4. If the target is a remote Node Exporter:
|
||||
- Check the machine is reachable over SSH.
|
||||
- Verify Node Exporter is installed and running (`systemctl status node_exporter`).
|
||||
- Confirm the scrape host/port in Manage → Settings for that machine.
|
||||
5. Restart if needed: `docker compose restart <service>`.
|
||||
|
||||
---
|
||||
|
||||
## Alert: `AlertmanagerDown`
|
||||
|
||||
**Severity**: critical
|
||||
**Meaning**: Prometheus cannot scrape Alertmanager; new alerts may not be delivered.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Check container status: `docker compose ps alertmanager`
|
||||
2. Review logs: `docker compose logs --tail 200 alertmanager`
|
||||
3. Validate config syntax:
|
||||
- `docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml`
|
||||
4. Verify SMTP environment variables are present if using email receivers.
|
||||
5. Restart: `docker compose restart alertmanager`
|
||||
|
||||
---
|
||||
|
||||
## Alert: `GrafanaDown`
|
||||
|
||||
**Severity**: warning
|
||||
**Meaning**: Grafana is unreachable; dashboards and iframe panels in Manage are unavailable.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Check container status and logs.
|
||||
2. Verify the OAuth client configuration is correct (`GF_AUTH_GENERIC_OAUTH_*`).
|
||||
3. If embedded panels are blank, confirm Grafana `allow_embedding = true` and cookie settings.
|
||||
4. Restart: `docker compose restart grafana`
|
||||
|
||||
---
|
||||
|
||||
## Routine Maintenance
|
||||
|
||||
### Check overall health
|
||||
|
||||
```bash
|
||||
cd /path/to/manage
|
||||
docker compose ps
|
||||
docker compose exec prometheus wget -qO- http://127.0.0.1:9090/-/healthy
|
||||
docker compose exec grafana wget -qO- http://127.0.0.1:3000/api/health
|
||||
docker compose exec loki wget -qO- http://127.0.0.1:3100/ready
|
||||
docker compose exec alertmanager wget -qO- http://127.0.0.1:9093/-/healthy
|
||||
```
|
||||
|
||||
### Reload Prometheus after rule/config changes
|
||||
|
||||
Prometheus is started with `--web.enable-lifecycle`, so a SIGHUP or HTTP call reloads config:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9090/-/reload
|
||||
```
|
||||
|
||||
### Inspect logs
|
||||
|
||||
```bash
|
||||
# All backend logs in Loki via Grafana Explore, or locally:
|
||||
docker compose logs --tail 500 backend
|
||||
|
||||
# Specific service:
|
||||
docker compose logs -f prometheus
|
||||
```
|
||||
|
||||
### Storage usage
|
||||
|
||||
```bash
|
||||
docker system df -v
|
||||
docker compose exec prometheus du -sh /prometheus
|
||||
docker compose exec loki du -sh /loki
|
||||
docker compose exec grafana du -sh /var/lib/grafana
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Backup and Disaster Recovery
|
||||
|
||||
The observability data lives in named volumes:
|
||||
|
||||
- `prometheus_data`
|
||||
- `loki_data`
|
||||
- `grafana_data`
|
||||
- `alertmanager_data`
|
||||
|
||||
### Backup volumes
|
||||
|
||||
```bash
|
||||
# Stop the stack to ensure consistency
|
||||
docker compose down
|
||||
|
||||
# Back up each volume to a tarball
|
||||
docker run --rm -v manage_prometheus_data:/data -v $(pwd)/backups:/backups alpine \
|
||||
tar czf /backups/prometheus-$(date +%F).tar.gz -C /data .
|
||||
docker run --rm -v manage_loki_data:/data -v $(pwd)/backups:/backups alpine \
|
||||
tar czf /backups/loki-$(date +%F).tar.gz -C /data .
|
||||
docker run --rm -v manage_grafana_data:/data -v $(pwd)/backups:/backups alpine \
|
||||
tar czf /backups/grafana-$(date +%F).tar.gz -C /data .
|
||||
docker run --rm -v manage_alertmanager_data:/data -v $(pwd)/backups:/backups alpine \
|
||||
tar czf /backups/alertmanager-$(date +%F).tar.gz -C /data .
|
||||
|
||||
# Start the stack again
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
> Replace `manage_` with your actual Docker Compose project name if different.
|
||||
|
||||
### Restore a volume
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
docker volume rm manage_prometheus_data
|
||||
docker volume create manage_prometheus_data
|
||||
docker run --rm -v manage_prometheus_data:/data -v $(pwd)/backups:/backups alpine \
|
||||
tar xzf /backups/prometheus-YYYY-MM-DD.tar.gz -C /data
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scaling Notes
|
||||
|
||||
- The current `deploy.resources` blocks are tuned for a small homelab. Raise memory limits if you monitor many machines or retain logs longer than 30 days.
|
||||
- Loki is configured for single-node filesystem storage. For larger deployments, migrate to object storage (S3/GCS/MinIO) and a shared index.
|
||||
- Prometheus remote-write or Thanos/Cortex can be added later for long-term metrics without changing application instrumentation.
|
||||
Reference in New Issue
Block a user