alex/manage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): persist standalone stack data and expose ports via env

Browse Source 
- Switch docker-compose.observability.yml from named volumes to host
  bind mounts under OBSERVABILITY_DATA_ROOT, defaulting to
  ./observability-data.
- Make all service ports configurable via environment variables
  (PROMETHEUS_PORT, LOKI_PORT, ALLOY_PORT, GRAFANA_PORT,
  ALERTMANAGER_PORT, NODE_EXPORTER_PORT).
- Add VITE_GRAFANA_URL handling to ObservabilityPage so Grafana links
  point to the configured standalone instance.
- Update docs/observability-runbooks.md with the env variable table,
  reachable-web-UI table, and backup/restore instructions for the new
  host-directory layout.
This commit is contained in:
Developer
2026-06-16 15:37:41 +00:00
parent e783d2c07b
commit 53dcd16735
3 changed files with 140 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/observability-runbooks.md
+66 -36
View File
@@ -141,72 +141,102 @@ docker compose exec grafana du -sh /var/lib/grafana
## Backup and Disaster Recovery
The observability data lives in named volumes:
The observability data lives on the host under `OBSERVABILITY_DATA_ROOT` (`./observability-data` by default). Subdirectories are created for each service:
- `prometheus_data`
- `loki_data`
- `grafana_data`
- `alertmanager_data`
- `prometheus`
- `loki`
- `grafana`
- `alertmanager`
- `alloy`
### Backup volumes
### Backup data
```bash
# Stop the stack to ensure consistency
docker compose down
docker compose -f docker-compose.observability.yml down
# Back up each volume to a tarball
docker run --rm -v manage_prometheus_data:/data -v $(pwd)/backups:/backups alpine \
tar czf /backups/prometheus-$(date +%F).tar.gz -C /data .
docker run --rm -v manage_loki_data:/data -v $(pwd)/backups:/backups alpine \
tar czf /backups/loki-$(date +%F).tar.gz -C /data .
docker run --rm -v manage_grafana_data:/data -v $(pwd)/backups:/backups alpine \
tar czf /backups/grafana-$(date +%F).tar.gz -C /data .
docker run --rm -v manage_alertmanager_data:/data -v $(pwd)/backups:/backups alpine \
tar czf /backups/alertmanager-$(date +%F).tar.gz -C /data .
# Back up the whole data directory
rsync -aP --delete "$OBSERVABILITY_DATA_ROOT" /mnt/backups/observability-data/
# Start the stack again
docker compose up -d
docker compose -f docker-compose.observability.yml up -d
```
> Replace `manage_` with your actual Docker Compose project name if different.
### Restore a volume
### Restore data
```bash
docker compose down
docker volume rm manage_prometheus_data
docker volume create manage_prometheus_data
docker run --rm -v manage_prometheus_data:/data -v $(pwd)/backups:/backups alpine \
tar xzf /backups/prometheus-YYYY-MM-DD.tar.gz -C /data
docker compose up -d
docker compose -f docker-compose.observability.yml down
rm -rf "$OBSERVABILITY_DATA_ROOT"
rsync -aP /mnt/backups/observability-data/ "$OBSERVABILITY_DATA_ROOT"
docker compose -f docker-compose.observability.yml up -d
```
---
## Standalone Observability Stack
You can run the observability services on their own without the Manage backend or frontend:
Run the observability services without the Manage backend or frontend:
```bash
cd /path/to/manage
cp .env.example .env
# edit .env as needed
docker compose -f docker-compose.observability.yml up -d
```
This starts Prometheus, Grafana, Loki, Alertmanager, Alloy and Node Exporter. Exposed ports:
### Reachable web UIs
| Service | URL |
|---------|-----|
| Grafana | <http://localhost:3000> |
| Prometheus | <http://localhost:9090> |
| Alertmanager | <http://localhost:9093> |
| Loki | <http://localhost:3100> |
| Node Exporter | <http://localhost:9100> |
| Alloy | <http://localhost:12345> |
Only three services expose a human-facing web interface:
| Service | Has UI | Default URL | Notes |
|---------|--------|-------------|-------|
| Grafana | yes | `http://localhost:3000` | Dashboards, log explore, alert management |
| Prometheus | yes | `http://localhost:9090` | Query, targets, alerts, config status |
| Alertmanager | yes | `http://localhost:9093` | Alerts, silences, routing status |
| Loki | no | `http://localhost:3100` | Log API only; browse logs through Grafana |
| Alloy | partial | `http://localhost:12345` | Agent debug UI for pipeline inspection |
| Node Exporter | no | `http://localhost:9100` | Metrics endpoint only (`/metrics`) |
Grafana defaults to `admin` / `admin`. Datasources and dashboards are provisioned automatically.
### Environment variables
| Variable | Default | Purpose |
|----------|---------|---------|
| `OBSERVABILITY_DATA_ROOT` | `./observability-data` | Host directory where all service data is stored persistently. Each service gets a subdirectory inside it. |
| `PROMETHEUS_PORT` | `9090` | Host port for Prometheus web UI and API. |
| `LOKI_PORT` | `3100` | Host port for Loki API. |
| `ALLOY_PORT` | `12345` | Host port for Alloy debug UI. |
| `GRAFANA_PORT` | `3000` | Host port for Grafana web UI. |
| `ALERTMANAGER_PORT` | `9093` | Host port for Alertmanager web UI. |
| `NODE_EXPORTER_PORT` | `9100` | Host port for Node Exporter metrics endpoint. |
| `GRAFANA_ADMIN_USER` | `admin` | Grafana admin username. |
| `GRAFANA_ADMIN_PASSWORD` | `admin` | Grafana admin password. Change this in production. |
| `GF_AUTH_GENERIC_OAUTH_CLIENT_ID` | empty | Generic OAuth client ID for Authentik or another provider. |
| `GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET` | empty | Generic OAuth client secret. |
| `GF_AUTH_GENERIC_OAUTH_AUTH_URL` | empty | OAuth authorization endpoint. |
| `GF_AUTH_GENERIC_OAUTH_TOKEN_URL` | empty | OAuth token endpoint. |
| `GF_AUTH_GENERIC_OAUTH_API_URL` | empty | OAuth userinfo endpoint. |
| `LOG_LEVEL` | `INFO` | Grafana log level. |
| `SMTP_HOST` | `smtp.example.com` | SMTP host for Alertmanager email notifications. |
| `SMTP_PORT` | `587` | SMTP port for Alertmanager. |
| `SMTP_USERNAME` | empty | SMTP username. |
| `SMTP_PASSWORD` | empty | SMTP password. |
| `SMTP_FROM_ADDRESS` | `no-reply@example.com` | From address for alert emails. |
| `ALERT_EMAIL_TO` | `admin@example.com` | Default recipient for alert emails. |
To scrape a Manage backend from this standalone stack, edit `monitoring/prometheus/prometheus.standalone.yml` and add a static target for the backend's `/metrics` endpoint, or drop a file-SD JSON file into `monitoring/prometheus/file-sd/`.
### Backing up standalone data
Because data is stored on the host under `OBSERVABILITY_DATA_ROOT`, you can back it up with normal filesystem tools:
```bash
rsync -aP --delete "$OBSERVABILITY_DATA_ROOT" /mnt/backups/observability-data/
```
Stop the stack first if you need a consistent snapshot.
## Scaling Notes
- The current `deploy.resources` blocks are tuned for a small homelab. Raise memory limits if you monitor many machines or retain logs longer than 30 days.