alex/manage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): add Traefik routing and required absolute data root

Browse Source 
- Make OBSERVABILITY_DATA_ROOT required in docker-compose.observability.yml
  so Portainer deployments must set an absolute host path.
- Add Traefik labels and the web external network to Grafana, Prometheus,
  and Alertmanager with env-driven hostnames, router/service names,
  internal ports, entrypoint, and cert resolver.
- Keep direct host ports as a fallback for local debugging.
- Update docs/observability-runbooks.md with required variables,
  Traefik hostname column, and a .env.observability example.
This commit is contained in:
Developer
2026-06-16 16:23:54 +00:00
parent 53dcd16735
commit 2d1674cb08
2 changed files with 82 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/observability-runbooks.md
+41 -19
View File
@@ -179,23 +179,32 @@ Run the observability services without the Manage backend or frontend:
```bash
cd /path/to/manage
cp .env.example .env
# edit .env as needed
docker compose -f docker-compose.observability.yml up -d
# create an env file with at least the required variables
cat > .env.observability <<EOF
OBSERVABILITY_DATA_ROOT=/var/lib/manage/observability
CERT_RESOLVER=myresolver
GRAFANA_APP_HOST=grafana.example.com
PROMETHEUS_APP_HOST=prometheus.example.com
ALERTMANAGER_APP_HOST=alertmanager.example.com
EOF
docker compose -f docker-compose.observability.yml --env-file .env.observability up -d
```
If you do not use Traefik, set `CERT_RESOLVER` to any non-empty value and do not attach the services to a `web` network. The direct host ports still work without Traefik.
### Reachable web UIs
Only three services expose a human-facing web interface:
Only three services expose a human-facing web interface. With Traefik they are served on their public hostnames; direct ports are still open on localhost for debugging.
| Service | Has UI | Default URL | Notes |
|---------|--------|-------------|-------|
| Grafana | yes | `http://localhost:3000` | Dashboards, log explore, alert management |
| Prometheus | yes | `http://localhost:9090` | Query, targets, alerts, config status |
| Alertmanager | yes | `http://localhost:9093` | Alerts, silences, routing status |
| Loki | no | `http://localhost:3100` | Log API only; browse logs through Grafana |
| Alloy | partial | `http://localhost:12345` | Agent debug UI for pipeline inspection |
| Node Exporter | no | `http://localhost:9100` | Metrics endpoint only (`/metrics`) |
| Service | Has UI | Direct URL | Traefik hostname variable | Notes |
|---------|--------|------------|---------------------------|-------|
| Grafana | yes | `http://localhost:3000` | `GRAFANA_APP_HOST` | Dashboards, log explore, alert management |
| Prometheus | yes | `http://localhost:9090` | `PROMETHEUS_APP_HOST` | Query, targets, alerts, config status |
| Alertmanager | yes | `http://localhost:9093` | `ALERTMANAGER_APP_HOST` | Alerts, silences, routing status |
| Loki | no | `http://localhost:3100` | none | Log API only; browse logs through Grafana |
| Alloy | partial | `http://localhost:12345` | none | Agent debug UI for pipeline inspection |
| Node Exporter | no | `http://localhost:9100` | none | Metrics endpoint only (`/metrics`) |
Grafana defaults to `admin` / `admin`. Datasources and dashboards are provisioned automatically.
@@ -203,13 +212,24 @@ Grafana defaults to `admin` / `admin`. Datasources and dashboards are provisione
| Variable | Default | Purpose |
|----------|---------|---------|
| `OBSERVABILITY_DATA_ROOT` | `./observability-data` | Host directory where all service data is stored persistently. Each service gets a subdirectory inside it. |
| `PROMETHEUS_PORT` | `9090` | Host port for Prometheus web UI and API. |
| `LOKI_PORT` | `3100` | Host port for Loki API. |
| `ALLOY_PORT` | `12345` | Host port for Alloy debug UI. |
| `GRAFANA_PORT` | `3000` | Host port for Grafana web UI. |
| `ALERTMANAGER_PORT` | `9093` | Host port for Alertmanager web UI. |
| `NODE_EXPORTER_PORT` | `9100` | Host port for Node Exporter metrics endpoint. |
| `OBSERVABILITY_DATA_ROOT` | required | Absolute host directory where all service data is stored. Each service gets a subdirectory inside it. Must be set before deploy. |
| `CERT_RESOLVER` | required | Traefik certificate resolver name (for example `letsencrypt` or `cloudflare`). Must be set before deploy. |
| `TRAEFIK_ENTRYPOINT` | `websecure` | Traefik entrypoint to use for the web UIs. |
| `GRAFANA_APP_HOST` | required | Public hostname for Grafana (for example `grafana.example.com`). |
| `GRAFANA_APP_NAME` | `grafana` | Traefik router/service name for Grafana. |
| `GRAFANA_APP_PORT` | `3000` | Internal port Traefik forwards to for Grafana. |
| `PROMETHEUS_APP_HOST` | required | Public hostname for Prometheus (for example `prometheus.example.com`). |
| `PROMETHEUS_APP_NAME` | `prometheus` | Traefik router/service name for Prometheus. |
| `PROMETHEUS_APP_PORT` | `9090` | Internal port Traefik forwards to for Prometheus. |
| `ALERTMANAGER_APP_HOST` | required | Public hostname for Alertmanager (for example `alertmanager.example.com`). |
| `ALERTMANAGER_APP_NAME` | `alertmanager` | Traefik router/service name for Alertmanager. |
| `ALERTMANAGER_APP_PORT` | `9093` | Internal port Traefik forwards to for Alertmanager. |
| `PROMETHEUS_PORT` | `9090` | Direct host port for Prometheus web UI and API. |
| `LOKI_PORT` | `3100` | Direct host port for Loki API. |
| `ALLOY_PORT` | `12345` | Direct host port for Alloy debug UI. |
| `GRAFANA_PORT` | `3000` | Direct host port for Grafana web UI. |
| `ALERTMANAGER_PORT` | `9093` | Direct host port for Alertmanager web UI. |
| `NODE_EXPORTER_PORT` | `9100` | Direct host port for Node Exporter metrics endpoint. |
| `GRAFANA_ADMIN_USER` | `admin` | Grafana admin username. |
| `GRAFANA_ADMIN_PASSWORD` | `admin` | Grafana admin password. Change this in production. |
| `GF_AUTH_GENERIC_OAUTH_CLIENT_ID` | empty | Generic OAuth client ID for Authentik or another provider. |
@@ -225,6 +245,8 @@ Grafana defaults to `admin` / `admin`. Datasources and dashboards are provisione
| `SMTP_FROM_ADDRESS` | `no-reply@example.com` | From address for alert emails. |
| `ALERT_EMAIL_TO` | `admin@example.com` | Default recipient for alert emails. |
Prometheus and Alertmanager do not have authentication. When exposing them through Traefik, add a basic-auth middleware or restrict access by network.
To scrape a Manage backend from this standalone stack, edit `monitoring/prometheus/prometheus.standalone.yml` and add a static target for the backend's `/metrics` endpoint, or drop a file-SD JSON file into `monitoring/prometheus/file-sd/`.
### Backing up standalone data