feat(observability): per-service root directories for config and data

- Replace OBSERVABILITY_DATA_ROOT with per-service *_ROOT variables in
  docker-compose.observability.yml.
- Each service root must contain config/ (mounted read-only) and data/
  (mounted read-write), grouping config and state together for Portainer.
- Update docs/observability-runbooks.md with the new variables, a setup
  script that copies repo configs into each service root, and updated
  backup/restore examples.

docs/observability-runbooks.md

@@ -181,17 +181,34 @@ Run the observability services without the Manage backend or frontend:
 cd /path/to/manage
 # create an env file with at least the required variables
 cat > .env.observability <<EOF
-OBSERVABILITY_DATA_ROOT=/var/lib/manage/observability
 CERT_RESOLVER=myresolver
+
+PROMETHEUS_ROOT=/var/lib/manage/observability/prometheus
+LOKI_ROOT=/var/lib/manage/observability/loki
+ALLOY_ROOT=/var/lib/manage/observability/alloy
+GRAFANA_ROOT=/var/lib/manage/observability/grafana
+ALERTMANAGER_ROOT=/var/lib/manage/observability/alertmanager
+
 GRAFANA_APP_HOST=grafana.example.com
 PROMETHEUS_APP_HOST=prometheus.example.com
 ALERTMANAGER_APP_HOST=alertmanager.example.com
 EOF
 
+# prepare config directories from the repository defaults
+mkdir -p "$PROMETHEUS_ROOT"/{config,data} "$LOKI_ROOT"/{config,data} "$ALLOY_ROOT"/{config,data} "$GRAFANA_ROOT"/{config,data} "$ALERTMANAGER_ROOT"/{config,data}
+cp monitoring/prometheus/prometheus.standalone.yml "$PROMETHEUS_ROOT/config/prometheus.yml"
+cp -r monitoring/prometheus/rules "$PROMETHEUS_ROOT/config/rules"
+cp -r monitoring/prometheus/file-sd "$PROMETHEUS_ROOT/config/file-sd"
+cp monitoring/loki/loki.yml "$LOKI_ROOT/config/loki.yml"
+cp monitoring/alloy/config.alloy "$ALLOY_ROOT/config/config.alloy"
+cp monitoring/grafana/grafana.ini "$GRAFANA_ROOT/config/grafana.ini"
+cp -r monitoring/grafana/provisioning "$GRAFANA_ROOT/config/provisioning"
+cp monitoring/alertmanager/alertmanager.yml "$ALERTMANAGER_ROOT/config/alertmanager.yml"
+
 docker compose -f docker-compose.observability.yml --env-file .env.observability up -d
 ```
 
-If you do not use Traefik, set `CERT_RESOLVER` to any non-empty value and do not attach the services to a `web` network. The direct host ports still work without Traefik.
+Each service root must contain `config/` (read-only config files) and `data/` (runtime state). If you do not use Traefik, set `CERT_RESOLVER` to any non-empty value and do not attach the services to a `web` network. The direct host ports still work without Traefik.
 
 ### Reachable web UIs
 
@@ -212,7 +229,11 @@ Grafana defaults to `admin` / `admin`. Datasources and dashboards are provisione
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
-| `OBSERVABILITY_DATA_ROOT` | required | Absolute host directory where all service data is stored. Each service gets a subdirectory inside it. Must be set before deploy. |
+| `PROMETHEUS_ROOT` | required | Absolute host directory for Prometheus config and data. Must contain `config/` and `data/`. |
+| `LOKI_ROOT` | required | Absolute host directory for Loki config and data. Must contain `config/` and `data/`. |
+| `ALLOY_ROOT` | required | Absolute host directory for Alloy config and data. Must contain `config/` and `data/`. |
+| `GRAFANA_ROOT` | required | Absolute host directory for Grafana config and data. Must contain `config/` and `data/`. |
+| `ALERTMANAGER_ROOT` | required | Absolute host directory for Alertmanager config and data. Must contain `config/` and `data/`. |
 | `CERT_RESOLVER` | required | Traefik certificate resolver name (for example `letsencrypt` or `cloudflare`). Must be set before deploy. |
 | `TRAEFIK_ENTRYPOINT` | `websecure` | Traefik entrypoint to use for the web UIs. |
 | `GRAFANA_APP_HOST` | required | Public hostname for Grafana (for example `grafana.example.com`). |
@@ -251,10 +272,12 @@ To scrape a Manage backend from this standalone stack, edit `monitoring/promethe
 
 ### Backing up standalone data
 
-Because data is stored on the host under `OBSERVABILITY_DATA_ROOT`, you can back it up with normal filesystem tools:
+Because config and data are stored on the host under each service root, you can back them up with normal filesystem tools:
 
 ```bash
-rsync -aP --delete "$OBSERVABILITY_DATA_ROOT" /mnt/backups/observability-data/
+for svc in prometheus loki alloy grafana alertmanager; do
+  rsync -aP --delete "/var/lib/manage/observability/$svc" "/mnt/backups/observability/$svc"
+done
 ```
 
 Stop the stack first if you need a consistent snapshot.