docs(observability): refresh docs for service-registry end state

After the observability-service-registry slices removed the observability env vars and the file-SD writer, several docs still instructed readers to set vars that no longer exist. Updated the live config instructions; historical decision-log entries are left intact. - README.md: removed VITE_GRAFANA_URL/VITE_PROMETHEUS_URL/ALERTMANAGER_URL from compose examples and the env-var block; added a note that observability is configured on the Services page; updated Notes. - frontend/README.md: dropped the stale VITE_* deep-link sentence. - docs/REQUIREMENTS.md: fixed one stale trailing phrase in the externalization decision-log entry (VITE_* no longer "remain"). - docs/monitoring-logging-design.md: added a "Superseded mechanisms" note under Implementation Plan so the Phase 2/3 file-SD + alertmanager_url details read as historical, not current wiring. - context.md: strengthened the status banner to cover the env->service- registry and file-SD->http_sd_configs shift; body marked historical. .env.example is assistant-edit-blocked; updated replacement text provided to the user separately.
2026-06-24 09:15:50 +00:00
parent b1a66a1ab7
commit c1610c93a1
5 changed files with 35 additions and 18 deletions
@@ -88,17 +88,18 @@ export VITE_OIDC_ISSUER=https://auth.example.com/application/o/manage/
 export VITE_OIDC_CLIENT_ID=manage
 export VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback
 export VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/
-export VITE_GRAFANA_URL=https://grafana.manage.example.com
-export VITE_PROMETHEUS_URL=https://prometheus.manage.example.com
 export MANAGE_ENCRYPTION_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

 docker compose up --build
 ```

+> Observability services (Grafana, Prometheus, Alertmanager) are configured in
+> the app on the **Services** page — no env vars for them.
+
 Inline one-liner example:

 ```bash
-BACKEND_APP_HOST=api.manage.example.com FRONTEND_APP_HOST=manage.example.com CERT_RESOLVER=letsencrypt VITE_OIDC_ISSUER=https://auth.example.com/application/o/manage/ VITE_OIDC_CLIENT_ID=manage VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/ VITE_GRAFANA_URL=https://grafana.manage.example.com VITE_PROMETHEUS_URL=https://prometheus.manage.example.com docker compose up --build
+BACKEND_APP_HOST=api.manage.example.com FRONTEND_APP_HOST=manage.example.com CERT_RESOLVER=letsencrypt VITE_OIDC_ISSUER=https://auth.example.com/application/o/manage/ VITE_OIDC_CLIENT_ID=manage VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/ docker compose up --build
 ```

 For local development, no SSH key is required unless you want to connect to remote SSH machines later:
@@ -145,9 +146,9 @@ VITE_OIDC_SCOPE=openid profile email
 VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback
 VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/

-# Grafana / Prometheus public URLs for frontend deep-links (service adapters read URLs from service records)
-VITE_GRAFANA_URL=https://grafana.manage.example.com
-VITE_PROMETHEUS_URL=https://prometheus.manage.example.com
+# Observability services (Grafana, Prometheus, Alertmanager) are configured in
+# the app on the Services page. The only observability env var is the optional
+# PROMETHEUS_ENABLED toggle (defaults on) for Manage's own /metrics endpoint.

 # Required: master key encrypting service secrets (API keys/tokens) at rest.
 # Generate one with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
@@ -185,5 +186,4 @@ cd frontend && npx tsc --noEmit && npm run build
 - Job templates are shell-quoted. Add new templates in `backend/src/media_library_viewer_api/jobs.py`.
 - Root-level Docker Compose files are provided for production (`docker-compose.yml`) and local development (`docker-compose.dev.yml`), and both rely on Compose interpolation rather than `env_file` entries. They deploy **only** the backend and frontend; Manage never deploys its own observability stack (see `docker-compose.observability.yml` for an optional standalone example).
 - The configurable dashboard stores widget instances in the backend SQLite settings database. New installs seed default Jellyfin activity and Backups widgets automatically.
- Grafana and Prometheus widget adapters resolve URLs from service records configured in the app; `VITE_GRAFANA_URL` / `VITE_PROMETHEUS_URL` are only used for frontend deep-links. No credentials are stored in widget config; service API keys are encrypted at rest with `MANAGE_ENCRYPTION_KEY`.
- `ALERTMANAGER_URL` is optional. When unset, the Alertmanager proxy endpoints return graceful "not configured" responses instead of erroring.
+- Grafana, Prometheus, and Alertmanager are configured as **service instances** in the app (Services page); their widget adapters resolve URLs from service records, and no observability URLs/credentials live in env vars. No credentials are stored in widget config; service API keys are encrypted at rest with `MANAGE_ENCRYPTION_KEY`. When no alertmanager service is configured, the alert proxy endpoints return graceful "not configured" responses.
@@ -1,13 +1,19 @@
 # Code Context

-> **Status (2026-06-23):** Manage no longer deploys an observability stack.
-> The root `docker-compose.yml` / `docker-compose.dev.yml` ship **only** the
-> backend and frontend; Grafana, Prometheus, Loki, Alertmanager, Alloy, and Node
-> Exporter were removed from them. Manage connects to **existing** instances.
-> The standalone example stack lives in `docker-compose.observability.yml`. Some
-> snippets below still reference the former in-compose services and are kept as
-> historical context; treat `docker-compose.observability.yml` as authoritative
-> for the stack layout.
+> **Status (2026-06-23):** Manage no longer deploys an observability stack,
+> and observability is no longer env-configured. The root `docker-compose.yml` /
+> `docker-compose.dev.yml` ship **only** the backend and frontend; Grafana,
+> Prometheus, Loki, Alertmanager, Alloy, and Node Exporter were removed from
+> them. Manage connects to **existing** instances, and Alertmanager/Grafana/
+> Prometheus are now **service-registry instances** configured in the UI (no
+> `ALERTMANAGER_URL`, `VITE_GRAFANA_URL`, `VITE_PROMETHEUS_URL`, or
+> `PROMETHEUS_FILE_SD_DIR` env vars). The file-SD writer was removed; external
+> Prometheus uses `http_sd_configs` against `GET /api/monitoring/prometheus-targets`.
+> The standalone example stack lives in `docker-compose.observability.yml`. The
+> detailed snippets below describe the **former** in-compose, env-configured
+> model and are kept as historical context only; treat the service registry
+> (Services page UI + `openspec/changes/observability-service-registry/`) and
+> `docker-compose.observability.yml` as authoritative.

 ## Files Retrieved

@@ -332,7 +332,7 @@ the widget/addon-pages model were removed. `MANAGE_ENCRYPTION_KEY` is now requir
 ## Decision Log

 - 2026-06-23: Made all observability integration service-registry first-class with zero service env config. Alertmanager is now a 6th service type (`integrations/alertmanager.py`) with an `active_alerts` widget, mirroring Grafana/Prometheus. The monitoring router resolves alertmanager/grafana/prometheus from service records (first-enabled-instance default, optional `?service_id=`); added `/api/monitoring/grafana-status` and `/prometheus-status` health probes. The Observability page discovers services from the registry (dropped `import.meta.env.VITE_GRAFANA_URL`) and renders health cards; the dashboard gained an `active_alerts` widget. Removed the backend `alertmanager_url`/`alertmanager_webhook_url` fields and the frontend `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` build args — observability services are configured in the UI only. Removed the `PROMETHEUS_FILE_SD_DIR` shared-file bridge + `write_prometheus_targets` file writer; external Prometheus now uses `http_sd_configs` against `GET /api/monitoring/prometheus-targets` (`build_node_exporter_targets` + that endpoint stay). The webhook receiver is log-only (dropped the outbound forward). The only observability env var remaining is `PROMETHEUS_ENABLED` (Manage's own `/metrics` toggle). See `openspec/changes/observability-service-registry/`.
- 2026-06-23: Removed all observability services from the root Compose files. `docker-compose.yml` and `docker-compose.dev.yml` now deploy **only** the backend and frontend; the `monitoring` network and the `prometheus`/`loki`/`alloy`/`grafana`/`alertmanager`/`node-exporter` services and their named volumes were deleted, and the `GRAFANA_APP_HOST` Traefik rule was removed. Manage connects to **existing** Grafana/Prometheus/Alertmanager instances and does not deploy them itself. The backend `alertmanager_url` default is now empty, and the `/api/monitoring/alerts` and `/alertmanager-status` endpoints return graceful "not configured" responses when `ALERTMANAGER_URL` is unset. The previous in-compose stack is preserved as an optional standalone example in `docker-compose.observability.yml` (config under `monitoring/`, documented in `docs/observability-runbooks.md`). `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` remain as optional frontend deep-link overrides.
+- 2026-06-23: Removed all observability services from the root Compose files. `docker-compose.yml` and `docker-compose.dev.yml` now deploy **only** the backend and frontend; the `monitoring` network and the `prometheus`/`loki`/`alloy`/`grafana`/`alertmanager`/`node-exporter` services and their named volumes were deleted, and the `GRAFANA_APP_HOST` Traefik rule was removed. Manage connects to **existing** Grafana/Prometheus/Alertmanager instances and does not deploy them itself. The backend `alertmanager_url` default is now empty, and the `/api/monitoring/alerts` and `/alertmanager-status` endpoints return graceful "not configured" responses when `ALERTMANAGER_URL` is unset. The previous in-compose stack is preserved as an optional standalone example in `docker-compose.observability.yml` (config under `monitoring/`, documented in `docs/observability-runbooks.md`). (The `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` overrides were later removed entirely — see the service-registry entry above.)
 - 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template.
 - 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring` → `/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`.
 - 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`.
@@ -307,6 +307,17 @@ The existing POSIX remote collector will be removed, and the Python backup alert

 ## Implementation Plan

+> **Superseded mechanisms (2026-06-23):** Phases 2 and 3 originally wired Manage
+> to a co-deployed Prometheus/Alertmanager via a shared-file **file-SD bridge**
+> (`PROMETHEUS_FILE_SD_DIR` + `write_prometheus_targets`) and an
+> `alertmanager_url` env var. Both have been removed. Manage now resolves
+> Alertmanager/Grafana/Prometheus from the **service registry**, and external
+> Prometheus instances consume node-exporter targets via **`http_sd_configs`**
+> against `GET /api/monitoring/prometheus-targets`. Treat the Phase 2/3
+> file/config details below as the historical implementation, not the current
+> wiring. See `openspec/changes/observability-service-registry/` and the
+> decision log in `docs/REQUIREMENTS.md`.
+
 ### Phase 0 — Foundation and Cleanup

 - [x] Add `prometheus-client` and `python-json-logger` to `backend/pyproject.toml`.
@@ -67,4 +67,4 @@ VITE_API_URL=http://your-backend-host:8000
 - **Production**: export the required Compose variables in your shell, run `docker compose up --build`, and manage local/remote machines from **Settings**.
 - **Observability**: Manage only deploys backend + frontend. It connects to **existing** Grafana/Prometheus/Alertmanager instances; see `docker-compose.observability.yml` for an optional standalone example stack. A machine can be `local` (the API host itself) or `ssh` (a remote host), and the UI treats both the same after configuration.

-In development, the Vite proxy handles `/api` requests automatically. `VITE_GRAFANA_URL` / `VITE_PROMETHEUS_URL` are optional frontend deep-link overrides for your existing instances.
+In development, the Vite proxy handles `/api` requests automatically. Observability services (Grafana, Prometheus, Alertmanager) are configured in the app on the Services page — there are no observability env vars.