diff --git a/openspec/changes/grafana-prometheus-polish/design.md b/openspec/changes/grafana-prometheus-polish/design.md deleted file mode 100644 index 2583617..0000000 --- a/openspec/changes/grafana-prometheus-polish/design.md +++ /dev/null @@ -1,64 +0,0 @@ -# Design — Grafana/Prometheus integration polish - -**Change:** `grafana-prometheus-polish` -**Phase:** design -**Date:** 2026-06-23 - -## Current state - -- Backend integrations: `integrations/grafana.py` (config `base_url`, secret `api_key`, widget `link`) and `integrations/prometheus.py` (config `base_url`, secret `api_key`, widget `metric`). -- Backend widget sources: `widgets/sources.py` has `GrafanaWidgetSource` and `PrometheusWidgetSource`. -- Frontend `SERVICE_REGISTRY` (`frontend/src/integrations/registry.ts`) mirrors those widget kinds. -- Observability page (`frontend/src/components/ObservabilityPage.tsx`) uses hard-coded `VITE_GRAFANA_URL` for Grafana links and shows only Alertmanager + Prometheus targets. -- Monitoring router (`backend/src/media_library_viewer_api/routers/monitoring.py`) exposes `/alerts`, `/alertmanager-status`, `/alertmanager-webhook`, `/prometheus-targets`, `/machines`. - -## Target state - -### Backend - -1. **New health endpoints** - - `GET /api/monitoring/grafana-status` — probes `GET {base_url}/api/health` (no auth needed for health) and returns `{up, version, error?}`. - - `GET /api/monitoring/prometheus-status` — probes `GET {base_url}/-/healthy` and returns `{up, version, error?}`. - - If multiple instances of a type exist, probe the first enabled instance and include its `service_id` + `name` in the response. - - If no instance is configured, return `up: false, error: "no_service_configured"`. - -2. **New widget kinds** - - Grafana: add `explore_link` kind with config `query` (optional), `datasource` (optional). Renders a deep-link to `/explore?orgId=1&left={...}` or similar. - - Prometheus: add `range_chart` kind with config `promql`, `step` (e.g. `1m`), `range_seconds` (e.g. `3600`). Renders `{ series: [{ timestamp, value }] }` for the frontend to chart. The adapter performs a range query (`/api/v1/query_range`) and reshapes the result. - -3. **Source adapters** - - Extend `GrafanaWidgetSource.fetch` to handle `explore_link`. - - Extend `PrometheusWidgetSource.fetch` to handle `range_chart`. - -### Frontend - -1. **Observability page** - - Replace hard-coded `GRAFANA_BASE_URL` with discovery from `useServiceInstances("grafana")`. - - Add `useGrafanaStatus()` and `usePrometheusStatus()` hooks using the new endpoints. - - Add health cards for Grafana and Prometheus next to Alertmanager. - - Grafana links (alerts panel, targets panel) use the discovered instance URL. - -2. **Widget registry** - - Add `explore_link` to `SERVICE_REGISTRY.grafana.widgets`. - - Add `range_chart` to `SERVICE_REGISTRY.prometheus.widgets`. - - Add presentational components `GrafanaExploreLinkWidget` and `PrometheusRangeChartWidget` (or extend existing components). - -3. **Types** - - Add status response types and widget config types. - -### Docs - -- Update `docs/REQUIREMENTS.md` observability section to describe service-instance-driven Grafana/Prometheus status and new widget kinds. -- `CHANGELOG.md` added/changed note. - -## Slices - -1. **Backend health endpoints + widget kind extensions.** -2. **Frontend Observability page + widget registry + new components.** -3. **Docs + changelog.** - -## Risks - -- Grafana Explore URL encoding is version-dependent. Target Grafana 10+ and document the assumption. -- Prometheus range queries can be expensive; cap `range_seconds` in the adapter (e.g. max 24h) and use a modest `step`. -- Multiple Grafana/Prometheus instances: keep it simple — use the first enabled instance for status; widgets already bind to a specific service instance. diff --git a/openspec/changes/grafana-prometheus-polish/proposal.md b/openspec/changes/grafana-prometheus-polish/proposal.md deleted file mode 100644 index f9a5535..0000000 --- a/openspec/changes/grafana-prometheus-polish/proposal.md +++ /dev/null @@ -1,44 +0,0 @@ -# Proposal — Grafana/Prometheus integration polish - -**Change:** `grafana-prometheus-polish` -**Phase:** proposal -**Date:** 2026-06-23 - -## Problem - -Grafana and Prometheus are already modeled as service-registry service types and provide dashboard widgets (`GrafanaLinkWidget`, `PrometheusMetricWidget`). The Observability page already shows Alertmanager alerts, Prometheus targets, and a hard-coded Grafana base URL (`VITE_GRAFANA_URL`). - -However, the integration is still piecemeal: - -1. **Hard-coded Grafana URL** in `frontend/src/components/ObservabilityPage.tsx` (`GRAFANA_BASE_URL = import.meta.env.VITE_GRAFANA_URL || "http://localhost:3000"`). This ignores any Grafana service instances configured in the registry. -2. **No Grafana health/status card** on the Observability page. Users cannot see whether their configured Grafana instance is reachable. -3. **No Prometheus health/status card**. The targets endpoint shows scrape config, but not whether Prometheus itself is up. -4. **Prometheus widget only supports instant queries**. There is no range-query chart widget or table widget. -5. **Grafana widget only supports dashboard deep-links**. There is no "open Explore with a query" shortcut. -6. **Observability page mixes service-bound and hard-coded sources**. It should derive Grafana/Prometheus URLs from configured service instances instead of env vars. - -## Goal - -Polish the Grafana/Prometheus integration so the Observability page and dashboard widgets feel first-class: - -1. Observability page discovers configured Grafana and Prometheus service instances and uses their `base_url` instead of `VITE_GRAFANA_URL`. -2. Add health/status cards for Grafana and Prometheus (reachable/unreachable + version if available). -3. Add backend health-check endpoints for Grafana (`/api/monitoring/grafana-status`) and Prometheus (`/api/monitoring/prometheus-status`) that proxy a lightweight health endpoint. -4. Extend widget kinds: - - Prometheus: add `range_chart` widget kind for simple sparkline charts. - - Grafana: add `explore_link` widget kind to open Grafana Explore with a pre-filled query. -5. Keep the existing `link` and `metric` widgets backward-compatible. - -## Non-goals - -- Do NOT build a full in-app metric explorer or charting library. -- Do NOT store Prometheus query history or persist widget state beyond existing dashboard widgets. -- Do NOT replace the Alertmanager proxy or Prometheus targets endpoint. - -## Acceptance - -- `VITE_GRAFANA_URL` is no longer required; the Observability page falls back to configured Grafana instances. -- `/api/monitoring/grafana-status` and `/api/monitoring/prometheus-status` return `{up, version, error?}`. -- New widget kinds are registered in backend integrations and frontend `SERVICE_REGISTRY`. -- Frontend lint/build/test and backend pytest remain green. -- Docs/CHANGELOG updated. diff --git a/openspec/changes/grafana-prometheus-polish/tasks.md b/openspec/changes/grafana-prometheus-polish/tasks.md deleted file mode 100644 index 5f9b700..0000000 --- a/openspec/changes/grafana-prometheus-polish/tasks.md +++ /dev/null @@ -1,106 +0,0 @@ -# Tasks — Grafana/Prometheus integration polish - -**Change:** `grafana-prometheus-polish` -**Phase:** tasks -**Date:** 2026-06-23 - -## Review workload forecast - -| Field | Value | -|-------|-------| -| Estimated changed lines | ~400–600 | -| Chained PRs recommended | Yes (2 PRs) | -| Chain strategy | stacked-to-main | - -## Slice 1: Backend — health endpoints + widget extensions - -**Goal:** Surface Grafana/Prometheus health and extend widget kinds. - -- [ ] **1.1 Add Grafana status endpoint** - - Files: `backend/src/media_library_viewer_api/routers/monitoring.py` - - Lines: ~35 - - Details: `GET /api/monitoring/grafana-status` discovers the first enabled `grafana` service, probes `/api/health`, returns `{ up, version, service_id, name, error? }`. - -- [ ] **1.2 Add Prometheus status endpoint** - - Files: `backend/src/media_library_viewer_api/routers/monitoring.py` - - Lines: ~35 - - Details: `GET /api/monitoring/prometheus-status` discovers the first enabled `prometheus` service, probes `/-/healthy` and `/api/v1/status/buildinfo` for version, returns `{ up, version, service_id, name, error? }`. - -- [ ] **1.3 Add `explore_link` widget kind to Grafana integration** - - Files: `backend/src/media_library_viewer_api/integrations/grafana.py`, `backend/src/media_library_viewer_api/widgets/sources.py` - - Lines: ~30 - - Details: New `GrafanaExploreWidgetConfig` with `query` and `datasource`; `GrafanaWidgetSource.fetch` builds an `/explore` deep-link. - -- [ ] **1.4 Add `range_chart` widget kind to Prometheus integration** - - Files: `backend/src/media_library_viewer_api/integrations/prometheus.py`, `backend/src/media_library_viewer_api/widgets/sources.py` - - Lines: ~60 - - Details: New `PrometheusRangeChartWidgetConfig` with `promql`, `range_seconds`, `step`; adapter calls `/api/v1/query_range`, caps `range_seconds` at 86400, reshapes to `{ series: [{ timestamp, value }] }`. - -- [ ] **1.5 Update backend tests** - - Files: `backend/tests/test_api.py`, `backend/tests/test_widgets.py` - - Lines: ~40 - - Details: Test status endpoints with mocked HTTP; test new widget kinds. - -- [ ] **1.6 Verify backend** - - Run: `cd backend && .venv/bin/ruff check . && PYTHONPATH=src .venv/bin/python -m pytest` - -**Slice 1 total:** ~200 changed lines. - -## Slice 2: Frontend — Observability page + widget registry - -**Goal:** Remove hard-coded Grafana URL and surface new widgets. - -- [ ] **2.1 Add status hooks** - - Files: `frontend/src/hooks/useObservability.ts`, `frontend/src/api/client.ts` - - Lines: ~30 - - Details: `fetchGrafanaStatus`, `fetchPrometheusStatus`, `useGrafanaStatus()`, `usePrometheusStatus()`. - -- [ ] **2.2 Update Observability page** - - Files: `frontend/src/components/ObservabilityPage.tsx` - - Lines: ~80 - - Details: Discover Grafana/Prometheus services; replace `GRAFANA_BASE_URL`; add health cards; use discovered URLs for links. - -- [ ] **2.3 Extend widget registry** - - Files: `frontend/src/integrations/registry.ts`, `frontend/src/widgets/*` - - Lines: ~80 - - Details: Add `GrafanaExploreLinkWidget` and `PrometheusRangeChartWidget` (or extend existing); register new kinds; update `registry.test.ts`. - -- [ ] **2.4 Update types** - - Files: `frontend/src/types/index.ts` - - Lines: ~20 - - Details: Add `GrafanaStatus`, `PrometheusStatus`, and widget config types. - -- [ ] **2.5 Update frontend tests** - - Files: `frontend/src/components/__tests__/ObservabilityPage.test.tsx` if it exists, or relevant widget tests - - Lines: ~30 - - Details: Mock new hooks/services; assert health cards render. - -- [ ] **2.6 Verify frontend** - - Run: `cd frontend && npm run lint && npm run build && npm run test` - -**Slice 2 total:** ~240 changed lines. - -## Slice 3: Docs + changelog - -**Goal:** Document the polished integration. - -- [ ] **3.1 Update REQUIREMENTS.md** - - Files: `docs/REQUIREMENTS.md` - - Lines: ~20 - - Details: Observability section describes service-instance-driven Grafana/Prometheus status cards and new widget kinds. - -- [ ] **3.2 Update CHANGELOG.md** - - Files: `CHANGELOG.md` - - Lines: ~15 - - Details: Added status endpoints, removed hard-coded `VITE_GRAFANA_URL` dependency, new widget kinds. - -- [ ] **3.3 Commit** - - Commit message: `feat(observability): polish Grafana/Prometheus integration` - -## Acceptance - -- `/api/monitoring/grafana-status` and `/api/monitoring/prometheus-status` return sensible JSON for configured/missing/unreachable services. -- Observability page no longer references `VITE_GRAFANA_URL`. -- New widget kinds appear in the "Add widget" flow and render data. -- Backend and frontend test suites pass. -- Docs and changelog updated. diff --git a/openspec/changes/observability-service-registry/design.md b/openspec/changes/observability-service-registry/design.md new file mode 100644 index 0000000..6beaf38 --- /dev/null +++ b/openspec/changes/observability-service-registry/design.md @@ -0,0 +1,221 @@ +# Design — Observability service registry + +**Change:** `observability-service-registry` +**Phase:** design +**Date:** 2026-06-23 + +## Current state + +- **Service registry** (`integrations/registry.py`): five types — `grafana`, + `prometheus`, `jellyfin`, `jellyseerr`, `nextcloud`, `ssh_tasks`. Grafana + (`grafana.py`: `base_url`, secret `api_key`, widget `link`) and Prometheus + (`prometheus.py`: `base_url`, secret `api_key`, widget `metric`) are already + service types. Their widget sources (`widgets/sources.py`: + `GrafanaWidgetSource`, `PrometheusWidgetSource`) already resolve URLs from the + `ServiceRecord`, not env vars. +- **Monitoring router** (`routers/monitoring.py`): `/alerts`, + `/alertmanager-status`, `/alertmanager-webhook`, `/prometheus-targets`, + `/machines`. The alertmanager endpoints read `settings.alertmanager_url` + (env). `_alertmanager_client()` returns `(requests.Session(), url)`; `_webhook_client()` + returns the forward target. +- **Config** (`config.py`): `prometheus_enabled` (Manage's own `/metrics` toggle — + stays), `prometheus_file_sd_dir`, `alertmanager_url=""`, + `alertmanager_webhook_url=""`. The latter three are removed by this change. + `services/targets.py::write_prometheus_targets` is the file-writer consuming + `prometheus_file_sd_dir`; `build_node_exporter_targets` + the + `/prometheus-targets` endpoint stay (used by external Prometheus via + `http_sd_configs`). +- **Frontend Observability page** (`ObservabilityPage.tsx`): hard-codes + `GRAFANA_BASE_URL = import.meta.env.VITE_GRAFANA_URL || "http://localhost:3000"` + and builds Node Exporter + Loki deep-links from it. Hooks + (`useObservability.ts`) call `/alerts`, `/alertmanager-status`, + `/prometheus-targets`, `/machines`. +- **Service resolution**: `settings_store.list_services(service_type)` orders by + `name ASC`; `get_service(id)`. There is **no `is_default`/primary flag**. +- **`VITE_PROMETHEUS_URL`** is referenced only in the Dockerfile/compose build + args, not in any frontend source (verified). `VITE_GRAFANA_URL` is read in + `ObservabilityPage.tsx` only. +- **Webhook relay**: `monitoring/alertmanager/alertmanager.yml` (standalone stack) + points `webhook_configs` at `http://backend:8000/api/monitoring/alertmanager-webhook`. + The receiver then optionally forwards to `ALERTMANAGER_WEBHOOK_URL`. + +## Target state + +### 1. New `alertmanager` integration type + +`integrations/alertmanager.py`, mirroring `prometheus.py`: + +```python +class AlertmanagerConfig(ServiceConfigBase): + base_url: str + timeout_seconds: int = 5 + +class AlertmanagerAlertsWidgetConfig(WidgetConfigBase): + severity_filter: str | None = None # optional: "critical", "warning", ... + +DEFINITION = ServiceDefinition( + service_type="alertmanager", + name="Alertmanager", + description="Alert routing and firing-alert summaries.", + config_model=AlertmanagerConfig, + secret_fields=[SecretField(key="api_key", label="API key", helper="Optional bearer token")], + widget_kinds=[widget_kind("active_alerts", "Active alerts", + "Firing-alert summary from this Alertmanager.", + model_cls=AlertmanagerAlertsWidgetConfig, + default_config={}, refresh_interval_ms=15_000)], +) +``` + +Register in `registry.py` (`SERVICE_DEFINITIONS`). Add `AlertmanagerWidgetSource` +to `SERVICE_ADAPTERS` in `widgets/sources.py`. + +### 2. Service resolution helper (shared) + +Add a single helper in the monitoring router used by the alertmanager endpoints +and the new grafana/prometheus status endpoints: + +```python +def _resolve_service_record( + store: SettingsStore, service_type: str, service_id: str | None +) -> ServiceRecord | None: + """Return the requested instance, else the first enabled instance of type.""" + if service_id: + row = store.get_service(service_id) + if row and row.get("service_type") == service_type and row.get("enabled", True): + return build_service_record(store, row) + return None + for row in store.list_services(service_type): + if row.get("enabled", True): + return build_service_record(store, row) + return None +``` + +This factors `build_service_record` (already in `widgets/sources.py`) and keeps +secret decryption in one place. **Default selection = first enabled instance of +type** (ordered by `name ASC`). No DB schema change. + +### 3. Rewired alertmanager endpoints + +`/alerts?service_id=` and `/alertmanager-status?service_id=` +become `Depends(get_settings_store)` endpoints: + +- Resolve the service record via `_resolve_service_record(store, "alertmanager", service_id)`. +- If `None` → return the existing not-configured bodies (`{"error": "alertmanager_not_configured"}`, `{"up": False, ...}`). +- Else build `base_url` from `service.config["base_url"]`, attach `Authorization: Bearer ` + if a secret is present, and call `/api/v1/alerts` / `/api/v2/status`. +- On exception → existing unreachable bodies. +- **Response gains `service_id` + `name`** on success so the UI can show which + instance was used. + +`_alertmanager_client()` and `_webhook_client()` (the env readers) are deleted. + +### 4. New health endpoints + +`GET /api/monitoring/grafana-status?service_id=` and +`GET /api/monitoring/prometheus-status?service_id=`: + +- Resolve via `_resolve_service_record(store, "grafana"|"prometheus", service_id)`. +- Probe Grafana `GET {base_url}/api/health` (200 → `{database: "ok"}`), Prometheus + `GET {base_url}/-/healthy` + `GET {base_url}/api/v1/status/buildinfo` (version). +- Return `{ up: bool, version: str, service_id: str, name: str, error: str | None }`. +- None configured → `{up: false, version: "", service_id: "", name: "", error: "no_service_configured"}`. +- Attach bearer token if a secret is present (some auth'd setups need it). + +**Decision — dedicated endpoints vs. widget-data path:** dedicated endpoints. +The Observability page needs page-level health for the default instance +independent of any widget instance; widget data fetch is per-widget and keyed on +a widget kind with a different shape. Reusing `_resolve_service_record` keeps the +service-resolution logic single-sourced; only the probe differs per type. + +### 5. `active_alerts` widget adapter + +`AlertmanagerWidgetSource.fetch(service, "active_alerts", config)` reuses the +existing `_summary_from_alerts` to shape `/api/v1/alerts` into the same summary +the page endpoint returns. Optional `severity_filter` limits severities. + +### 6. Webhook relay decision + +**Decision:** drop the outbound relay (`alertmanager_webhook_url` env var and the +forward `POST`). The `POST /alertmanager-webhook` receiver **stays** as +**log-only** (it records received alerts at INFO for audit/debug and returns +`{"status": "received"}`). Rationale: the inbound→outbound relay to a second +webhook is a niche feature whose target is the env var we are removing; a +log-only receiver remains useful and keeps the standalone Alertmanager config +working unchanged. + +### 7. `PROMETHEUS_FILE_SD_DIR` decision + +**Decision:** **remove it.** It is a vestige of the in-project Prometheus model +where Manage's container shared a volume with a Prometheus container and wrote +`node_exporter_targets.json` into it for `file_sd_configs`. In the +connect-to-existing model Prometheus is a separate deployment, and a shared +volume across deployments is awkward (network share / rsync hack). + +The same data is already served over HTTP by +`GET /api/monitoring/prometheus-targets`, so an external Prometheus consumes it +via standard `http_sd_configs` instead — pull-based, no shared volume, no env var: + +```yaml +# external Prometheus +- job_name: node-exporter-remote + http_sd_configs: + - url: https://manage.example.com/api/monitoring/prometheus-targets + refresh_interval: 30s +``` + +Concretely in this change: delete the `prometheus_file_sd_dir` config field, the +`write_prometheus_targets()` **file-writer** call sites (startup + machine +create/update/delete hooks), and the `PROMETHEUS_FILE_SD_DIR` compose/Dockerfile +mounts. **Keep** `services/targets.py::build_node_exporter_targets` (the data +builder) and the `/api/monitoring/prometheus-targets` HTTP endpoint. + +`PROMETHEUS_ENABLED` **stays** as an env var — it toggles Manage's own +`/metrics` endpoint, which is app instrumentation (Role 1), not service +integration. It is the single legitimate observability env survivor. + +## Frontend changes + +1. **Observability page** (`ObservabilityPage.tsx`): + - Discover grafana services via `useServiceInstances("grafana")`; pick the + first enabled; build `GRAFANA_BASE_URL` from its `base_url`. Remove the + `import.meta.env.VITE_GRAFANA_URL` read. Empty-state when none configured + (link to `/services`). + - Add `useGrafanaStatus()` and `usePrometheusStatus()` hooks; add Grafana and + Prometheus `HealthCard`s alongside Alertmanager. +2. **Hooks/client** (`useObservability.ts`, `api/client.ts`): + - `fetchGrafanaStatus(serviceId?)`, `fetchPrometheusStatus(serviceId?)`, + `useGrafanaStatus`, `usePrometheusStatus`. + - Existing `fetchAlertmanagerAlerts` / `fetchAlertmanagerStatus` keep their + signatures (no service id needed for the page's default instance). +3. **Widget registry** (`integrations/registry.ts`): add `alertmanager` binding + with the `active_alerts` kind → new `AlertmanagerAlertsWidget` component. +4. **Types** (`types/index.ts`): `GrafanaStatus`, `PrometheusStatus`, + `AlertmanagerAlertsWidgetConfig`; add `service_id`/`name` to status shapes. +5. **`VITE_PROMETHEUS_URL`** is unused in source — only its removal from the + Dockerfile/compose build args is needed. + +## Data model / compatibility + +- No DB schema change. Alertmanager instances live in the existing `services` + table as `service_type="alertmanager"`. +- **Breaking for `ALERTMANAGER_URL` users:** after upgrade the env var is ignored; + the Alertmanager instance must be (re)created on the Services page. Document in + CHANGELOG. No data migration (it was an env var, not a DB row). +- **Type-contract cleanup (pre-existing drift):** the down-branch + `/alertmanager-status` bodies omit `name`/`peers`. The frontend `AlertmanagerStatus` + type declares them required but never reads them. Fix by adding `name: ""`, + `peers: []` to the not-configured/unreachable branches (and `service_id`) so + the response is internally consistent. + +## Open questions + +1. **Default-instance flag.** v1 uses "first enabled instance of type" (by name). + If multiple Grafana/Prometheus/Alertmanager instances exist, status reflects + only the first. A per-type "primary" flag is deferred — confirm this is + acceptable, or whether to add the flag in this change (adds a UI affordance + + DB column; would grow scope past one slice). +2. **Service-id in page endpoints.** Should `/alerts?service_id=` be surfaced in + the Observability page UI (an instance selector), or kept backend-only for + future use? v1 leaves the page on the default instance. +3. **`.env.example`** is assistant-edit-blocked by safety policy. The four env + vars must be removed manually by the user; tracked as a task follow-up. diff --git a/openspec/changes/observability-service-registry/proposal.md b/openspec/changes/observability-service-registry/proposal.md new file mode 100644 index 0000000..6ef3ccb --- /dev/null +++ b/openspec/changes/observability-service-registry/proposal.md @@ -0,0 +1,118 @@ +# Proposal — Observability service registry + +**Change:** `observability-service-registry` +**Phase:** proposal +**Date:** 2026-06-23 + +## Problem + +Grafana and Prometheus are already modeled as **service-registry service types** +with dashboard widgets (`GrafanaLinkWidget`, `PrometheusMetricWidget`) whose +adapters resolve URLs from service records — not env vars. The Observability +page, however, still treats observability as an env-configured afterthought: + +1. **Alertmanager is not a service type.** It has no `integrations/alertmanager.py` + definition. Its connection URL comes from the `ALERTMANAGER_URL` **environment + variable** (`config.py`), read directly by the monitoring router. This is the + only observability component still env-bound, and it cannot be configured, + named, multi-instanced, or surfaced as a widget like Grafana/Prometheus. +2. **The Observability page hard-codes `VITE_GRAFANA_URL`** + (`ObservabilityPage.tsx` → `GRAFANA_BASE_URL = import.meta.env.VITE_GRAFANA_URL`). + It ignores any Grafana service instances configured in the registry, so the + deep-links break unless the env var is also set. +3. **No Grafana/Prometheus health card.** The page shows Alertmanager health and + Prometheus *targets*, but never reports whether the configured Grafana or + Prometheus instance is actually reachable. +4. **Two env-bound relay knobs** with no service-registry equivalent: + `ALERTMANAGER_WEBHOOK_URL` (the router forwards received webhooks onward) and + the leftover `VITE_PROMETHEUS_URL` build arg. + +The result is a split-brain model: Grafana/Prometheus are first-class registry +services, but Alertmanager is an env var and the page reads a different env var +for Grafana than the registry holds. + +## Proposal + +Make **all** observability integration (Alertmanager, Grafana, Prometheus) a +first-class, UI-configured part of the existing service registry, with a +dedicated Observability page and dashboard widgets — and **zero observability +service env vars**. + +1. **Add `alertmanager` as a 6th service integration type** + (`integrations/alertmanager.py`): config (`base_url`, `timeout_seconds`), + optional secret (`api_key`/bearer token for auth'd instances), and an + `active_alerts` widget kind. Register it in `integrations/registry.py`, + following the exact pattern of `grafana.py`/`prometheus.py`. +2. **Rewire the monitoring router to resolve Alertmanager from a service record** + instead of `config.alertmanager_url`. `/alerts` and `/alertmanager-status` + accept an optional `?service_id=` and default to the first enabled + `alertmanager` instance. Graceful "not configured" responses are preserved + when none exists. +3. **Add health/status endpoints** for Grafana (`/api/monitoring/grafana-status`) + and Prometheus (`/api/monitoring/prometheus-status`) that resolve a service + record and probe its health endpoint, returning `{ up, version, service_id, + name, error? }`. +4. **Observability page discovers services** from the registry. Grafana deep-links + come from the discovered instance's `base_url`; `VITE_GRAFANA_URL` is removed. + Health cards for Alertmanager, Grafana, and Prometheus are shown. +5. **Add an `active_alerts` Alertmanager dashboard widget** (backend adapter + + frontend registry + component), mirroring the Grafana/Prometheus widgets. +6. **Delete observability service env vars** entirely: backend + `alertmanager_url`, `alertmanager_webhook_url`; frontend `VITE_GRAFANA_URL`, + `VITE_PROMETHEUS_URL`. Drop the inbound→outbound webhook relay (the receiver + stays, log-only). + +## Non-goals + +- Do **not** build an in-app metric explorer or charting library. Manage stays a + thin dashboard: deep-links/iframe to Grafana only. +- Do **not** introduce a "primary/default instance" flag on service records in + this change. Default selection is "first enabled instance of type" (see Open + questions). A default flag is a cross-cutting registry feature deferred to a + later change. +- Do **not** remove the Prometheus target **data** path: `build_node_exporter_targets` + and the `/prometheus-targets` HTTP endpoint stay. Only the **file writer** + (`write_prometheus_targets` + `prometheus_file_sd_dir`) is removed; external + Prometheus uses `http_sd_configs` against the endpoint instead. +- Do **not** persist Prometheus query history or alert acknowledgements in Manage. +- Do **not** migrate the existing `/machines` + Node Exporter target model; it is + orthogonal to service instances. + +## Acceptance + +- `alertmanager` appears in `GET /api/services/types` and can be + created/edited/deleted like the other service types. +- `ALERTMANAGER_URL`, `ALERTMANAGER_WEBHOOK_URL`, `VITE_GRAFANA_URL`, and + `VITE_PROMETHEUS_URL` have **no remaining references** in backend source, + frontend source, or the root compose files. +- `/alerts` and `/alertmanager-status` work against a configured alertmanager + service instance and return the existing "not configured" shape when none is + configured. +- `/api/monitoring/grafana-status` and `/prometheus-status` return sensible JSON + for configured / missing / unreachable services. +- The Observability page renders Alertmanager, Grafana, and Prometheus health + cards and builds Grafana deep-links from the discovered service instance. +- An `active_alerts` widget can be added to the dashboard bound to an + alertmanager service and renders firing alerts. +- Backend `pytest` + `ruff` green; frontend `npm run lint`, `npm run build` + (tsc -b + vite build), `npm run test` green. +- `docs/REQUIREMENTS.md` decision log, `CHANGELOG.md`, and `backend/README.md` + updated. + +## Impact + +- **Backend:** new integration module; monitoring router rewire + two new + endpoints; config.py loses three fields (`alertmanager_url`, + `alertmanager_webhook_url`, `prometheus_file_sd_dir`) and the file-writer call + sites; widget adapter + registry. +- **Frontend:** Observability page rewire (service discovery, new health cards, + Grafana URL source); new alertmanager widget component + registry entry; types; + removal of two `import.meta.env` reads. +- **Config/deploy:** root compose files and Dockerfile build args lose the + observability service env vars and the `prometheus-file-sd` volume mount; + `.env.example` needs a manual update (assistant-edit-blocked — tracked as a + task follow-up). +- **Compatibility:** `ALERTMANAGER_URL` users must re-create their Alertmanager + as a service instance after upgrading; admins using `file_sd_configs` against + the shared volume must switch to `http_sd_configs` against + `/api/monitoring/prometheus-targets`. Both breaking, called out in CHANGELOG. diff --git a/openspec/changes/observability-service-registry/tasks.md b/openspec/changes/observability-service-registry/tasks.md new file mode 100644 index 0000000..211c43b --- /dev/null +++ b/openspec/changes/observability-service-registry/tasks.md @@ -0,0 +1,274 @@ +# Tasks — Observability service registry + +**Change:** `observability-service-registry` +**Phase:** tasks +**Date:** 2026-06-23 + +## Review workload forecast + +| Field | Value | +|-------|-------| +| Estimated changed lines | ~790 | +| Chained PRs recommended | Yes (5 slices) | +| Chain strategy | stacked-to-main | +| Slice order | 1 (alertmanager type) → 2 (router rewire + status) → 3 (frontend) → 4 (drop file-SD writer) → 5 (env removal + docs) | + +Each slice is committed separately (user pref). Every slice must leave +`cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest` **and** +`cd frontend && npm run lint && npm run build && npm run test` green. + +--- + +## Slice 1: Backend — add `alertmanager` service type + widget + +**Goal:** Alertmanager becomes a first-class registry service with an +`active_alerts` widget, mirroring Grafana/Prometheus. + +- [ ] **1.1 Add integration module** + - Files: `backend/src/media_library_viewer_api/integrations/alertmanager.py` (new) + - Lines: ~45 + - Details: `AlertmanagerConfig` (`base_url`, `timeout_seconds=5`), + `AlertmanagerAlertsWidgetConfig` (`severity_filter` optional), + `DEFINITION` with `service_type="alertmanager"`, secret `api_key`, widget kind + `active_alerts`. Follow `prometheus.py` exactly. + +- [ ] **1.2 Register the type** + - Files: `backend/src/media_library_viewer_api/integrations/registry.py` + - Lines: ~2 + - Details: import `DEFINITION as ALERTMANAGER`; add to `SERVICE_DEFINITIONS`. + +- [ ] **1.3 Add widget source adapter** + - Files: `backend/src/media_library_viewer_api/widgets/sources.py` + - Lines: ~40 + - Details: `AlertmanagerWidgetSource` — reuse `_summary_from_alerts` (move it to + a shared import or keep in monitoring router and import). `fetch` calls + `{base_url}/api/v1/alerts` with optional bearer token, optional + `severity_filter`, returns the summary shape. Register in `SERVICE_ADAPTERS`. + +- [ ] **1.4 Update backend tests** + - Files: `backend/tests/test_services.py`, `backend/tests/test_widgets.py` + - Lines: ~50 + - Details: assert `alertmanager` in service types (test count becomes 6); + test the adapter with mocked HTTP (alerts summary) and missing service. + +- [ ] **1.5 Verify backend** + - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest` + +- [ ] **1.6 Commit** + - Message: `feat(observability): add alertmanager service type and widget` + +**Slice 1 total:** ~140 changed lines. + +--- + +## Slice 2: Backend — rewire monitoring router + health endpoints + +**Goal:** Alertmanager is resolved from a service record; Grafana/Prometheus +health endpoints added; webhook relay dropped. + +- [ ] **2.1 Add service resolution helper** + - Files: `backend/src/media_library_viewer_api/routers/monitoring.py` + - Lines: ~20 + - Details: `_resolve_service_record(store, service_type, service_id)` returning + the requested or first-enabled `ServiceRecord` (reuses + `build_service_record` from `widgets/sources.py`). Delete + `_alertmanager_client`/`_webhook_client` env readers. + +- [ ] **2.2 Rewire `/alerts` and `/alertmanager-status`** + - Files: `backend/src/media_library_viewer_api/routers/monitoring.py` + - Lines: ~40 + - Details: `Depends(get_settings_store)`; optional `service_id` query param; + resolve via helper; attach bearer token; on `None` return not-configured body; + add `service_id` + `name` to success responses; add `name`/`peers` to + down-branches (fix type drift). + +- [ ] **2.3 Make webhook receiver log-only** + - Files: `backend/src/media_library_viewer_api/routers/monitoring.py` + - Lines: ~-15 (net delete) + - Details: remove the outbound forward `POST` + `_webhook_client`; keep the + receiver logging received alerts and returning `{"status": "received"}`. + +- [ ] **2.4 Add grafana/prometheus status endpoints** + - Files: `backend/src/media_library_viewer_api/routers/monitoring.py` + - Lines: ~60 + - Details: `GET /api/monitoring/grafana-status?service_id=` (probe + `/api/health`), `GET /api/monitoring/prometheus-status?service_id=` (probe + `/-/healthy` + `/api/v1/status/buildinfo`). Return + `{ up, version, service_id, name, error? }`; none-configured → + `error: "no_service_configured"`. + +- [ ] **2.5 Update backend tests** + - Files: `backend/tests/test_api.py` + - Lines: ~70 + - Details: rewire existing `TestAlertmanager` to seed an alertmanager service + instance instead of mocking `_alertmanager_client`; add not-configured (no + instance) cases; add `TestGrafanaStatus` / `TestPrometheusStatus` (configured, + missing, unreachable); assert webhook receiver is log-only. + +- [ ] **2.6 Verify backend** + - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest` + +- [ ] **2.7 Commit** + - Message: `feat(observability): resolve services from registry, add health endpoints` + +**Slice 2 total:** ~210 changed lines. + +--- + +## Slice 3: Frontend — service discovery, health cards, alertmanager widget + +**Goal:** Observability page discovers services; `VITE_*` URL reads removed; +alertmanager widget added. (If this slice exceeds ~400 lines, split 3a page / +3b widget — see note.) + +- [ ] **3.1 Add status hooks + client fns** + - Files: `frontend/src/api/client.ts`, `frontend/src/hooks/useObservability.ts` + - Lines: ~30 + - Details: `fetchGrafanaStatus(serviceId?)`, `fetchPrometheusStatus(serviceId?)`, + `useGrafanaStatus`, `usePrometheusStatus`. + +- [ ] **3.2 Rewire Observability page** + - Files: `frontend/src/components/ObservabilityPage.tsx` + - Lines: ~90 + - Details: `useServiceInstances("grafana")` → derive `GRAFANA_BASE_URL` from + first enabled instance; remove `import.meta.env.VITE_GRAFANA_URL`; empty-state + linking to `/services` when none; add Grafana + Prometheus `HealthCard`s. + +- [ ] **3.3 Add alertmanager widget** + - Files: `frontend/src/widgets/AlertmanagerAlertsWidget.tsx` (new), + `frontend/src/integrations/registry.ts`, `frontend/src/widgets/index.ts` + - Lines: ~90 + - Details: presentational component reusing the alert summary shape; register + `alertmanager` binding with `active_alerts` kind. + +- [ ] **3.4 Update types** + - Files: `frontend/src/types/index.ts` + - Lines: ~20 + - Details: `GrafanaStatus`, `PrometheusStatus`; add `service_id`/`name` to + status types; `AlertmanagerAlertsWidgetConfig`. + +- [ ] **3.5 Update frontend tests** + - Files: `frontend/src/integrations/registry.test.ts`, widget/page tests as needed + - Lines: ~40 + - Details: assert alertmanager binding resolves; mock status hooks. + +- [ ] **3.6 Verify frontend** + - Run: `cd frontend && npm run lint && npm run build && npm run test` + +- [ ] **3.7 Commit** + - Message: `feat(observability): service discovery, health cards, alertmanager widget` + +**Slice 3 total:** ~270 changed lines. + +--- + +## Slice 4: Backend — remove file-SD writer + +**Goal:** Drop the shared-file Prometheus bridge (`PROMETHEUS_FILE_SD_DIR`); +external Prometheus uses `http_sd_configs` against the existing endpoint. + +- [ ] **4.1 Remove the file-writer call sites** + - Files: `backend/src/media_library_viewer_api/main.py` (startup call), + `backend/src/media_library_viewer_api/routers/settings.py` + (post machine create/update/delete calls) + - Lines: ~-15 + - Details: delete `write_prometheus_targets(...)` invocations; remove the + import. **Keep** `services/targets.py::build_node_exporter_targets` and the + `GET /api/monitoring/prometheus-targets` endpoint (external Prometheus + consumes via `http_sd_configs`). + +- [ ] **4.2 Remove the file-writer + config field** + - Files: `backend/src/media_library_viewer_api/services/targets.py`, + `backend/src/media_library_viewer_api/config.py` + - Lines: ~-30 + - Details: delete `write_prometheus_targets` (the writer) from `targets.py`, + leaving `build_node_exporter_targets`; delete `prometheus_file_sd_dir` from + `config.py`. Prune now-unused imports (`Path`, etc.). + +- [ ] **4.3 Remove compose mount + env var** + - Files: `docker-compose.yml`, `docker-compose.dev.yml` + - Lines: ~-4 + - Details: drop `PROMETHEUS_FILE_SD_DIR` backend env var and any + `prometheus-file-sd` volume mount/bind reference (no Prometheus container + remains to read it). + +- [ ] **4.4 Update backend tests** + - Files: `backend/tests/test_targets.py`, `backend/tests/test_api.py` + - Lines: ~-20 / ~+5 + - Details: drop tests covering `write_prometheus_targets`; keep/extend tests + for `build_node_exporter_targets` and the `/prometheus-targets` endpoint. + Drop any `write_targets.assert_called_once` assertion in + `TestSettingsMachines`. + +- [ ] **4.5 Verify backend** + - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest` + +- [ ] **4.6 Commit** + - Message: `refactor(observability): drop file-SD writer for http_sd_configs` + +**Slice 4 total:** ~70 changed lines (net negative). + +--- + +## Slice 5: Env removal + docs + +**Goal:** Delete the remaining observability service env vars from config/compose +and document the change. + +- [ ] **5.1 Remove backend env fields** + - Files: `backend/src/media_library_viewer_api/config.py` + - Lines: ~-2 + - Details: delete `alertmanager_url` and `alertmanager_webhook_url` fields. + +- [ ] **5.2 Remove compose / Dockerfile build args** + - Files: `docker-compose.yml`, `docker-compose.dev.yml`, `frontend/Dockerfile` + - Lines: ~-8 + - Details: drop `ALERTMANAGER_URL`, `ALERTMANAGER_WEBHOOK_URL` from backend + env; drop `VITE_GRAFANA_URL`, `VITE_PROMETHEUS_URL` from frontend build args + (both compose files) and `ARG`/`ENV` (Dockerfile). + +- [ ] **5.3 Update docs** + - Files: `docs/REQUIREMENTS.md`, `CHANGELOG.md`, `backend/README.md`, + `docs/monitoring-logging-design.md` + - Lines: ~60 + - Details: REQUIREMENTS decision-log entry (alertmanager as a service type; + observability env vars removed incl. `PROMETHEUS_FILE_SD_DIR`; http_sd_configs + replaces the file bridge; thin-dashboard model unchanged); CHANGELOG + added/changed/**BREAKING** (`ALERTMANAGER_URL` users re-create the instance; + `file_sd_configs` users switch to `http_sd_configs`); backend README + monitoring section. + +- [ ] **5.4 Manual follow-up (assistant-blocked) — `.env.example`** + - `.env.example` is blocked by the safety policy. Note for the user: remove + `ALERTMANAGER_URL`, `ALERTMANAGER_WEBHOOK_URL`, `VITE_GRAFANA_URL`, + `VITE_PROMETHEUS_URL`, and `PROMETHEUS_FILE_SD_DIR`; keep `PROMETHEUS_ENABLED`. + - No code lines. + +- [ ] **5.5 Verify** + - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest` + - Run: `cd frontend && npm run lint && npm run build && npm run test` + - Grep-gate: confirm no remaining references to the removed env vars in + `backend/src`, `frontend/src`, root compose files, Dockerfile. + +- [ ] **5.6 Commit** + - Message: `chore(observability): remove remaining observability env vars, docs` + +**Slice 5 total:** ~110 changed lines. + +--- + +## Acceptance + +- `alertmanager` is a service type; observability services are UI-configured only. +- The observability service env vars (`ALERTMANAGER_URL`, + `ALERTMANAGER_WEBHOOK_URL`, `VITE_GRAFANA_URL`, `VITE_PROMETHEUS_URL`, + `PROMETHEUS_FILE_SD_DIR`) have no remaining references. `PROMETHEUS_ENABLED` + (Manage's own `/metrics` toggle) is the only observability env var that remains. +- `/alerts`, `/alertmanager-status`, `/grafana-status`, `/prometheus-status` work + against service instances with graceful missing/unreachable states. +- Observability page shows Alertmanager/Grafana/Prometheus health and derives + Grafana links from the registry (no `VITE_*` URL). +- `active_alerts` alertmanager widget renders on the dashboard. +- External Prometheus consumes node-exporter targets via `http_sd_configs` + against `/api/monitoring/prometheus-targets`; no shared volume remains. +- All backend and frontend test suites green; docs + CHANGELOG updated.