manage/openspec/changes/archive/observability-service-registry/tasks.md

Tasks — Observability service registry

Change: observability-service-registry Phase: tasks Date: 2026-06-23

Review workload forecast

Field	Value
Estimated changed lines	~790
Chained PRs recommended	Yes (5 slices)
Chain strategy	stacked-to-main
Slice order	1 (alertmanager type) → 2 (router rewire + status) → 3 (frontend) → 4 (drop file-SD writer) → 5 (env removal + docs)

Each slice is committed separately (user pref). Every slice must leave cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest and cd frontend && npm run lint && npm run build && npm run test green.

---

Slice 1: Backend — add alertmanager service type + widget

Goal: Alertmanager becomes a first-class registry service with an active_alerts widget, mirroring Grafana/Prometheus.

• 1.1 Add integration module

  • Files: backend/src/media_library_viewer_api/integrations/alertmanager.py (new)
  • Lines: ~45
  • Details: AlertmanagerConfig (base_url, timeout_seconds=5), AlertmanagerAlertsWidgetConfig (severity_filter optional), DEFINITION with service_type="alertmanager", secret api_key, widget kind active_alerts. Follow prometheus.py exactly.

• 1.2 Register the type

  • Files: backend/src/media_library_viewer_api/integrations/registry.py
  • Lines: ~2
  • Details: import DEFINITION as ALERTMANAGER; add to SERVICE_DEFINITIONS.

• 1.3 Add widget source adapter

  • Files: backend/src/media_library_viewer_api/widgets/sources.py
  • Lines: ~40
  • Details: AlertmanagerWidgetSource — reuse _summary_from_alerts (move it to a shared import or keep in monitoring router and import). fetch calls {base_url}/api/v1/alerts with optional bearer token, optional severity_filter, returns the summary shape. Register in SERVICE_ADAPTERS.

• 1.4 Update backend tests

  • Files: backend/tests/test_services.py, backend/tests/test_widgets.py
  • Lines: ~50
  • Details: assert alertmanager in service types (test count becomes 6); test the adapter with mocked HTTP (alerts summary) and missing service.

• 1.5 Verify backend

  • Run: cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest

• 1.6 Commit

  • Message: feat(observability): add alertmanager service type and widget

Slice 1 total: ~140 changed lines.

---

Slice 2: Backend — rewire monitoring router + health endpoints

Goal: Alertmanager is resolved from a service record; Grafana/Prometheus health endpoints added; webhook relay dropped.

• 2.1 Add service resolution helper

  • Files: backend/src/media_library_viewer_api/routers/monitoring.py
  • Lines: ~20
  • Details: _resolve_service_record(store, service_type, service_id) returning the requested or first-enabled ServiceRecord (reuses build_service_record from widgets/sources.py). Delete _alertmanager_client/_webhook_client env readers.

• 2.2 Rewire /alerts and /alertmanager-status

  • Files: backend/src/media_library_viewer_api/routers/monitoring.py
  • Lines: ~40
  • Details: Depends(get_settings_store); optional service_id query param; resolve via helper; attach bearer token; on None return not-configured body; add service_id + name to success responses; add name/peers to down-branches (fix type drift).

• 2.3 Make webhook receiver log-only

  • Files: backend/src/media_library_viewer_api/routers/monitoring.py
  • Lines: ~-15 (net delete)
  • Details: remove the outbound forward POST + _webhook_client; keep the receiver logging received alerts and returning {"status": "received"}.

• 2.4 Add grafana/prometheus status endpoints

  • Files: backend/src/media_library_viewer_api/routers/monitoring.py
  • Lines: ~60
  • Details: GET /api/monitoring/grafana-status?service_id= (probe /api/health), GET /api/monitoring/prometheus-status?service_id= (probe /-/healthy + /api/v1/status/buildinfo). Return { up, version, service_id, name, error? }; none-configured → error: "no_service_configured".

• 2.5 Update backend tests

  • Files: backend/tests/test_api.py
  • Lines: ~70
  • Details: rewire existing TestAlertmanager to seed an alertmanager service instance instead of mocking _alertmanager_client; add not-configured (no instance) cases; add TestGrafanaStatus / TestPrometheusStatus (configured, missing, unreachable); assert webhook receiver is log-only.

• 2.6 Verify backend

  • Run: cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest

• 2.7 Commit

  • Message: feat(observability): resolve services from registry, add health endpoints

Slice 2 total: ~210 changed lines.

---

Slice 3: Frontend — service discovery, health cards, alertmanager widget

Goal: Observability page discovers services; VITE_* URL reads removed; alertmanager widget added. (If this slice exceeds ~400 lines, split 3a page / 3b widget — see note.)

• 3.1 Add status hooks + client fns

  • Files: frontend/src/api/client.ts, frontend/src/hooks/useObservability.ts
  • Lines: ~30
  • Details: fetchGrafanaStatus(serviceId?), fetchPrometheusStatus(serviceId?), useGrafanaStatus, usePrometheusStatus.

• 3.2 Rewire Observability page

  • Files: frontend/src/components/ObservabilityPage.tsx
  • Lines: ~90
  • Details: useServiceInstances("grafana") → derive GRAFANA_BASE_URL from first enabled instance; remove import.meta.env.VITE_GRAFANA_URL; empty-state linking to /services when none; add Grafana + Prometheus HealthCards.

• 3.3 Add alertmanager widget

  • Files: frontend/src/widgets/AlertmanagerAlertsWidget.tsx (new), frontend/src/integrations/registry.ts, frontend/src/widgets/index.ts
  • Lines: ~90
  • Details: presentational component reusing the alert summary shape; register alertmanager binding with active_alerts kind.

• 3.4 Update types

  • Files: frontend/src/types/index.ts
  • Lines: ~20
  • Details: GrafanaStatus, PrometheusStatus; add service_id/name to status types; AlertmanagerAlertsWidgetConfig.

• 3.5 Update frontend tests

  • Files: frontend/src/integrations/registry.test.ts, widget/page tests as needed
  • Lines: ~40
  • Details: assert alertmanager binding resolves; mock status hooks.

• 3.6 Verify frontend

  • Run: cd frontend && npm run lint && npm run build && npm run test

• 3.7 Commit

  • Message: feat(observability): service discovery, health cards, alertmanager widget

Slice 3 total: ~270 changed lines.

---

Slice 4: Backend — remove file-SD writer

Goal: Drop the shared-file Prometheus bridge (PROMETHEUS_FILE_SD_DIR); external Prometheus uses http_sd_configs against the existing endpoint.

• 4.1 Remove the file-writer call sites

  • Files: backend/src/media_library_viewer_api/main.py (startup call), backend/src/media_library_viewer_api/routers/settings.py (post machine create/update/delete calls)
  • Lines: ~-15
  • Details: delete write_prometheus_targets(...) invocations; remove the import. Keep services/targets.py::build_node_exporter_targets and the GET /api/monitoring/prometheus-targets endpoint (external Prometheus consumes via http_sd_configs).

• 4.2 Remove the file-writer + config field

  • Files: backend/src/media_library_viewer_api/services/targets.py, backend/src/media_library_viewer_api/config.py
  • Lines: ~-30
  • Details: delete write_prometheus_targets (the writer) from targets.py, leaving build_node_exporter_targets; delete prometheus_file_sd_dir from config.py. Prune now-unused imports (Path, etc.).

• 4.3 Remove compose mount + env var

  • Files: docker-compose.yml, docker-compose.dev.yml
  • Lines: ~-4
  • Details: drop PROMETHEUS_FILE_SD_DIR backend env var and any prometheus-file-sd volume mount/bind reference (no Prometheus container remains to read it).

• 4.4 Update backend tests

  • Files: backend/tests/test_targets.py, backend/tests/test_api.py
  • Lines: ~-20 / ~+5
  • Details: drop tests covering write_prometheus_targets; keep/extend tests for build_node_exporter_targets and the /prometheus-targets endpoint. Drop any write_targets.assert_called_once assertion in TestSettingsMachines.

• 4.5 Verify backend

  • Run: cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest

• 4.6 Commit

  • Message: refactor(observability): drop file-SD writer for http_sd_configs

Slice 4 total: ~70 changed lines (net negative).

---

Slice 5: Env removal + docs

Goal: Delete the remaining observability service env vars from config/compose and document the change.

• 5.1 Remove backend env fields

  • Files: backend/src/media_library_viewer_api/config.py
  • Lines: ~-2
  • Details: delete alertmanager_url and alertmanager_webhook_url fields.

• 5.2 Remove compose / Dockerfile build args

  • Files: docker-compose.yml, docker-compose.dev.yml, frontend/Dockerfile
  • Lines: ~-8
  • Details: drop ALERTMANAGER_URL, ALERTMANAGER_WEBHOOK_URL from backend env; drop VITE_GRAFANA_URL, VITE_PROMETHEUS_URL from frontend build args (both compose files) and ARG/ENV (Dockerfile).

• 5.3 Update docs

  • Files: docs/REQUIREMENTS.md, CHANGELOG.md, backend/README.md, docs/monitoring-logging-design.md
  • Lines: ~60
  • Details: REQUIREMENTS decision-log entry (alertmanager as a service type; observability env vars removed incl. PROMETHEUS_FILE_SD_DIR; http_sd_configs replaces the file bridge; thin-dashboard model unchanged); CHANGELOG added/changed/BREAKING (ALERTMANAGER_URL users re-create the instance; file_sd_configs users switch to http_sd_configs); backend README monitoring section.

• 5.4 Manual follow-up (assistant-blocked) — .env.example

  • .env.example is blocked by the safety policy. Note for the user: remove ALERTMANAGER_URL, ALERTMANAGER_WEBHOOK_URL, VITE_GRAFANA_URL, VITE_PROMETHEUS_URL, and PROMETHEUS_FILE_SD_DIR; keep PROMETHEUS_ENABLED.
  • No code lines.

• 5.5 Verify

  • Run: cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest
  • Run: cd frontend && npm run lint && npm run build && npm run test
  • Grep-gate: confirm no remaining references to the removed env vars in backend/src, frontend/src, root compose files, Dockerfile.

• 5.6 Commit

  • Message: chore(observability): remove remaining observability env vars, docs

Slice 5 total: ~110 changed lines.

---

Acceptance

• alertmanager is a service type; observability services are UI-configured only.
• The observability service env vars (ALERTMANAGER_URL, ALERTMANAGER_WEBHOOK_URL, VITE_GRAFANA_URL, VITE_PROMETHEUS_URL, PROMETHEUS_FILE_SD_DIR) have no remaining references. PROMETHEUS_ENABLED (Manage's own /metrics toggle) is the only observability env var that remains.
• /alerts, /alertmanager-status, /grafana-status, /prometheus-status work against service instances with graceful missing/unreachable states.
• Observability page shows Alertmanager/Grafana/Prometheus health and derives Grafana links from the registry (no VITE_* URL).
• active_alerts alertmanager widget renders on the dashboard.
• External Prometheus consumes node-exporter targets via http_sd_configs against /api/monitoring/prometheus-targets; no shared volume remains.
• All backend and frontend test suites green; docs + CHANGELOG updated.