manage/openspec/changes/archive/observability-service-registry/tasks.md

# Tasks — Observability service registry

**Change:** `observability-service-registry`
**Phase:** tasks
**Date:** 2026-06-23

## Review workload forecast

| Field | Value |
|-------|-------|
| Estimated changed lines | ~790 |
| Chained PRs recommended | Yes (5 slices) |
| Chain strategy | stacked-to-main |
| Slice order | 1 (alertmanager type) → 2 (router rewire + status) → 3 (frontend) → 4 (drop file-SD writer) → 5 (env removal + docs) |

Each slice is committed separately (user pref). Every slice must leave
`cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest` **and**
`cd frontend && npm run lint && npm run build && npm run test` green.

---

## Slice 1: Backend — add `alertmanager` service type + widget

**Goal:** Alertmanager becomes a first-class registry service with an
`active_alerts` widget, mirroring Grafana/Prometheus.

- [ ] **1.1 Add integration module**
  - Files: `backend/src/media_library_viewer_api/integrations/alertmanager.py` (new)
  - Lines: ~45
  - Details: `AlertmanagerConfig` (`base_url`, `timeout_seconds=5`),
    `AlertmanagerAlertsWidgetConfig` (`severity_filter` optional),
    `DEFINITION` with `service_type="alertmanager"`, secret `api_key`, widget kind
    `active_alerts`. Follow `prometheus.py` exactly.

- [ ] **1.2 Register the type**
  - Files: `backend/src/media_library_viewer_api/integrations/registry.py`
  - Lines: ~2
  - Details: import `DEFINITION as ALERTMANAGER`; add to `SERVICE_DEFINITIONS`.

- [ ] **1.3 Add widget source adapter**
  - Files: `backend/src/media_library_viewer_api/widgets/sources.py`
  - Lines: ~40
  - Details: `AlertmanagerWidgetSource` — reuse `_summary_from_alerts` (move it to
    a shared import or keep in monitoring router and import). `fetch` calls
    `{base_url}/api/v1/alerts` with optional bearer token, optional
    `severity_filter`, returns the summary shape. Register in `SERVICE_ADAPTERS`.

- [ ] **1.4 Update backend tests**
  - Files: `backend/tests/test_services.py`, `backend/tests/test_widgets.py`
  - Lines: ~50
  - Details: assert `alertmanager` in service types (test count becomes 6);
    test the adapter with mocked HTTP (alerts summary) and missing service.

- [ ] **1.5 Verify backend**
  - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest`

- [ ] **1.6 Commit**
  - Message: `feat(observability): add alertmanager service type and widget`

**Slice 1 total:** ~140 changed lines.

---

## Slice 2: Backend — rewire monitoring router + health endpoints

**Goal:** Alertmanager is resolved from a service record; Grafana/Prometheus
health endpoints added; webhook relay dropped.

- [ ] **2.1 Add service resolution helper**
  - Files: `backend/src/media_library_viewer_api/routers/monitoring.py`
  - Lines: ~20
  - Details: `_resolve_service_record(store, service_type, service_id)` returning
    the requested or first-enabled `ServiceRecord` (reuses
    `build_service_record` from `widgets/sources.py`). Delete
    `_alertmanager_client`/`_webhook_client` env readers.

- [ ] **2.2 Rewire `/alerts` and `/alertmanager-status`**
  - Files: `backend/src/media_library_viewer_api/routers/monitoring.py`
  - Lines: ~40
  - Details: `Depends(get_settings_store)`; optional `service_id` query param;
    resolve via helper; attach bearer token; on `None` return not-configured body;
    add `service_id` + `name` to success responses; add `name`/`peers` to
    down-branches (fix type drift).

- [ ] **2.3 Make webhook receiver log-only**
  - Files: `backend/src/media_library_viewer_api/routers/monitoring.py`
  - Lines: ~-15 (net delete)
  - Details: remove the outbound forward `POST` + `_webhook_client`; keep the
    receiver logging received alerts and returning `{"status": "received"}`.

- [ ] **2.4 Add grafana/prometheus status endpoints**
  - Files: `backend/src/media_library_viewer_api/routers/monitoring.py`
  - Lines: ~60
  - Details: `GET /api/monitoring/grafana-status?service_id=` (probe
    `/api/health`), `GET /api/monitoring/prometheus-status?service_id=` (probe
    `/-/healthy` + `/api/v1/status/buildinfo`). Return
    `{ up, version, service_id, name, error? }`; none-configured →
    `error: "no_service_configured"`.

- [ ] **2.5 Update backend tests**
  - Files: `backend/tests/test_api.py`
  - Lines: ~70
  - Details: rewire existing `TestAlertmanager` to seed an alertmanager service
    instance instead of mocking `_alertmanager_client`; add not-configured (no
    instance) cases; add `TestGrafanaStatus` / `TestPrometheusStatus` (configured,
    missing, unreachable); assert webhook receiver is log-only.

- [ ] **2.6 Verify backend**
  - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest`

- [ ] **2.7 Commit**
  - Message: `feat(observability): resolve services from registry, add health endpoints`

**Slice 2 total:** ~210 changed lines.

---

## Slice 3: Frontend — service discovery, health cards, alertmanager widget

**Goal:** Observability page discovers services; `VITE_*` URL reads removed;
alertmanager widget added. (If this slice exceeds ~400 lines, split 3a page /
3b widget — see note.)

- [ ] **3.1 Add status hooks + client fns**
  - Files: `frontend/src/api/client.ts`, `frontend/src/hooks/useObservability.ts`
  - Lines: ~30
  - Details: `fetchGrafanaStatus(serviceId?)`, `fetchPrometheusStatus(serviceId?)`,
    `useGrafanaStatus`, `usePrometheusStatus`.

- [ ] **3.2 Rewire Observability page**
  - Files: `frontend/src/components/ObservabilityPage.tsx`
  - Lines: ~90
  - Details: `useServiceInstances("grafana")` → derive `GRAFANA_BASE_URL` from
    first enabled instance; remove `import.meta.env.VITE_GRAFANA_URL`; empty-state
    linking to `/services` when none; add Grafana + Prometheus `HealthCard`s.

- [ ] **3.3 Add alertmanager widget**
  - Files: `frontend/src/widgets/AlertmanagerAlertsWidget.tsx` (new),
    `frontend/src/integrations/registry.ts`, `frontend/src/widgets/index.ts`
  - Lines: ~90
  - Details: presentational component reusing the alert summary shape; register
    `alertmanager` binding with `active_alerts` kind.

- [ ] **3.4 Update types**
  - Files: `frontend/src/types/index.ts`
  - Lines: ~20
  - Details: `GrafanaStatus`, `PrometheusStatus`; add `service_id`/`name` to
    status types; `AlertmanagerAlertsWidgetConfig`.

- [ ] **3.5 Update frontend tests**
  - Files: `frontend/src/integrations/registry.test.ts`, widget/page tests as needed
  - Lines: ~40
  - Details: assert alertmanager binding resolves; mock status hooks.

- [ ] **3.6 Verify frontend**
  - Run: `cd frontend && npm run lint && npm run build && npm run test`

- [ ] **3.7 Commit**
  - Message: `feat(observability): service discovery, health cards, alertmanager widget`

**Slice 3 total:** ~270 changed lines.

---

## Slice 4: Backend — remove file-SD writer

**Goal:** Drop the shared-file Prometheus bridge (`PROMETHEUS_FILE_SD_DIR`);
external Prometheus uses `http_sd_configs` against the existing endpoint.

- [ ] **4.1 Remove the file-writer call sites**
  - Files: `backend/src/media_library_viewer_api/main.py` (startup call),
    `backend/src/media_library_viewer_api/routers/settings.py`
    (post machine create/update/delete calls)
  - Lines: ~-15
  - Details: delete `write_prometheus_targets(...)` invocations; remove the
    import. **Keep** `services/targets.py::build_node_exporter_targets` and the
    `GET /api/monitoring/prometheus-targets` endpoint (external Prometheus
    consumes via `http_sd_configs`).

- [ ] **4.2 Remove the file-writer + config field**
  - Files: `backend/src/media_library_viewer_api/services/targets.py`,
    `backend/src/media_library_viewer_api/config.py`
  - Lines: ~-30
  - Details: delete `write_prometheus_targets` (the writer) from `targets.py`,
    leaving `build_node_exporter_targets`; delete `prometheus_file_sd_dir` from
    `config.py`. Prune now-unused imports (`Path`, etc.).

- [ ] **4.3 Remove compose mount + env var**
  - Files: `docker-compose.yml`, `docker-compose.dev.yml`
  - Lines: ~-4
  - Details: drop `PROMETHEUS_FILE_SD_DIR` backend env var and any
    `prometheus-file-sd` volume mount/bind reference (no Prometheus container
    remains to read it).

- [ ] **4.4 Update backend tests**
  - Files: `backend/tests/test_targets.py`, `backend/tests/test_api.py`
  - Lines: ~-20 / ~+5
  - Details: drop tests covering `write_prometheus_targets`; keep/extend tests
    for `build_node_exporter_targets` and the `/prometheus-targets` endpoint.
    Drop any `write_targets.assert_called_once` assertion in
    `TestSettingsMachines`.

- [ ] **4.5 Verify backend**
  - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest`

- [ ] **4.6 Commit**
  - Message: `refactor(observability): drop file-SD writer for http_sd_configs`

**Slice 4 total:** ~70 changed lines (net negative).

---

## Slice 5: Env removal + docs

**Goal:** Delete the remaining observability service env vars from config/compose
and document the change.

- [ ] **5.1 Remove backend env fields**
  - Files: `backend/src/media_library_viewer_api/config.py`
  - Lines: ~-2
  - Details: delete `alertmanager_url` and `alertmanager_webhook_url` fields.

- [ ] **5.2 Remove compose / Dockerfile build args**
  - Files: `docker-compose.yml`, `docker-compose.dev.yml`, `frontend/Dockerfile`
  - Lines: ~-8
  - Details: drop `ALERTMANAGER_URL`, `ALERTMANAGER_WEBHOOK_URL` from backend
    env; drop `VITE_GRAFANA_URL`, `VITE_PROMETHEUS_URL` from frontend build args
    (both compose files) and `ARG`/`ENV` (Dockerfile).

- [ ] **5.3 Update docs**
  - Files: `docs/REQUIREMENTS.md`, `CHANGELOG.md`, `backend/README.md`,
    `docs/monitoring-logging-design.md`
  - Lines: ~60
  - Details: REQUIREMENTS decision-log entry (alertmanager as a service type;
    observability env vars removed incl. `PROMETHEUS_FILE_SD_DIR`; http_sd_configs
    replaces the file bridge; thin-dashboard model unchanged); CHANGELOG
    added/changed/**BREAKING** (`ALERTMANAGER_URL` users re-create the instance;
    `file_sd_configs` users switch to `http_sd_configs`); backend README
    monitoring section.

- [ ] **5.4 Manual follow-up (assistant-blocked) — `.env.example`**
  - `.env.example` is blocked by the safety policy. Note for the user: remove
    `ALERTMANAGER_URL`, `ALERTMANAGER_WEBHOOK_URL`, `VITE_GRAFANA_URL`,
    `VITE_PROMETHEUS_URL`, and `PROMETHEUS_FILE_SD_DIR`; keep `PROMETHEUS_ENABLED`.
  - No code lines.

- [ ] **5.5 Verify**
  - Run: `cd backend && .venv/bin/ruff check . && .venv/bin/python -m pytest`
  - Run: `cd frontend && npm run lint && npm run build && npm run test`
  - Grep-gate: confirm no remaining references to the removed env vars in
    `backend/src`, `frontend/src`, root compose files, Dockerfile.

- [ ] **5.6 Commit**
  - Message: `chore(observability): remove remaining observability env vars, docs`

**Slice 5 total:** ~110 changed lines.

---

## Acceptance

- `alertmanager` is a service type; observability services are UI-configured only.
- The observability service env vars (`ALERTMANAGER_URL`,
  `ALERTMANAGER_WEBHOOK_URL`, `VITE_GRAFANA_URL`, `VITE_PROMETHEUS_URL`,
  `PROMETHEUS_FILE_SD_DIR`) have no remaining references. `PROMETHEUS_ENABLED`
  (Manage's own `/metrics` toggle) is the only observability env var that remains.
- `/alerts`, `/alertmanager-status`, `/grafana-status`, `/prometheus-status` work
  against service instances with graceful missing/unreachable states.
- Observability page shows Alertmanager/Grafana/Prometheus health and derives
  Grafana links from the registry (no `VITE_*` URL).
- `active_alerts` alertmanager widget renders on the dashboard.
- External Prometheus consumes node-exporter targets via `http_sd_configs`
  against `/api/monitoring/prometheus-targets`; no shared volume remains.
- All backend and frontend test suites green; docs + CHANGELOG updated.