manage/openspec/changes/archive/observability-service-registry/design.md

# Design — Observability service registry

**Change:** `observability-service-registry`
**Phase:** design
**Date:** 2026-06-23

## Current state

- **Service registry** (`integrations/registry.py`): five types — `grafana`,
  `prometheus`, `jellyfin`, `jellyseerr`, `nextcloud`, `ssh_tasks`. Grafana
  (`grafana.py`: `base_url`, secret `api_key`, widget `link`) and Prometheus
  (`prometheus.py`: `base_url`, secret `api_key`, widget `metric`) are already
  service types. Their widget sources (`widgets/sources.py`:
  `GrafanaWidgetSource`, `PrometheusWidgetSource`) already resolve URLs from the
  `ServiceRecord`, not env vars.
- **Monitoring router** (`routers/monitoring.py`): `/alerts`,
  `/alertmanager-status`, `/alertmanager-webhook`, `/prometheus-targets`,
  `/machines`. The alertmanager endpoints read `settings.alertmanager_url`
  (env). `_alertmanager_client()` returns `(requests.Session(), url)`; `_webhook_client()`
  returns the forward target.
- **Config** (`config.py`): `prometheus_enabled` (Manage's own `/metrics` toggle —
  stays), `prometheus_file_sd_dir`, `alertmanager_url=""`,
  `alertmanager_webhook_url=""`. The latter three are removed by this change.
  `services/targets.py::write_prometheus_targets` is the file-writer consuming
  `prometheus_file_sd_dir`; `build_node_exporter_targets` + the
  `/prometheus-targets` endpoint stay (used by external Prometheus via
  `http_sd_configs`).
- **Frontend Observability page** (`ObservabilityPage.tsx`): hard-codes
  `GRAFANA_BASE_URL = import.meta.env.VITE_GRAFANA_URL || "http://localhost:3000"`
  and builds Node Exporter + Loki deep-links from it. Hooks
  (`useObservability.ts`) call `/alerts`, `/alertmanager-status`,
  `/prometheus-targets`, `/machines`.
- **Service resolution**: `settings_store.list_services(service_type)` orders by
  `name ASC`; `get_service(id)`. There is **no `is_default`/primary flag**.
- **`VITE_PROMETHEUS_URL`** is referenced only in the Dockerfile/compose build
  args, not in any frontend source (verified). `VITE_GRAFANA_URL` is read in
  `ObservabilityPage.tsx` only.
- **Webhook relay**: `monitoring/alertmanager/alertmanager.yml` (standalone stack)
  points `webhook_configs` at `http://backend:8000/api/monitoring/alertmanager-webhook`.
  The receiver then optionally forwards to `ALERTMANAGER_WEBHOOK_URL`.

## Target state

### 1. New `alertmanager` integration type

`integrations/alertmanager.py`, mirroring `prometheus.py`:

```python
class AlertmanagerConfig(ServiceConfigBase):
    base_url: str
    timeout_seconds: int = 5

class AlertmanagerAlertsWidgetConfig(WidgetConfigBase):
    severity_filter: str | None = None  # optional: "critical", "warning", ...

DEFINITION = ServiceDefinition(
    service_type="alertmanager",
    name="Alertmanager",
    description="Alert routing and firing-alert summaries.",
    config_model=AlertmanagerConfig,
    secret_fields=[SecretField(key="api_key", label="API key", helper="Optional bearer token")],
    widget_kinds=[widget_kind("active_alerts", "Active alerts",
                              "Firing-alert summary from this Alertmanager.",
                              model_cls=AlertmanagerAlertsWidgetConfig,
                              default_config={}, refresh_interval_ms=15_000)],
)
```

Register in `registry.py` (`SERVICE_DEFINITIONS`). Add `AlertmanagerWidgetSource`
to `SERVICE_ADAPTERS` in `widgets/sources.py`.

### 2. Service resolution helper (shared)

Add a single helper in the monitoring router used by the alertmanager endpoints
and the new grafana/prometheus status endpoints:

```python
def _resolve_service_record(
    store: SettingsStore, service_type: str, service_id: str | None
) -> ServiceRecord | None:
    """Return the requested instance, else the first enabled instance of type."""
    if service_id:
        row = store.get_service(service_id)
        if row and row.get("service_type") == service_type and row.get("enabled", True):
            return build_service_record(store, row)
        return None
    for row in store.list_services(service_type):
        if row.get("enabled", True):
            return build_service_record(store, row)
    return None
```

This factors `build_service_record` (already in `widgets/sources.py`) and keeps
secret decryption in one place. **Default selection = first enabled instance of
type** (ordered by `name ASC`). No DB schema change.

### 3. Rewired alertmanager endpoints

`/alerts?service_id=<optional>` and `/alertmanager-status?service_id=<optional>`
become `Depends(get_settings_store)` endpoints:

- Resolve the service record via `_resolve_service_record(store, "alertmanager", service_id)`.
- If `None` → return the existing not-configured bodies (`{"error": "alertmanager_not_configured"}`, `{"up": False, ...}`).
- Else build `base_url` from `service.config["base_url"]`, attach `Authorization: Bearer <api_key>`
  if a secret is present, and call `/api/v1/alerts` / `/api/v2/status`.
- On exception → existing unreachable bodies.
- **Response gains `service_id` + `name`** on success so the UI can show which
  instance was used.

`_alertmanager_client()` and `_webhook_client()` (the env readers) are deleted.

### 4. New health endpoints

`GET /api/monitoring/grafana-status?service_id=` and
`GET /api/monitoring/prometheus-status?service_id=`:

- Resolve via `_resolve_service_record(store, "grafana"|"prometheus", service_id)`.
- Probe Grafana `GET {base_url}/api/health` (200 → `{database: "ok"}`), Prometheus
  `GET {base_url}/-/healthy` + `GET {base_url}/api/v1/status/buildinfo` (version).
- Return `{ up: bool, version: str, service_id: str, name: str, error: str | None }`.
- None configured → `{up: false, version: "", service_id: "", name: "", error: "no_service_configured"}`.
- Attach bearer token if a secret is present (some auth'd setups need it).

**Decision — dedicated endpoints vs. widget-data path:** dedicated endpoints.
The Observability page needs page-level health for the default instance
independent of any widget instance; widget data fetch is per-widget and keyed on
a widget kind with a different shape. Reusing `_resolve_service_record` keeps the
service-resolution logic single-sourced; only the probe differs per type.

### 5. `active_alerts` widget adapter

`AlertmanagerWidgetSource.fetch(service, "active_alerts", config)` reuses the
existing `_summary_from_alerts` to shape `/api/v1/alerts` into the same summary
the page endpoint returns. Optional `severity_filter` limits severities.

### 6. Webhook relay decision

**Decision:** drop the outbound relay (`alertmanager_webhook_url` env var and the
forward `POST`). The `POST /alertmanager-webhook` receiver **stays** as
**log-only** (it records received alerts at INFO for audit/debug and returns
`{"status": "received"}`). Rationale: the inbound→outbound relay to a second
webhook is a niche feature whose target is the env var we are removing; a
log-only receiver remains useful and keeps the standalone Alertmanager config
working unchanged.

### 7. `PROMETHEUS_FILE_SD_DIR` decision

**Decision:** **remove it.** It is a vestige of the in-project Prometheus model
where Manage's container shared a volume with a Prometheus container and wrote
`node_exporter_targets.json` into it for `file_sd_configs`. In the
connect-to-existing model Prometheus is a separate deployment, and a shared
volume across deployments is awkward (network share / rsync hack).

The same data is already served over HTTP by
`GET /api/monitoring/prometheus-targets`, so an external Prometheus consumes it
via standard `http_sd_configs` instead — pull-based, no shared volume, no env var:

```yaml
# external Prometheus
- job_name: node-exporter-remote
  http_sd_configs:
    - url: https://manage.example.com/api/monitoring/prometheus-targets
      refresh_interval: 30s
```

Concretely in this change: delete the `prometheus_file_sd_dir` config field, the
`write_prometheus_targets()` **file-writer** call sites (startup + machine
create/update/delete hooks), and the `PROMETHEUS_FILE_SD_DIR` compose/Dockerfile
mounts. **Keep** `services/targets.py::build_node_exporter_targets` (the data
builder) and the `/api/monitoring/prometheus-targets` HTTP endpoint.

`PROMETHEUS_ENABLED` **stays** as an env var — it toggles Manage's own
`/metrics` endpoint, which is app instrumentation (Role 1), not service
integration. It is the single legitimate observability env survivor.

## Frontend changes

1. **Observability page** (`ObservabilityPage.tsx`):
   - Discover grafana services via `useServiceInstances("grafana")`; pick the
     first enabled; build `GRAFANA_BASE_URL` from its `base_url`. Remove the
     `import.meta.env.VITE_GRAFANA_URL` read. Empty-state when none configured
     (link to `/services`).
   - Add `useGrafanaStatus()` and `usePrometheusStatus()` hooks; add Grafana and
     Prometheus `HealthCard`s alongside Alertmanager.
2. **Hooks/client** (`useObservability.ts`, `api/client.ts`):
   - `fetchGrafanaStatus(serviceId?)`, `fetchPrometheusStatus(serviceId?)`,
     `useGrafanaStatus`, `usePrometheusStatus`.
   - Existing `fetchAlertmanagerAlerts` / `fetchAlertmanagerStatus` keep their
     signatures (no service id needed for the page's default instance).
3. **Widget registry** (`integrations/registry.ts`): add `alertmanager` binding
   with the `active_alerts` kind → new `AlertmanagerAlertsWidget` component.
4. **Types** (`types/index.ts`): `GrafanaStatus`, `PrometheusStatus`,
   `AlertmanagerAlertsWidgetConfig`; add `service_id`/`name` to status shapes.
5. **`VITE_PROMETHEUS_URL`** is unused in source — only its removal from the
   Dockerfile/compose build args is needed.

## Data model / compatibility

- No DB schema change. Alertmanager instances live in the existing `services`
  table as `service_type="alertmanager"`.
- **Breaking for `ALERTMANAGER_URL` users:** after upgrade the env var is ignored;
  the Alertmanager instance must be (re)created on the Services page. Document in
  CHANGELOG. No data migration (it was an env var, not a DB row).
- **Type-contract cleanup (pre-existing drift):** the down-branch
  `/alertmanager-status` bodies omit `name`/`peers`. The frontend `AlertmanagerStatus`
  type declares them required but never reads them. Fix by adding `name: ""`,
  `peers: []` to the not-configured/unreachable branches (and `service_id`) so
  the response is internally consistent.

## Open questions

1. **Default-instance flag.** v1 uses "first enabled instance of type" (by name).
   If multiple Grafana/Prometheus/Alertmanager instances exist, status reflects
   only the first. A per-type "primary" flag is deferred — confirm this is
   acceptable, or whether to add the flag in this change (adds a UI affordance +
   DB column; would grow scope past one slice).
2. **Service-id in page endpoints.** Should `/alerts?service_id=` be surfaced in
   the Observability page UI (an instance selector), or kept backend-only for
   future use? v1 leaves the page on the default instance.
3. **`.env.example`** is assistant-edit-blocked by safety policy. The four env
   vars must be removed manually by the user; tracked as a task follow-up.