manage/openspec/changes/archive/service-registry/design.md

# Design: Runtime Service Registry

**Change:** `service-registry`
**Phase:** design
**Date:** 2026-06-19

## 1. Architecture overview

```
┌────────────────────────────────────────────────────────────────────┐
│                              Browser                                │
│   /services/:type/:id  ─►  ServicePage  ─►  frontend SERVICE_REGISTRY
│   Dashboard             ─►  WidgetInstance ─►  widget component     │
└────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌────────────────────────────────────────────────────────────────────┐
│                  FastAPI /api/services + /api/widgets               │
│   CRUD service instances · registry metadata · widget data         │
└────────────────────────────────────────────────────────────────────┘
                                  │
       ┌──────────────────────────┼───────────────────────────┐
       ▼                          ▼                           ▼
  ServiceStore (SQLite)    integrations/ definitions     source adapters
  services table           (Pydantic, closed registry)   (resolve service
  dashboard_widgets table  grafana/prometheus/jellyfin/   → decrypt → call)
                          nextcloud/ssh_tasks
```

Two closed, compile-time registries cooperate:

- **`integrations.registry.SERVICE_DEFINITIONS`** maps `service_type → ServiceDefinition`.
  Each definition declares config schema, secret fields, and widget kinds.
- The widget types available to the dashboard are **derived** from
  `SERVICE_DEFINITIONS` at startup, not hand-maintained.

## 2. Backend data model

### 2.1 New `services` table

Extend `SettingsStore.init_schema()`:

```sql
CREATE TABLE IF NOT EXISTS services (
    id TEXT PRIMARY KEY,
    service_type TEXT NOT NULL,
    name TEXT NOT NULL,
    config_json TEXT NOT NULL DEFAULT '{}',
    secrets_json TEXT NOT NULL DEFAULT '{}',   -- encrypted blob (Fernet)
    enabled INTEGER NOT NULL DEFAULT 1,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_services_type ON services(service_type);
```

- `config_json` — non-secret config validated against the service definition's
  `config_schema`.
- `secrets_json` — a JSON object of `{field_name: ciphertext}` produced by the
  encryption helper. Never returned to the client in plaintext; only the boolean
  "is set" flags are surfaced.

### 2.2 `dashboard_widgets` schema change

The existing table gains two columns and loses the global meaning of `widget_type`:

```sql
ALTER TABLE dashboard_widgets ADD COLUMN service_id TEXT;
ALTER TABLE dashboard_widgets ADD COLUMN widget_kind TEXT;
```

- `widget_kind` is the kind declared by the service definition (e.g. `"link"`,
  `"metric"`, `"activity"`).
- `service_id` references `services.id`.
- `widget_type` is retained temporarily as `"{service_type}.{widget_kind}"` for
  backwards-compatible reads during the transition, then dropped in the final slice.
- The old `addon_id` column is dropped; addon identity is now `service_type`.

## 3. Service definitions (Pydantic, in repo)

New package: `backend/src/media_library_viewer_api/integrations/`
(chosen to avoid collision with the existing `services/` infra package).

### 3.1 Base classes — `integrations/base.py`

```python
from typing import Any, ClassVar
from pydantic import BaseModel, Field

class SecretField(BaseModel):
    key: str
    label: str
    required: bool = False
    helper: str | None = None

class WidgetKind(BaseModel):
    kind: str                       # e.g. "link", "metric", "activity"
    name: str
    description: str
    config_schema: dict[str, Any]   # JSON schema for widget config
    default_config: dict[str, Any] = {}
    refresh_interval_ms: int = 0

class ServiceConfigBase(BaseModel):
    """Subclass per service to define non-secret config fields."""

class ServiceDefinition(BaseModel):
    service_type: ClassVar[str]
    name: ClassVar[str]
    description: ClassVar[str]
    config_schema: ClassVar[dict[str, Any]]
    secret_fields: ClassVar[list[SecretField]]
    widget_kinds: ClassVar[list[WidgetKind]]

    # Adapters are referenced by dotted path or registered separately;
    # see §4. The definition itself stays a pure data/schema object.
```

### 3.2 Example — `integrations/grafana.py`

```python
class GrafanaConfig(ServiceConfigBase):
    base_url: str = Field(..., description="Grafana base URL, e.g. https://grafana.example.com")

GRAFANA_DEFINITION = ServiceDefinition(
    service_type="grafana",
    name="Grafana",
    description="Dashboards, metrics, and logs.",
    config_schema=GrafanaConfig.model_json_schema(),
    secret_fields=[SecretField(key="api_key", label="API key", helper="Service account token")],
    widget_kinds=[
        WidgetKind(
            kind="link",
            name="Dashboard link",
            description="Deep-link to a Grafana dashboard or panel.",
            config_schema={
                "type": "object",
                "properties": {
                    "dashboard_uid": {"type": "string"},
                    "panel_id": {"type": "integer"},
                },
                "required": ["dashboard_uid"],
            },
            default_config={"dashboard_uid": ""},
            refresh_interval_ms=0,
        ),
    ],
)
```

Other definition modules follow the same shape: `prometheus.py`, `jellyfin.py`,
`nextcloud.py`, `ssh_tasks.py`.

### 3.3 Registry — `integrations/registry.py`

```python
SERVICE_DEFINITIONS: dict[str, ServiceDefinition] = {
    "grafana": GRAFANA_DEFINITION,
    "prometheus": PROMETHEUS_DEFINITION,
    "jellyfin": JELLYFIN_DEFINITION,
    "nextcloud": NEXTCLOUD_DEFINITION,
    "ssh_tasks": SSH_TASKS_DEFINITION,
}

def list_service_types() -> list[str]: ...
def get_service_definition(service_type: str) -> ServiceDefinition | None: ...
def get_widget_kind(service_type: str, widget_kind: str) -> WidgetKind | None: ...
```

The closed `widgets/registry.py` from Phase 1 is **retired**; its metadata is now derived
from `SERVICE_DEFINITIONS`.

## 4. Source adapters

`widgets/sources.py` is refactored so each adapter resolves a **service instance** rather
than reading `get_settings()`:

```python
class WidgetSource(Protocol):
    async def fetch(
        self,
        service: ServiceRecord,         # config + decrypted secrets
        widget_kind: str,
        config: dict[str, Any],
    ) -> dict[str, Any]: ...
```

- `ServiceRecord` is a runtime object built by `ServiceStore` carrying the decrypted
  secret dict in memory for the duration of the fetch.
- `SOURCE_ADAPTERS` is keyed by `service_type`.
- The data endpoint loads the widget's `service_id`, builds the `ServiceRecord`, then
  calls `adapter.fetch(service, widget_kind, widget_config)`.

## 5. Encryption — `services/secrets.py`

```python
from cryptography.fernet import Fernet, InvalidToken

def get_encryption_key() -> bytes:
    raw = os.environ.get("MANAGE_ENCRYPTION_KEY")
    if not raw:
        raise RuntimeError("MANAGE_ENCRYPTION_KEY is required")
    return raw.encode()

def encrypt_secrets(values: dict[str, str]) -> dict[str, str]: ...
def decrypt_secrets(blob: dict[str, str]) -> dict[str, str]: ...
```

- `cryptography.fernet.Fernet` (already a transitive dependency to verify).
- Startup validation: `validate_auth_settings` is extended to require
  `MANAGE_ENCRYPTION_KEY` and to reject an obviously invalid key.
- Secrets are encrypted field-by-field so the "which secrets are set" metadata is cheap
  to compute without decrypting.

## 6. REST API

### Services

| Method | Path | Handler |
|--------|------|---------|
| GET | `/api/services/types` | List service definitions (metadata + config schema + widget kinds). |
| GET | `/api/services` | List service instances (no plaintext secrets; only "set" flags). |
| POST | `/api/services` | Create instance (validates type, config, secret schema). |
| PUT | `/api/services/{id}` | Update instance. |
| DELETE | `/api/services/{id}` | Delete instance; **cascade-deletes** widgets referencing it in the same transaction. |

### Widgets (unchanged paths, new semantics)

| Method | Path | Handler |
|--------|------|---------|
| GET | `/api/widgets/instances` | List widgets; each carries `service_id`, `widget_kind`. |
| POST/PUT/DELETE | `/api/widgets/instances/{id}` | CRUD; validation uses service definition's widget schema. |
| GET | `/api/widgets/instances/{id}/data` | Resolve service → adapter → fetch. |

`GET /api/widgets/types` and `/api/widgets/sources` are removed; widget metadata is
served via `/api/services/types` (widget kinds under each service).

## 7. Frontend

### 7.1 New `frontend/src/integrations/registry.ts`

Closed frontend registry mirroring the backend: `serviceType → ServiceDefinition`
(config fields, secret fields with `secret: true`, widget kinds, default refresh
intervals, and a `component` for the service page).

### 7.2 Service pages

- Route: `/services/:serviceType/:serviceId` (replaces `/addons/:addonId`).
- `ServicePage` looks up the definition and renders the service-specific component,
  a config editor, and the list of widget kinds that can be added to the dashboard.
- `App.tsx` removes the `/addons/:addonId` route; old addon URLs redirect to the
  default service of that type (or a not-found alert).

### 7.3 Dashboard config dialog

- "Add widget" flow becomes: **pick service → pick widget kind → configure**.
- The widget card shows the parent service name.

### 7.4 Types / API / hooks

- `frontend/src/api/services.ts` + `hooks/useServices.ts` for the services API.
- `frontend/src/types/index.ts` gains `ServiceInstance`, `ServiceInstanceInput`,
  `ServiceTypeInfo`, `ServiceWidgetKind`.

## 8. Migration and breaking changes

- **DB migration on startup:** add `services` table; add `service_id` / `widget_kind`
  columns to `dashboard_widgets`; drop `addon_id`.
- **Machine app fields removed:** `jellyfin_url`, `jellyfin_user_id`, `jellyfin_api_key`,
  `jellyseerr_url`, `jellyseerr_api_key` are dropped from machine records and the
  `MonitoringMachine` model. Machines keep SSH + node_exporter transport fields only.
- **Env vars removed from `config.py`:** `grafana_url`, `prometheus_url`. (Grafana/Prometheus
  URLs now live on service records.) `MANAGE_ENCRYPTION_KEY` is added as required.
- **Default widget seeding** is removed; a fresh install starts with no widgets. The user
  adds Jellyfin/Backups widgets after configuring the corresponding services.
- **`docs/REQUIREMENTS.md` and `README.md`** updated to describe services, the
  `MANAGE_ENCRYPTION_KEY` requirement, and the breaking upgrade note.

## 9. File-level plan

### Create (backend)

| File | Purpose |
|------|---------|
| `integrations/__init__.py` | Package marker. |
| `integrations/base.py` | `ServiceDefinition`, `WidgetKind`, `SecretField`, `ServiceConfigBase`. |
| `integrations/registry.py` | Closed `SERVICE_DEFINITIONS` + helpers. |
| `integrations/grafana.py`, `prometheus.py`, `jellyfin.py`, `nextcloud.py`, `ssh_tasks.py` | One module per service. |
| `services/secrets.py` | Fernet encrypt/decrypt + key validation. |
| `services/service_store.py` | CRUD for `services` table; decrypt-on-read for adapters. |
| `routers/services.py` | `/api/services*` endpoints. |
| `models/services.py` | Pydantic request/response models. |

### Modify (backend)

| File | Change |
|-------|--------|
| `services/settings_store.py` | `services` table; widget columns; drop machine app fields. |
| `widgets/sources.py` | Adapters take a `ServiceRecord`. |
| `widgets/registry.py` | Retired (metadata served by `integrations/registry.py`). |
| `routers/widgets.py` | Validate against service widget schema; resolve service on data fetch. |
| `config.py` | Remove `grafana_url`/`prometheus_url`; document `MANAGE_ENCRYPTION_KEY` (read in `secrets.py`). |
| `main.py` | Register `services_router`; validate encryption key on startup. |
| `dependencies.py` | Jellyfin/SSH resolution now goes via services, not machine app fields. |

### Create (frontend)

| File | Purpose |
|------|---------|
| `integrations/registry.ts` | Closed frontend service registry. |
| `api/services.ts`, `hooks/useServices.ts` | Services API + hooks. |
| `pages/ServicePage.tsx` | Generic `/services/:type/:id` page. |
| `integrations/components/*` | Per-service page components. |

### Modify (frontend)

| File | Change |
|-------|--------|
| `App.tsx` | Replace `/addons/:addonId` with `/services/:serviceType/:serviceId`. |
| `components/WidgetConfigDialog.tsx` | Service → widget-kind picker. |
| `widgets/registry.ts` | Retired; widgets derived from service registry. |
| `types/index.ts` | Service types; widget gains `service_id` + `widget_kind`. |
| `pages/Settings.tsx` | Remove machine Jellyfin/Jellyseerr fields. |

## 10. Slice boundaries (chained PRs)

Each slice keeps `pytest`, `ruff`, `npm run lint`, and `npm run build` green.

1. **Backend foundation** — encryption helper, `integrations/` base + 5 definitions +
   registry, `services` table + store, `/api/services*` endpoints, tests. No widget
   changes yet.
2. **Backend widget rebind** — add `service_id`/`widget_kind` to widgets, refactor
   adapters to take a `ServiceRecord`, retire old `widgets/registry.py`, update data
   endpoint.
3. **Frontend services runtime** — types, API, hooks, `integrations/registry.ts`,
   service pages, route swap, remove addon pages.
4. **Frontend dashboard + settings rework** — service-based widget picker, drop machine
   app fields from Settings, remove `grafana_url`/`prometheus_url` from config,
   re-seed behavior, docs (`README.md`, `REQUIREMENTS.md`), changelog breaking-change
   note.

Estimated total: ~2,000–2,400 changed lines across four PRs.

## 11. Decisions resolved

1. **Deleting a service that still has widgets** → **cascade delete.** The store deletes
   every `dashboard_widgets` row referencing the service inside the same transaction as
   the service delete. Simple and safe in SQLite; no 409 pre-check.
2. **`MANAGE_ENCRYPTION_KEY` dev default** → **always required.** No fallback, even when
   `AUTH_ENABLED=false`. Startup fails fast if it is missing or not a valid Fernet key.
3. **SSH task runner shape** → **multi-instance, reusable tasks, persisted run history.**
   See §12 for the full model.

## 12. SSH task runner model

The SSH task runner is the most involved service type. Instances absorb the SSH task
execution role currently held by machines; tasks stay global and reusable; every
invocation is logged.

### 12.1 Instances

- `service_type = "ssh_tasks"`.
- Each instance is an SSH endpoint: `host`, `port`, `username`, `ssh_key_id`, optional
  `passphrase`. Connection config lives on the service record; the SSH key itself stays
  in the existing saved-key registry (referenced by `ssh_key_id`).
- Multi-instance by design ("home server", "media box", …).

### 12.2 Tasks (global, reusable)

- Saved tasks remain a **global** registry (`name`, `task_type` shell/python, `content`,
  `enabled`). A task is **not** owned by an instance.
- Each task gains `default_service_id` (replaces the old `default_machine_id`) — the
  instance it targets by default. At run time the caller may override the target
  instance.
- A task can therefore run against any instance; the link is captured per-run.

### 12.3 Run history (logs)

A new `service_task_runs` table records every invocation:

```sql
CREATE TABLE IF NOT EXISTS service_task_runs (
    id TEXT PRIMARY KEY,
    task_id TEXT NOT NULL,
    service_id TEXT NOT NULL,
    status TEXT NOT NULL,        -- success | failure | timeout | error
    exit_status INTEGER,
    duration_ms INTEGER,
    stdout_tail TEXT,
    stderr_tail TEXT,
    error TEXT,
    created_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_service_task_runs_service ON service_task_runs(service_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_service_task_runs_task ON service_task_runs(task_id, created_at DESC);
```

- Populated by the SSH task adapter on every widget data fetch and by the Actions
  runner on manual runs.
- Surfaced on the instance's service page as a log/history list, and on the task detail
  as recent runs.
- Replaces the legacy `saved_task_runs` concept once the Actions page is rebuilt on
  services (Slice 4 / a follow-up).

### 12.4 SSH task widget

Widget config for `ssh_tasks` becomes `{ task_id, service_id? }`:

- If `service_id` is omitted, the task's `default_service_id` is used.
- The adapter loads the task, resolves the instance, runs it, appends a
  `service_task_runs` row, and returns the trimmed stdout/stderr/exit status.

### 12.5 Relationship to machines

- The SSH task execution role moves **out of machines** into `ssh_tasks` instances.
- Machines **keep** their role for the File Browser and node_exporter monitoring
  transport in this change, to avoid also reworking Files/Monitoring here.
- Practical consequence: an SSH host used for both files and tasks may be defined twice
  (once as a machine, once as an ssh_tasks instance) during the transition. Unifying
  machines under services is an explicit **follow-up change**, not part of this one.