manage/openspec/changes/service-registry/design.md

Design: Runtime Service Registry

Change: service-registry Phase: design Date: 2026-06-19

1. Architecture overview

┌────────────────────────────────────────────────────────────────────┐
│                              Browser                                │
│   /services/:type/:id  ─►  ServicePage  ─►  frontend SERVICE_REGISTRY
│   Dashboard             ─►  WidgetInstance ─►  widget component     │
└────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌────────────────────────────────────────────────────────────────────┐
│                  FastAPI /api/services + /api/widgets               │
│   CRUD service instances · registry metadata · widget data         │
└────────────────────────────────────────────────────────────────────┘
                                  │
       ┌──────────────────────────┼───────────────────────────┐
       ▼                          ▼                           ▼
  ServiceStore (SQLite)    integrations/ definitions     source adapters
  services table           (Pydantic, closed registry)   (resolve service
  dashboard_widgets table  grafana/prometheus/jellyfin/   → decrypt → call)
                          nextcloud/ssh_tasks

Two closed, compile-time registries cooperate:

• integrations.registry.SERVICE_DEFINITIONS maps service_type → ServiceDefinition. Each definition declares config schema, secret fields, and widget kinds.
• The widget types available to the dashboard are derived from SERVICE_DEFINITIONS at startup, not hand-maintained.

2. Backend data model

2.1 New services table

Extend SettingsStore.init_schema():

CREATE TABLE IF NOT EXISTS services (
    id TEXT PRIMARY KEY,
    service_type TEXT NOT NULL,
    name TEXT NOT NULL,
    config_json TEXT NOT NULL DEFAULT '{}',
    secrets_json TEXT NOT NULL DEFAULT '{}',   -- encrypted blob (Fernet)
    enabled INTEGER NOT NULL DEFAULT 1,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_services_type ON services(service_type);

• config_json — non-secret config validated against the service definition's config_schema.
• secrets_json — a JSON object of {field_name: ciphertext} produced by the encryption helper. Never returned to the client in plaintext; only the boolean "is set" flags are surfaced.

2.2 dashboard_widgets schema change

The existing table gains two columns and loses the global meaning of widget_type:

ALTER TABLE dashboard_widgets ADD COLUMN service_id TEXT;
ALTER TABLE dashboard_widgets ADD COLUMN widget_kind TEXT;

• widget_kind is the kind declared by the service definition (e.g. "link", "metric", "activity").
• service_id references services.id.
• widget_type is retained temporarily as "{service_type}.{widget_kind}" for backwards-compatible reads during the transition, then dropped in the final slice.
• The old addon_id column is dropped; addon identity is now service_type.

3. Service definitions (Pydantic, in repo)

New package: backend/src/media_library_viewer_api/integrations/ (chosen to avoid collision with the existing services/ infra package).

3.1 Base classes — integrations/base.py

from typing import Any, ClassVar
from pydantic import BaseModel, Field

class SecretField(BaseModel):
    key: str
    label: str
    required: bool = False
    helper: str | None = None

class WidgetKind(BaseModel):
    kind: str                       # e.g. "link", "metric", "activity"
    name: str
    description: str
    config_schema: dict[str, Any]   # JSON schema for widget config
    default_config: dict[str, Any] = {}
    refresh_interval_ms: int = 0

class ServiceConfigBase(BaseModel):
    """Subclass per service to define non-secret config fields."""

class ServiceDefinition(BaseModel):
    service_type: ClassVar[str]
    name: ClassVar[str]
    description: ClassVar[str]
    config_schema: ClassVar[dict[str, Any]]
    secret_fields: ClassVar[list[SecretField]]
    widget_kinds: ClassVar[list[WidgetKind]]

    # Adapters are referenced by dotted path or registered separately;
    # see §4. The definition itself stays a pure data/schema object.

3.2 Example — integrations/grafana.py

class GrafanaConfig(ServiceConfigBase):
    base_url: str = Field(..., description="Grafana base URL, e.g. https://grafana.example.com")

GRAFANA_DEFINITION = ServiceDefinition(
    service_type="grafana",
    name="Grafana",
    description="Dashboards, metrics, and logs.",
    config_schema=GrafanaConfig.model_json_schema(),
    secret_fields=[SecretField(key="api_key", label="API key", helper="Service account token")],
    widget_kinds=[
        WidgetKind(
            kind="link",
            name="Dashboard link",
            description="Deep-link to a Grafana dashboard or panel.",
            config_schema={
                "type": "object",
                "properties": {
                    "dashboard_uid": {"type": "string"},
                    "panel_id": {"type": "integer"},
                },
                "required": ["dashboard_uid"],
            },
            default_config={"dashboard_uid": ""},
            refresh_interval_ms=0,
        ),
    ],
)

Other definition modules follow the same shape: prometheus.py, jellyfin.py, nextcloud.py, ssh_tasks.py.

3.3 Registry — integrations/registry.py

SERVICE_DEFINITIONS: dict[str, ServiceDefinition] = {
    "grafana": GRAFANA_DEFINITION,
    "prometheus": PROMETHEUS_DEFINITION,
    "jellyfin": JELLYFIN_DEFINITION,
    "nextcloud": NEXTCLOUD_DEFINITION,
    "ssh_tasks": SSH_TASKS_DEFINITION,
}

def list_service_types() -> list[str]: ...
def get_service_definition(service_type: str) -> ServiceDefinition | None: ...
def get_widget_kind(service_type: str, widget_kind: str) -> WidgetKind | None: ...

The closed widgets/registry.py from Phase 1 is retired; its metadata is now derived from SERVICE_DEFINITIONS.

4. Source adapters

widgets/sources.py is refactored so each adapter resolves a service instance rather than reading get_settings():

class WidgetSource(Protocol):
    async def fetch(
        self,
        service: ServiceRecord,         # config + decrypted secrets
        widget_kind: str,
        config: dict[str, Any],
    ) -> dict[str, Any]: ...

• ServiceRecord is a runtime object built by ServiceStore carrying the decrypted secret dict in memory for the duration of the fetch.
• SOURCE_ADAPTERS is keyed by service_type.
• The data endpoint loads the widget's service_id, builds the ServiceRecord, then calls adapter.fetch(service, widget_kind, widget_config).

5. Encryption — services/secrets.py

from cryptography.fernet import Fernet, InvalidToken

def get_encryption_key() -> bytes:
    raw = os.environ.get("MANAGE_ENCRYPTION_KEY")
    if not raw:
        raise RuntimeError("MANAGE_ENCRYPTION_KEY is required")
    return raw.encode()

def encrypt_secrets(values: dict[str, str]) -> dict[str, str]: ...
def decrypt_secrets(blob: dict[str, str]) -> dict[str, str]: ...

• cryptography.fernet.Fernet (already a transitive dependency to verify).
• Startup validation: validate_auth_settings is extended to require MANAGE_ENCRYPTION_KEY and to reject an obviously invalid key.
• Secrets are encrypted field-by-field so the "which secrets are set" metadata is cheap to compute without decrypting.

6. REST API

Services

Method	Path	Handler
GET	/api/services/types	List service definitions (metadata + config schema + widget kinds).
GET	/api/services	List service instances (no plaintext secrets; only "set" flags).
POST	/api/services	Create instance (validates type, config, secret schema).
PUT	/api/services/{id}	Update instance.
DELETE	/api/services/{id}	Delete instance; cascade-deletes widgets referencing it in the same transaction.

Widgets (unchanged paths, new semantics)

Method	Path	Handler
GET	/api/widgets/instances	List widgets; each carries service_id, widget_kind.
POST/PUT/DELETE	/api/widgets/instances/{id}	CRUD; validation uses service definition's widget schema.
GET	/api/widgets/instances/{id}/data	Resolve service → adapter → fetch.

GET /api/widgets/types and /api/widgets/sources are removed; widget metadata is served via /api/services/types (widget kinds under each service).

7. Frontend

7.1 New frontend/src/integrations/registry.ts

Closed frontend registry mirroring the backend: serviceType → ServiceDefinition (config fields, secret fields with secret: true, widget kinds, default refresh intervals, and a component for the service page).

7.2 Service pages

• Route: /services/:serviceType/:serviceId (replaces /addons/:addonId).
• ServicePage looks up the definition and renders the service-specific component, a config editor, and the list of widget kinds that can be added to the dashboard.
• App.tsx removes the /addons/:addonId route; old addon URLs redirect to the default service of that type (or a not-found alert).

7.3 Dashboard config dialog

• "Add widget" flow becomes: pick service → pick widget kind → configure.
• The widget card shows the parent service name.

7.4 Types / API / hooks

• frontend/src/api/services.ts + hooks/useServices.ts for the services API.
• frontend/src/types/index.ts gains ServiceInstance, ServiceInstanceInput, ServiceTypeInfo, ServiceWidgetKind.

8. Migration and breaking changes

• DB migration on startup: add services table; add service_id / widget_kind columns to dashboard_widgets; drop addon_id.
• Machine app fields removed: jellyfin_url, jellyfin_user_id, jellyfin_api_key, jellyseerr_url, jellyseerr_api_key are dropped from machine records and the MonitoringMachine model. Machines keep SSH + node_exporter transport fields only.
• Env vars removed from config.py: grafana_url, prometheus_url. (Grafana/Prometheus URLs now live on service records.) MANAGE_ENCRYPTION_KEY is added as required.
• Default widget seeding is removed; a fresh install starts with no widgets. The user adds Jellyfin/Backups widgets after configuring the corresponding services.
• docs/REQUIREMENTS.md and README.md updated to describe services, the MANAGE_ENCRYPTION_KEY requirement, and the breaking upgrade note.

9. File-level plan

Create (backend)

File	Purpose
integrations/__init__.py	Package marker.
integrations/base.py	ServiceDefinition, WidgetKind, SecretField, ServiceConfigBase.
integrations/registry.py	Closed SERVICE_DEFINITIONS + helpers.
integrations/grafana.py, prometheus.py, jellyfin.py, nextcloud.py, ssh_tasks.py	One module per service.
services/secrets.py	Fernet encrypt/decrypt + key validation.
services/service_store.py	CRUD for services table; decrypt-on-read for adapters.
routers/services.py	/api/services* endpoints.
models/services.py	Pydantic request/response models.

Modify (backend)

File	Change
services/settings_store.py	services table; widget columns; drop machine app fields.
widgets/sources.py	Adapters take a ServiceRecord.
widgets/registry.py	Retired (metadata served by integrations/registry.py).
routers/widgets.py	Validate against service widget schema; resolve service on data fetch.
config.py	Remove grafana_url/prometheus_url; document MANAGE_ENCRYPTION_KEY (read in secrets.py).
main.py	Register services_router; validate encryption key on startup.
dependencies.py	Jellyfin/SSH resolution now goes via services, not machine app fields.

Create (frontend)

File	Purpose
integrations/registry.ts	Closed frontend service registry.
api/services.ts, hooks/useServices.ts	Services API + hooks.
pages/ServicePage.tsx	Generic /services/:type/:id page.
integrations/components/*	Per-service page components.

Modify (frontend)

File	Change
App.tsx	Replace /addons/:addonId with /services/:serviceType/:serviceId.
components/WidgetConfigDialog.tsx	Service → widget-kind picker.
widgets/registry.ts	Retired; widgets derived from service registry.
types/index.ts	Service types; widget gains service_id + widget_kind.
pages/Settings.tsx	Remove machine Jellyfin/Jellyseerr fields.

10. Slice boundaries (chained PRs)

Each slice keeps pytest, ruff, npm run lint, and npm run build green.

1. Backend foundation — encryption helper, integrations/ base + 5 definitions + registry, services table + store, /api/services* endpoints, tests. No widget changes yet.
2. Backend widget rebind — add service_id/widget_kind to widgets, refactor adapters to take a ServiceRecord, retire old widgets/registry.py, update data endpoint.
3. Frontend services runtime — types, API, hooks, integrations/registry.ts, service pages, route swap, remove addon pages.
4. Frontend dashboard + settings rework — service-based widget picker, drop machine app fields from Settings, remove grafana_url/prometheus_url from config, re-seed behavior, docs (README.md, REQUIREMENTS.md), changelog breaking-change note.

Estimated total: ~2,000–2,400 changed lines across four PRs.

11. Decisions resolved

1. Deleting a service that still has widgets → cascade delete. The store deletes every dashboard_widgets row referencing the service inside the same transaction as the service delete. Simple and safe in SQLite; no 409 pre-check.
2. MANAGE_ENCRYPTION_KEY dev default → always required. No fallback, even when AUTH_ENABLED=false. Startup fails fast if it is missing or not a valid Fernet key.
3. SSH task runner shape → multi-instance, reusable tasks, persisted run history. See §12 for the full model.

12. SSH task runner model

The SSH task runner is the most involved service type. Instances absorb the SSH task execution role currently held by machines; tasks stay global and reusable; every invocation is logged.

12.1 Instances

• service_type = "ssh_tasks".
• Each instance is an SSH endpoint: host, port, username, ssh_key_id, optional passphrase. Connection config lives on the service record; the SSH key itself stays in the existing saved-key registry (referenced by ssh_key_id).
• Multi-instance by design ("home server", "media box", …).

12.2 Tasks (global, reusable)

• Saved tasks remain a global registry (name, task_type shell/python, content, enabled). A task is not owned by an instance.
• Each task gains default_service_id (replaces the old default_machine_id) — the instance it targets by default. At run time the caller may override the target instance.
• A task can therefore run against any instance; the link is captured per-run.

12.3 Run history (logs)

A new service_task_runs table records every invocation:

CREATE TABLE IF NOT EXISTS service_task_runs (
    id TEXT PRIMARY KEY,
    task_id TEXT NOT NULL,
    service_id TEXT NOT NULL,
    status TEXT NOT NULL,        -- success | failure | timeout | error
    exit_status INTEGER,
    duration_ms INTEGER,
    stdout_tail TEXT,
    stderr_tail TEXT,
    error TEXT,
    created_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_service_task_runs_service ON service_task_runs(service_id, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_service_task_runs_task ON service_task_runs(task_id, created_at DESC);

• Populated by the SSH task adapter on every widget data fetch and by the Actions runner on manual runs.
• Surfaced on the instance's service page as a log/history list, and on the task detail as recent runs.
• Replaces the legacy saved_task_runs concept once the Actions page is rebuilt on services (Slice 4 / a follow-up).

12.4 SSH task widget

Widget config for ssh_tasks becomes { task_id, service_id? }:

• If service_id is omitted, the task's default_service_id is used.
• The adapter loads the task, resolves the instance, runs it, appends a service_task_runs row, and returns the trimmed stdout/stderr/exit status.

12.5 Relationship to machines

• The SSH task execution role moves out of machines into ssh_tasks instances.
• Machines keep their role for the File Browser and node_exporter monitoring transport in this change, to avoid also reworking Files/Monitoring here.
• Practical consequence: an SSH host used for both files and tasks may be defined twice (once as a machine, once as an ssh_tasks instance) during the transition. Unifying machines under services is an explicit follow-up change, not part of this one.