manage/openspec/changes/service-registry/design.md

Design: Runtime Service Registry

Change: service-registry Phase: design Date: 2026-06-19

1. Architecture overview

┌────────────────────────────────────────────────────────────────────┐
│                              Browser                                │
│   /services/:type/:id  ─►  ServicePage  ─►  frontend SERVICE_REGISTRY
│   Dashboard             ─►  WidgetInstance ─►  widget component     │
└────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌────────────────────────────────────────────────────────────────────┐
│                  FastAPI /api/services + /api/widgets               │
│   CRUD service instances · registry metadata · widget data         │
└────────────────────────────────────────────────────────────────────┘
                                  │
       ┌──────────────────────────┼───────────────────────────┐
       ▼                          ▼                           ▼
  ServiceStore (SQLite)    integrations/ definitions     source adapters
  services table           (Pydantic, closed registry)   (resolve service
  dashboard_widgets table  grafana/prometheus/jellyfin/   → decrypt → call)
                          nextcloud/ssh_tasks

Two closed, compile-time registries cooperate:

• integrations.registry.SERVICE_DEFINITIONS maps service_type → ServiceDefinition. Each definition declares config schema, secret fields, and widget kinds.
• The widget types available to the dashboard are derived from SERVICE_DEFINITIONS at startup, not hand-maintained.

2. Backend data model

2.1 New services table

Extend SettingsStore.init_schema():

CREATE TABLE IF NOT EXISTS services (
    id TEXT PRIMARY KEY,
    service_type TEXT NOT NULL,
    name TEXT NOT NULL,
    config_json TEXT NOT NULL DEFAULT '{}',
    secrets_json TEXT NOT NULL DEFAULT '{}',   -- encrypted blob (Fernet)
    enabled INTEGER NOT NULL DEFAULT 1,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_services_type ON services(service_type);

• config_json — non-secret config validated against the service definition's config_schema.
• secrets_json — a JSON object of {field_name: ciphertext} produced by the encryption helper. Never returned to the client in plaintext; only the boolean "is set" flags are surfaced.

2.2 dashboard_widgets schema change

The existing table gains two columns and loses the global meaning of widget_type:

ALTER TABLE dashboard_widgets ADD COLUMN service_id TEXT;
ALTER TABLE dashboard_widgets ADD COLUMN widget_kind TEXT;

• widget_kind is the kind declared by the service definition (e.g. "link", "metric", "activity").
• service_id references services.id.
• widget_type is retained temporarily as "{service_type}.{widget_kind}" for backwards-compatible reads during the transition, then dropped in the final slice.
• The old addon_id column is dropped; addon identity is now service_type.

3. Service definitions (Pydantic, in repo)

New package: backend/src/media_library_viewer_api/integrations/ (chosen to avoid collision with the existing services/ infra package).

3.1 Base classes — integrations/base.py

from typing import Any, ClassVar
from pydantic import BaseModel, Field

class SecretField(BaseModel):
    key: str
    label: str
    required: bool = False
    helper: str | None = None

class WidgetKind(BaseModel):
    kind: str                       # e.g. "link", "metric", "activity"
    name: str
    description: str
    config_schema: dict[str, Any]   # JSON schema for widget config
    default_config: dict[str, Any] = {}
    refresh_interval_ms: int = 0

class ServiceConfigBase(BaseModel):
    """Subclass per service to define non-secret config fields."""

class ServiceDefinition(BaseModel):
    service_type: ClassVar[str]
    name: ClassVar[str]
    description: ClassVar[str]
    config_schema: ClassVar[dict[str, Any]]
    secret_fields: ClassVar[list[SecretField]]
    widget_kinds: ClassVar[list[WidgetKind]]

    # Adapters are referenced by dotted path or registered separately;
    # see §4. The definition itself stays a pure data/schema object.

3.2 Example — integrations/grafana.py

class GrafanaConfig(ServiceConfigBase):
    base_url: str = Field(..., description="Grafana base URL, e.g. https://grafana.example.com")

GRAFANA_DEFINITION = ServiceDefinition(
    service_type="grafana",
    name="Grafana",
    description="Dashboards, metrics, and logs.",
    config_schema=GrafanaConfig.model_json_schema(),
    secret_fields=[SecretField(key="api_key", label="API key", helper="Service account token")],
    widget_kinds=[
        WidgetKind(
            kind="link",
            name="Dashboard link",
            description="Deep-link to a Grafana dashboard or panel.",
            config_schema={
                "type": "object",
                "properties": {
                    "dashboard_uid": {"type": "string"},
                    "panel_id": {"type": "integer"},
                },
                "required": ["dashboard_uid"],
            },
            default_config={"dashboard_uid": ""},
            refresh_interval_ms=0,
        ),
    ],
)

Other definition modules follow the same shape: prometheus.py, jellyfin.py, nextcloud.py, ssh_tasks.py.

3.3 Registry — integrations/registry.py

SERVICE_DEFINITIONS: dict[str, ServiceDefinition] = {
    "grafana": GRAFANA_DEFINITION,
    "prometheus": PROMETHEUS_DEFINITION,
    "jellyfin": JELLYFIN_DEFINITION,
    "nextcloud": NEXTCLOUD_DEFINITION,
    "ssh_tasks": SSH_TASKS_DEFINITION,
}

def list_service_types() -> list[str]: ...
def get_service_definition(service_type: str) -> ServiceDefinition | None: ...
def get_widget_kind(service_type: str, widget_kind: str) -> WidgetKind | None: ...

The closed widgets/registry.py from Phase 1 is retired; its metadata is now derived from SERVICE_DEFINITIONS.

4. Source adapters

widgets/sources.py is refactored so each adapter resolves a service instance rather than reading get_settings():

class WidgetSource(Protocol):
    async def fetch(
        self,
        service: ServiceRecord,         # config + decrypted secrets
        widget_kind: str,
        config: dict[str, Any],
    ) -> dict[str, Any]: ...

• ServiceRecord is a runtime object built by ServiceStore carrying the decrypted secret dict in memory for the duration of the fetch.
• SOURCE_ADAPTERS is keyed by service_type.
• The data endpoint loads the widget's service_id, builds the ServiceRecord, then calls adapter.fetch(service, widget_kind, widget_config).

5. Encryption — services/secrets.py

from cryptography.fernet import Fernet, InvalidToken

def get_encryption_key() -> bytes:
    raw = os.environ.get("MANAGE_ENCRYPTION_KEY")
    if not raw:
        raise RuntimeError("MANAGE_ENCRYPTION_KEY is required")
    return raw.encode()

def encrypt_secrets(values: dict[str, str]) -> dict[str, str]: ...
def decrypt_secrets(blob: dict[str, str]) -> dict[str, str]: ...

• cryptography.fernet.Fernet (already a transitive dependency to verify).
• Startup validation: validate_auth_settings is extended to require MANAGE_ENCRYPTION_KEY and to reject an obviously invalid key.
• Secrets are encrypted field-by-field so the "which secrets are set" metadata is cheap to compute without decrypting.

6. REST API

Services

Method	Path	Handler
GET	/api/services/types	List service definitions (metadata + config schema + widget kinds).
GET	/api/services	List service instances (no plaintext secrets; only "set" flags).
POST	/api/services	Create instance (validates type, config, secret schema).
PUT	/api/services/{id}	Update instance.
DELETE	/api/services/{id}	Delete instance (and refuse while widgets reference it, or cascade).

Widgets (unchanged paths, new semantics)

Method	Path	Handler
GET	/api/widgets/instances	List widgets; each carries service_id, widget_kind.
POST/PUT/DELETE	/api/widgets/instances/{id}	CRUD; validation uses service definition's widget schema.
GET	/api/widgets/instances/{id}/data	Resolve service → adapter → fetch.

GET /api/widgets/types and /api/widgets/sources are removed; widget metadata is served via /api/services/types (widget kinds under each service).

7. Frontend

7.1 New frontend/src/integrations/registry.ts

Closed frontend registry mirroring the backend: serviceType → ServiceDefinition (config fields, secret fields with secret: true, widget kinds, default refresh intervals, and a component for the service page).

7.2 Service pages

• Route: /services/:serviceType/:serviceId (replaces /addons/:addonId).
• ServicePage looks up the definition and renders the service-specific component, a config editor, and the list of widget kinds that can be added to the dashboard.
• App.tsx removes the /addons/:addonId route; old addon URLs redirect to the default service of that type (or a not-found alert).

7.3 Dashboard config dialog

• "Add widget" flow becomes: pick service → pick widget kind → configure.
• The widget card shows the parent service name.

7.4 Types / API / hooks

• frontend/src/api/services.ts + hooks/useServices.ts for the services API.
• frontend/src/types/index.ts gains ServiceInstance, ServiceInstanceInput, ServiceTypeInfo, ServiceWidgetKind.

8. Migration and breaking changes

• DB migration on startup: add services table; add service_id / widget_kind columns to dashboard_widgets; drop addon_id.
• Machine app fields removed: jellyfin_url, jellyfin_user_id, jellyfin_api_key, jellyseerr_url, jellyseerr_api_key are dropped from machine records and the MonitoringMachine model. Machines keep SSH + node_exporter transport fields only.
• Env vars removed from config.py: grafana_url, prometheus_url. (Grafana/Prometheus URLs now live on service records.) MANAGE_ENCRYPTION_KEY is added as required.
• Default widget seeding is removed; a fresh install starts with no widgets. The user adds Jellyfin/Backups widgets after configuring the corresponding services.
• docs/REQUIREMENTS.md and README.md updated to describe services, the MANAGE_ENCRYPTION_KEY requirement, and the breaking upgrade note.

9. File-level plan

Create (backend)

File	Purpose
integrations/__init__.py	Package marker.
integrations/base.py	ServiceDefinition, WidgetKind, SecretField, ServiceConfigBase.
integrations/registry.py	Closed SERVICE_DEFINITIONS + helpers.
integrations/grafana.py, prometheus.py, jellyfin.py, nextcloud.py, ssh_tasks.py	One module per service.
services/secrets.py	Fernet encrypt/decrypt + key validation.
services/service_store.py	CRUD for services table; decrypt-on-read for adapters.
routers/services.py	/api/services* endpoints.
models/services.py	Pydantic request/response models.

Modify (backend)

File	Change
services/settings_store.py	services table; widget columns; drop machine app fields.
widgets/sources.py	Adapters take a ServiceRecord.
widgets/registry.py	Retired (metadata served by integrations/registry.py).
routers/widgets.py	Validate against service widget schema; resolve service on data fetch.
config.py	Remove grafana_url/prometheus_url; document MANAGE_ENCRYPTION_KEY (read in secrets.py).
main.py	Register services_router; validate encryption key on startup.
dependencies.py	Jellyfin/SSH resolution now goes via services, not machine app fields.

Create (frontend)

File	Purpose
integrations/registry.ts	Closed frontend service registry.
api/services.ts, hooks/useServices.ts	Services API + hooks.
pages/ServicePage.tsx	Generic /services/:type/:id page.
integrations/components/*	Per-service page components.

Modify (frontend)

File	Change
App.tsx	Replace /addons/:addonId with /services/:serviceType/:serviceId.
components/WidgetConfigDialog.tsx	Service → widget-kind picker.
widgets/registry.ts	Retired; widgets derived from service registry.
types/index.ts	Service types; widget gains service_id + widget_kind.
pages/Settings.tsx	Remove machine Jellyfin/Jellyseerr fields.

10. Slice boundaries (chained PRs)

Each slice keeps pytest, ruff, npm run lint, and npm run build green.

1. Backend foundation — encryption helper, integrations/ base + 5 definitions + registry, services table + store, /api/services* endpoints, tests. No widget changes yet.
2. Backend widget rebind — add service_id/widget_kind to widgets, refactor adapters to take a ServiceRecord, retire old widgets/registry.py, update data endpoint.
3. Frontend services runtime — types, API, hooks, integrations/registry.ts, service pages, route swap, remove addon pages.
4. Frontend dashboard + settings rework — service-based widget picker, drop machine app fields from Settings, remove grafana_url/prometheus_url from config, re-seed behavior, docs (README.md, REQUIREMENTS.md), changelog breaking-change note.

Estimated total: ~2,000–2,400 changed lines across four PRs.

11. Open questions to resolve before apply

1. Should deleting a service that still has widgets block (return 409) or cascade-delete the widgets? Recommend block with 409 and require the user to remove widgets first.
2. Should MANAGE_ENCRYPTION_KEY have a development default (e.g. derived from a fixed dev key when AUTH_ENABLED=false)? Recommend no — require it always to avoid accidental plaintext in dev.
3. Does the SSH task runner service hold the SSH key reference, or does it reference a machine? Recommend the service record holds machine_id (transport) + optional task-scoped overrides; the saved-task registry stays unchanged.