# Design: Runtime Service Registry **Change:** `service-registry` **Phase:** design **Date:** 2026-06-19 ## 1. Architecture overview ``` ┌────────────────────────────────────────────────────────────────────┐ │ Browser │ │ /services/:type/:id ─► ServicePage ─► frontend SERVICE_REGISTRY │ Dashboard ─► WidgetInstance ─► widget component │ └────────────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────────────┐ │ FastAPI /api/services + /api/widgets │ │ CRUD service instances · registry metadata · widget data │ └────────────────────────────────────────────────────────────────────┘ │ ┌──────────────────────────┼───────────────────────────┐ ▼ ▼ ▼ ServiceStore (SQLite) integrations/ definitions source adapters services table (Pydantic, closed registry) (resolve service dashboard_widgets table grafana/prometheus/jellyfin/ → decrypt → call) nextcloud/ssh_tasks ``` Two closed, compile-time registries cooperate: - **`integrations.registry.SERVICE_DEFINITIONS`** maps `service_type → ServiceDefinition`. Each definition declares config schema, secret fields, and widget kinds. - The widget types available to the dashboard are **derived** from `SERVICE_DEFINITIONS` at startup, not hand-maintained. ## 2. Backend data model ### 2.1 New `services` table Extend `SettingsStore.init_schema()`: ```sql CREATE TABLE IF NOT EXISTS services ( id TEXT PRIMARY KEY, service_type TEXT NOT NULL, name TEXT NOT NULL, config_json TEXT NOT NULL DEFAULT '{}', secrets_json TEXT NOT NULL DEFAULT '{}', -- encrypted blob (Fernet) enabled INTEGER NOT NULL DEFAULT 1, created_at INTEGER NOT NULL, updated_at INTEGER NOT NULL ); CREATE INDEX IF NOT EXISTS idx_services_type ON services(service_type); ``` - `config_json` — non-secret config validated against the service definition's `config_schema`. - `secrets_json` — a JSON object of `{field_name: ciphertext}` produced by the encryption helper. Never returned to the client in plaintext; only the boolean "is set" flags are surfaced. ### 2.2 `dashboard_widgets` schema change The existing table gains two columns and loses the global meaning of `widget_type`: ```sql ALTER TABLE dashboard_widgets ADD COLUMN service_id TEXT; ALTER TABLE dashboard_widgets ADD COLUMN widget_kind TEXT; ``` - `widget_kind` is the kind declared by the service definition (e.g. `"link"`, `"metric"`, `"activity"`). - `service_id` references `services.id`. - `widget_type` is retained temporarily as `"{service_type}.{widget_kind}"` for backwards-compatible reads during the transition, then dropped in the final slice. - The old `addon_id` column is dropped; addon identity is now `service_type`. ## 3. Service definitions (Pydantic, in repo) New package: `backend/src/media_library_viewer_api/integrations/` (chosen to avoid collision with the existing `services/` infra package). ### 3.1 Base classes — `integrations/base.py` ```python from typing import Any, ClassVar from pydantic import BaseModel, Field class SecretField(BaseModel): key: str label: str required: bool = False helper: str | None = None class WidgetKind(BaseModel): kind: str # e.g. "link", "metric", "activity" name: str description: str config_schema: dict[str, Any] # JSON schema for widget config default_config: dict[str, Any] = {} refresh_interval_ms: int = 0 class ServiceConfigBase(BaseModel): """Subclass per service to define non-secret config fields.""" class ServiceDefinition(BaseModel): service_type: ClassVar[str] name: ClassVar[str] description: ClassVar[str] config_schema: ClassVar[dict[str, Any]] secret_fields: ClassVar[list[SecretField]] widget_kinds: ClassVar[list[WidgetKind]] # Adapters are referenced by dotted path or registered separately; # see §4. The definition itself stays a pure data/schema object. ``` ### 3.2 Example — `integrations/grafana.py` ```python class GrafanaConfig(ServiceConfigBase): base_url: str = Field(..., description="Grafana base URL, e.g. https://grafana.example.com") GRAFANA_DEFINITION = ServiceDefinition( service_type="grafana", name="Grafana", description="Dashboards, metrics, and logs.", config_schema=GrafanaConfig.model_json_schema(), secret_fields=[SecretField(key="api_key", label="API key", helper="Service account token")], widget_kinds=[ WidgetKind( kind="link", name="Dashboard link", description="Deep-link to a Grafana dashboard or panel.", config_schema={ "type": "object", "properties": { "dashboard_uid": {"type": "string"}, "panel_id": {"type": "integer"}, }, "required": ["dashboard_uid"], }, default_config={"dashboard_uid": ""}, refresh_interval_ms=0, ), ], ) ``` Other definition modules follow the same shape: `prometheus.py`, `jellyfin.py`, `nextcloud.py`, `ssh_tasks.py`. ### 3.3 Registry — `integrations/registry.py` ```python SERVICE_DEFINITIONS: dict[str, ServiceDefinition] = { "grafana": GRAFANA_DEFINITION, "prometheus": PROMETHEUS_DEFINITION, "jellyfin": JELLYFIN_DEFINITION, "nextcloud": NEXTCLOUD_DEFINITION, "ssh_tasks": SSH_TASKS_DEFINITION, } def list_service_types() -> list[str]: ... def get_service_definition(service_type: str) -> ServiceDefinition | None: ... def get_widget_kind(service_type: str, widget_kind: str) -> WidgetKind | None: ... ``` The closed `widgets/registry.py` from Phase 1 is **retired**; its metadata is now derived from `SERVICE_DEFINITIONS`. ## 4. Source adapters `widgets/sources.py` is refactored so each adapter resolves a **service instance** rather than reading `get_settings()`: ```python class WidgetSource(Protocol): async def fetch( self, service: ServiceRecord, # config + decrypted secrets widget_kind: str, config: dict[str, Any], ) -> dict[str, Any]: ... ``` - `ServiceRecord` is a runtime object built by `ServiceStore` carrying the decrypted secret dict in memory for the duration of the fetch. - `SOURCE_ADAPTERS` is keyed by `service_type`. - The data endpoint loads the widget's `service_id`, builds the `ServiceRecord`, then calls `adapter.fetch(service, widget_kind, widget_config)`. ## 5. Encryption — `services/secrets.py` ```python from cryptography.fernet import Fernet, InvalidToken def get_encryption_key() -> bytes: raw = os.environ.get("MANAGE_ENCRYPTION_KEY") if not raw: raise RuntimeError("MANAGE_ENCRYPTION_KEY is required") return raw.encode() def encrypt_secrets(values: dict[str, str]) -> dict[str, str]: ... def decrypt_secrets(blob: dict[str, str]) -> dict[str, str]: ... ``` - `cryptography.fernet.Fernet` (already a transitive dependency to verify). - Startup validation: `validate_auth_settings` is extended to require `MANAGE_ENCRYPTION_KEY` and to reject an obviously invalid key. - Secrets are encrypted field-by-field so the "which secrets are set" metadata is cheap to compute without decrypting. ## 6. REST API ### Services | Method | Path | Handler | |--------|------|---------| | GET | `/api/services/types` | List service definitions (metadata + config schema + widget kinds). | | GET | `/api/services` | List service instances (no plaintext secrets; only "set" flags). | | POST | `/api/services` | Create instance (validates type, config, secret schema). | | PUT | `/api/services/{id}` | Update instance. | | DELETE | `/api/services/{id}` | Delete instance; **cascade-deletes** widgets referencing it in the same transaction. | ### Widgets (unchanged paths, new semantics) | Method | Path | Handler | |--------|------|---------| | GET | `/api/widgets/instances` | List widgets; each carries `service_id`, `widget_kind`. | | POST/PUT/DELETE | `/api/widgets/instances/{id}` | CRUD; validation uses service definition's widget schema. | | GET | `/api/widgets/instances/{id}/data` | Resolve service → adapter → fetch. | `GET /api/widgets/types` and `/api/widgets/sources` are removed; widget metadata is served via `/api/services/types` (widget kinds under each service). ## 7. Frontend ### 7.1 New `frontend/src/integrations/registry.ts` Closed frontend registry mirroring the backend: `serviceType → ServiceDefinition` (config fields, secret fields with `secret: true`, widget kinds, default refresh intervals, and a `component` for the service page). ### 7.2 Service pages - Route: `/services/:serviceType/:serviceId` (replaces `/addons/:addonId`). - `ServicePage` looks up the definition and renders the service-specific component, a config editor, and the list of widget kinds that can be added to the dashboard. - `App.tsx` removes the `/addons/:addonId` route; old addon URLs redirect to the default service of that type (or a not-found alert). ### 7.3 Dashboard config dialog - "Add widget" flow becomes: **pick service → pick widget kind → configure**. - The widget card shows the parent service name. ### 7.4 Types / API / hooks - `frontend/src/api/services.ts` + `hooks/useServices.ts` for the services API. - `frontend/src/types/index.ts` gains `ServiceInstance`, `ServiceInstanceInput`, `ServiceTypeInfo`, `ServiceWidgetKind`. ## 8. Migration and breaking changes - **DB migration on startup:** add `services` table; add `service_id` / `widget_kind` columns to `dashboard_widgets`; drop `addon_id`. - **Machine app fields removed:** `jellyfin_url`, `jellyfin_user_id`, `jellyfin_api_key`, `jellyseerr_url`, `jellyseerr_api_key` are dropped from machine records and the `MonitoringMachine` model. Machines keep SSH + node_exporter transport fields only. - **Env vars removed from `config.py`:** `grafana_url`, `prometheus_url`. (Grafana/Prometheus URLs now live on service records.) `MANAGE_ENCRYPTION_KEY` is added as required. - **Default widget seeding** is removed; a fresh install starts with no widgets. The user adds Jellyfin/Backups widgets after configuring the corresponding services. - **`docs/REQUIREMENTS.md` and `README.md`** updated to describe services, the `MANAGE_ENCRYPTION_KEY` requirement, and the breaking upgrade note. ## 9. File-level plan ### Create (backend) | File | Purpose | |------|---------| | `integrations/__init__.py` | Package marker. | | `integrations/base.py` | `ServiceDefinition`, `WidgetKind`, `SecretField`, `ServiceConfigBase`. | | `integrations/registry.py` | Closed `SERVICE_DEFINITIONS` + helpers. | | `integrations/grafana.py`, `prometheus.py`, `jellyfin.py`, `nextcloud.py`, `ssh_tasks.py` | One module per service. | | `services/secrets.py` | Fernet encrypt/decrypt + key validation. | | `services/service_store.py` | CRUD for `services` table; decrypt-on-read for adapters. | | `routers/services.py` | `/api/services*` endpoints. | | `models/services.py` | Pydantic request/response models. | ### Modify (backend) | File | Change | |-------|--------| | `services/settings_store.py` | `services` table; widget columns; drop machine app fields. | | `widgets/sources.py` | Adapters take a `ServiceRecord`. | | `widgets/registry.py` | Retired (metadata served by `integrations/registry.py`). | | `routers/widgets.py` | Validate against service widget schema; resolve service on data fetch. | | `config.py` | Remove `grafana_url`/`prometheus_url`; document `MANAGE_ENCRYPTION_KEY` (read in `secrets.py`). | | `main.py` | Register `services_router`; validate encryption key on startup. | | `dependencies.py` | Jellyfin/SSH resolution now goes via services, not machine app fields. | ### Create (frontend) | File | Purpose | |------|---------| | `integrations/registry.ts` | Closed frontend service registry. | | `api/services.ts`, `hooks/useServices.ts` | Services API + hooks. | | `pages/ServicePage.tsx` | Generic `/services/:type/:id` page. | | `integrations/components/*` | Per-service page components. | ### Modify (frontend) | File | Change | |-------|--------| | `App.tsx` | Replace `/addons/:addonId` with `/services/:serviceType/:serviceId`. | | `components/WidgetConfigDialog.tsx` | Service → widget-kind picker. | | `widgets/registry.ts` | Retired; widgets derived from service registry. | | `types/index.ts` | Service types; widget gains `service_id` + `widget_kind`. | | `pages/Settings.tsx` | Remove machine Jellyfin/Jellyseerr fields. | ## 10. Slice boundaries (chained PRs) Each slice keeps `pytest`, `ruff`, `npm run lint`, and `npm run build` green. 1. **Backend foundation** — encryption helper, `integrations/` base + 5 definitions + registry, `services` table + store, `/api/services*` endpoints, tests. No widget changes yet. 2. **Backend widget rebind** — add `service_id`/`widget_kind` to widgets, refactor adapters to take a `ServiceRecord`, retire old `widgets/registry.py`, update data endpoint. 3. **Frontend services runtime** — types, API, hooks, `integrations/registry.ts`, service pages, route swap, remove addon pages. 4. **Frontend dashboard + settings rework** — service-based widget picker, drop machine app fields from Settings, remove `grafana_url`/`prometheus_url` from config, re-seed behavior, docs (`README.md`, `REQUIREMENTS.md`), changelog breaking-change note. Estimated total: ~2,000–2,400 changed lines across four PRs. ## 11. Decisions resolved 1. **Deleting a service that still has widgets** → **cascade delete.** The store deletes every `dashboard_widgets` row referencing the service inside the same transaction as the service delete. Simple and safe in SQLite; no 409 pre-check. 2. **`MANAGE_ENCRYPTION_KEY` dev default** → **always required.** No fallback, even when `AUTH_ENABLED=false`. Startup fails fast if it is missing or not a valid Fernet key. 3. **SSH task runner shape** → **multi-instance, reusable tasks, persisted run history.** See §12 for the full model. ## 12. SSH task runner model The SSH task runner is the most involved service type. Instances absorb the SSH task execution role currently held by machines; tasks stay global and reusable; every invocation is logged. ### 12.1 Instances - `service_type = "ssh_tasks"`. - Each instance is an SSH endpoint: `host`, `port`, `username`, `ssh_key_id`, optional `passphrase`. Connection config lives on the service record; the SSH key itself stays in the existing saved-key registry (referenced by `ssh_key_id`). - Multi-instance by design ("home server", "media box", …). ### 12.2 Tasks (global, reusable) - Saved tasks remain a **global** registry (`name`, `task_type` shell/python, `content`, `enabled`). A task is **not** owned by an instance. - Each task gains `default_service_id` (replaces the old `default_machine_id`) — the instance it targets by default. At run time the caller may override the target instance. - A task can therefore run against any instance; the link is captured per-run. ### 12.3 Run history (logs) A new `service_task_runs` table records every invocation: ```sql CREATE TABLE IF NOT EXISTS service_task_runs ( id TEXT PRIMARY KEY, task_id TEXT NOT NULL, service_id TEXT NOT NULL, status TEXT NOT NULL, -- success | failure | timeout | error exit_status INTEGER, duration_ms INTEGER, stdout_tail TEXT, stderr_tail TEXT, error TEXT, created_at INTEGER NOT NULL ); CREATE INDEX IF NOT EXISTS idx_service_task_runs_service ON service_task_runs(service_id, created_at DESC); CREATE INDEX IF NOT EXISTS idx_service_task_runs_task ON service_task_runs(task_id, created_at DESC); ``` - Populated by the SSH task adapter on every widget data fetch and by the Actions runner on manual runs. - Surfaced on the instance's service page as a log/history list, and on the task detail as recent runs. - Replaces the legacy `saved_task_runs` concept once the Actions page is rebuilt on services (Slice 4 / a follow-up). ### 12.4 SSH task widget Widget config for `ssh_tasks` becomes `{ task_id, service_id? }`: - If `service_id` is omitted, the task's `default_service_id` is used. - The adapter loads the task, resolves the instance, runs it, appends a `service_task_runs` row, and returns the trimmed stdout/stderr/exit status. ### 12.5 Relationship to machines - The SSH task execution role moves **out of machines** into `ssh_tasks` instances. - Machines **keep** their role for the File Browser and node_exporter monitoring transport in this change, to avoid also reworking Files/Monitoring here. - Practical consequence: an SSH host used for both files and tasks may be defined twice (once as a machine, once as an ssh_tasks instance) during the transition. Unifying machines under services is an explicit **follow-up change**, not part of this one.