Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 9459de5c07 | |||
| 9782280a03 | |||
| 0ad6a04053 |
@@ -0,0 +1,420 @@
|
|||||||
|
# Design: Runtime Service Registry
|
||||||
|
|
||||||
|
**Change:** `service-registry`
|
||||||
|
**Phase:** design
|
||||||
|
**Date:** 2026-06-19
|
||||||
|
|
||||||
|
## 1. Architecture overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Browser │
|
||||||
|
│ /services/:type/:id ─► ServicePage ─► frontend SERVICE_REGISTRY
|
||||||
|
│ Dashboard ─► WidgetInstance ─► widget component │
|
||||||
|
└────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ FastAPI /api/services + /api/widgets │
|
||||||
|
│ CRUD service instances · registry metadata · widget data │
|
||||||
|
└────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
┌──────────────────────────┼───────────────────────────┐
|
||||||
|
▼ ▼ ▼
|
||||||
|
ServiceStore (SQLite) integrations/ definitions source adapters
|
||||||
|
services table (Pydantic, closed registry) (resolve service
|
||||||
|
dashboard_widgets table grafana/prometheus/jellyfin/ → decrypt → call)
|
||||||
|
nextcloud/ssh_tasks
|
||||||
|
```
|
||||||
|
|
||||||
|
Two closed, compile-time registries cooperate:
|
||||||
|
|
||||||
|
- **`integrations.registry.SERVICE_DEFINITIONS`** maps `service_type → ServiceDefinition`.
|
||||||
|
Each definition declares config schema, secret fields, and widget kinds.
|
||||||
|
- The widget types available to the dashboard are **derived** from
|
||||||
|
`SERVICE_DEFINITIONS` at startup, not hand-maintained.
|
||||||
|
|
||||||
|
## 2. Backend data model
|
||||||
|
|
||||||
|
### 2.1 New `services` table
|
||||||
|
|
||||||
|
Extend `SettingsStore.init_schema()`:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS services (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
service_type TEXT NOT NULL,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
config_json TEXT NOT NULL DEFAULT '{}',
|
||||||
|
secrets_json TEXT NOT NULL DEFAULT '{}', -- encrypted blob (Fernet)
|
||||||
|
enabled INTEGER NOT NULL DEFAULT 1,
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_services_type ON services(service_type);
|
||||||
|
```
|
||||||
|
|
||||||
|
- `config_json` — non-secret config validated against the service definition's
|
||||||
|
`config_schema`.
|
||||||
|
- `secrets_json` — a JSON object of `{field_name: ciphertext}` produced by the
|
||||||
|
encryption helper. Never returned to the client in plaintext; only the boolean
|
||||||
|
"is set" flags are surfaced.
|
||||||
|
|
||||||
|
### 2.2 `dashboard_widgets` schema change
|
||||||
|
|
||||||
|
The existing table gains two columns and loses the global meaning of `widget_type`:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
ALTER TABLE dashboard_widgets ADD COLUMN service_id TEXT;
|
||||||
|
ALTER TABLE dashboard_widgets ADD COLUMN widget_kind TEXT;
|
||||||
|
```
|
||||||
|
|
||||||
|
- `widget_kind` is the kind declared by the service definition (e.g. `"link"`,
|
||||||
|
`"metric"`, `"activity"`).
|
||||||
|
- `service_id` references `services.id`.
|
||||||
|
- `widget_type` is retained temporarily as `"{service_type}.{widget_kind}"` for
|
||||||
|
backwards-compatible reads during the transition, then dropped in the final slice.
|
||||||
|
- The old `addon_id` column is dropped; addon identity is now `service_type`.
|
||||||
|
|
||||||
|
## 3. Service definitions (Pydantic, in repo)
|
||||||
|
|
||||||
|
New package: `backend/src/media_library_viewer_api/integrations/`
|
||||||
|
(chosen to avoid collision with the existing `services/` infra package).
|
||||||
|
|
||||||
|
### 3.1 Base classes — `integrations/base.py`
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import Any, ClassVar
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
class SecretField(BaseModel):
|
||||||
|
key: str
|
||||||
|
label: str
|
||||||
|
required: bool = False
|
||||||
|
helper: str | None = None
|
||||||
|
|
||||||
|
class WidgetKind(BaseModel):
|
||||||
|
kind: str # e.g. "link", "metric", "activity"
|
||||||
|
name: str
|
||||||
|
description: str
|
||||||
|
config_schema: dict[str, Any] # JSON schema for widget config
|
||||||
|
default_config: dict[str, Any] = {}
|
||||||
|
refresh_interval_ms: int = 0
|
||||||
|
|
||||||
|
class ServiceConfigBase(BaseModel):
|
||||||
|
"""Subclass per service to define non-secret config fields."""
|
||||||
|
|
||||||
|
class ServiceDefinition(BaseModel):
|
||||||
|
service_type: ClassVar[str]
|
||||||
|
name: ClassVar[str]
|
||||||
|
description: ClassVar[str]
|
||||||
|
config_schema: ClassVar[dict[str, Any]]
|
||||||
|
secret_fields: ClassVar[list[SecretField]]
|
||||||
|
widget_kinds: ClassVar[list[WidgetKind]]
|
||||||
|
|
||||||
|
# Adapters are referenced by dotted path or registered separately;
|
||||||
|
# see §4. The definition itself stays a pure data/schema object.
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Example — `integrations/grafana.py`
|
||||||
|
|
||||||
|
```python
|
||||||
|
class GrafanaConfig(ServiceConfigBase):
|
||||||
|
base_url: str = Field(..., description="Grafana base URL, e.g. https://grafana.example.com")
|
||||||
|
|
||||||
|
GRAFANA_DEFINITION = ServiceDefinition(
|
||||||
|
service_type="grafana",
|
||||||
|
name="Grafana",
|
||||||
|
description="Dashboards, metrics, and logs.",
|
||||||
|
config_schema=GrafanaConfig.model_json_schema(),
|
||||||
|
secret_fields=[SecretField(key="api_key", label="API key", helper="Service account token")],
|
||||||
|
widget_kinds=[
|
||||||
|
WidgetKind(
|
||||||
|
kind="link",
|
||||||
|
name="Dashboard link",
|
||||||
|
description="Deep-link to a Grafana dashboard or panel.",
|
||||||
|
config_schema={
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"dashboard_uid": {"type": "string"},
|
||||||
|
"panel_id": {"type": "integer"},
|
||||||
|
},
|
||||||
|
"required": ["dashboard_uid"],
|
||||||
|
},
|
||||||
|
default_config={"dashboard_uid": ""},
|
||||||
|
refresh_interval_ms=0,
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Other definition modules follow the same shape: `prometheus.py`, `jellyfin.py`,
|
||||||
|
`nextcloud.py`, `ssh_tasks.py`.
|
||||||
|
|
||||||
|
### 3.3 Registry — `integrations/registry.py`
|
||||||
|
|
||||||
|
```python
|
||||||
|
SERVICE_DEFINITIONS: dict[str, ServiceDefinition] = {
|
||||||
|
"grafana": GRAFANA_DEFINITION,
|
||||||
|
"prometheus": PROMETHEUS_DEFINITION,
|
||||||
|
"jellyfin": JELLYFIN_DEFINITION,
|
||||||
|
"nextcloud": NEXTCLOUD_DEFINITION,
|
||||||
|
"ssh_tasks": SSH_TASKS_DEFINITION,
|
||||||
|
}
|
||||||
|
|
||||||
|
def list_service_types() -> list[str]: ...
|
||||||
|
def get_service_definition(service_type: str) -> ServiceDefinition | None: ...
|
||||||
|
def get_widget_kind(service_type: str, widget_kind: str) -> WidgetKind | None: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
The closed `widgets/registry.py` from Phase 1 is **retired**; its metadata is now derived
|
||||||
|
from `SERVICE_DEFINITIONS`.
|
||||||
|
|
||||||
|
## 4. Source adapters
|
||||||
|
|
||||||
|
`widgets/sources.py` is refactored so each adapter resolves a **service instance** rather
|
||||||
|
than reading `get_settings()`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class WidgetSource(Protocol):
|
||||||
|
async def fetch(
|
||||||
|
self,
|
||||||
|
service: ServiceRecord, # config + decrypted secrets
|
||||||
|
widget_kind: str,
|
||||||
|
config: dict[str, Any],
|
||||||
|
) -> dict[str, Any]: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
- `ServiceRecord` is a runtime object built by `ServiceStore` carrying the decrypted
|
||||||
|
secret dict in memory for the duration of the fetch.
|
||||||
|
- `SOURCE_ADAPTERS` is keyed by `service_type`.
|
||||||
|
- The data endpoint loads the widget's `service_id`, builds the `ServiceRecord`, then
|
||||||
|
calls `adapter.fetch(service, widget_kind, widget_config)`.
|
||||||
|
|
||||||
|
## 5. Encryption — `services/secrets.py`
|
||||||
|
|
||||||
|
```python
|
||||||
|
from cryptography.fernet import Fernet, InvalidToken
|
||||||
|
|
||||||
|
def get_encryption_key() -> bytes:
|
||||||
|
raw = os.environ.get("MANAGE_ENCRYPTION_KEY")
|
||||||
|
if not raw:
|
||||||
|
raise RuntimeError("MANAGE_ENCRYPTION_KEY is required")
|
||||||
|
return raw.encode()
|
||||||
|
|
||||||
|
def encrypt_secrets(values: dict[str, str]) -> dict[str, str]: ...
|
||||||
|
def decrypt_secrets(blob: dict[str, str]) -> dict[str, str]: ...
|
||||||
|
```
|
||||||
|
|
||||||
|
- `cryptography.fernet.Fernet` (already a transitive dependency to verify).
|
||||||
|
- Startup validation: `validate_auth_settings` is extended to require
|
||||||
|
`MANAGE_ENCRYPTION_KEY` and to reject an obviously invalid key.
|
||||||
|
- Secrets are encrypted field-by-field so the "which secrets are set" metadata is cheap
|
||||||
|
to compute without decrypting.
|
||||||
|
|
||||||
|
## 6. REST API
|
||||||
|
|
||||||
|
### Services
|
||||||
|
|
||||||
|
| Method | Path | Handler |
|
||||||
|
|--------|------|---------|
|
||||||
|
| GET | `/api/services/types` | List service definitions (metadata + config schema + widget kinds). |
|
||||||
|
| GET | `/api/services` | List service instances (no plaintext secrets; only "set" flags). |
|
||||||
|
| POST | `/api/services` | Create instance (validates type, config, secret schema). |
|
||||||
|
| PUT | `/api/services/{id}` | Update instance. |
|
||||||
|
| DELETE | `/api/services/{id}` | Delete instance; **cascade-deletes** widgets referencing it in the same transaction. |
|
||||||
|
|
||||||
|
### Widgets (unchanged paths, new semantics)
|
||||||
|
|
||||||
|
| Method | Path | Handler |
|
||||||
|
|--------|------|---------|
|
||||||
|
| GET | `/api/widgets/instances` | List widgets; each carries `service_id`, `widget_kind`. |
|
||||||
|
| POST/PUT/DELETE | `/api/widgets/instances/{id}` | CRUD; validation uses service definition's widget schema. |
|
||||||
|
| GET | `/api/widgets/instances/{id}/data` | Resolve service → adapter → fetch. |
|
||||||
|
|
||||||
|
`GET /api/widgets/types` and `/api/widgets/sources` are removed; widget metadata is
|
||||||
|
served via `/api/services/types` (widget kinds under each service).
|
||||||
|
|
||||||
|
## 7. Frontend
|
||||||
|
|
||||||
|
### 7.1 New `frontend/src/integrations/registry.ts`
|
||||||
|
|
||||||
|
Closed frontend registry mirroring the backend: `serviceType → ServiceDefinition`
|
||||||
|
(config fields, secret fields with `secret: true`, widget kinds, default refresh
|
||||||
|
intervals, and a `component` for the service page).
|
||||||
|
|
||||||
|
### 7.2 Service pages
|
||||||
|
|
||||||
|
- Route: `/services/:serviceType/:serviceId` (replaces `/addons/:addonId`).
|
||||||
|
- `ServicePage` looks up the definition and renders the service-specific component,
|
||||||
|
a config editor, and the list of widget kinds that can be added to the dashboard.
|
||||||
|
- `App.tsx` removes the `/addons/:addonId` route; old addon URLs redirect to the
|
||||||
|
default service of that type (or a not-found alert).
|
||||||
|
|
||||||
|
### 7.3 Dashboard config dialog
|
||||||
|
|
||||||
|
- "Add widget" flow becomes: **pick service → pick widget kind → configure**.
|
||||||
|
- The widget card shows the parent service name.
|
||||||
|
|
||||||
|
### 7.4 Types / API / hooks
|
||||||
|
|
||||||
|
- `frontend/src/api/services.ts` + `hooks/useServices.ts` for the services API.
|
||||||
|
- `frontend/src/types/index.ts` gains `ServiceInstance`, `ServiceInstanceInput`,
|
||||||
|
`ServiceTypeInfo`, `ServiceWidgetKind`.
|
||||||
|
|
||||||
|
## 8. Migration and breaking changes
|
||||||
|
|
||||||
|
- **DB migration on startup:** add `services` table; add `service_id` / `widget_kind`
|
||||||
|
columns to `dashboard_widgets`; drop `addon_id`.
|
||||||
|
- **Machine app fields removed:** `jellyfin_url`, `jellyfin_user_id`, `jellyfin_api_key`,
|
||||||
|
`jellyseerr_url`, `jellyseerr_api_key` are dropped from machine records and the
|
||||||
|
`MonitoringMachine` model. Machines keep SSH + node_exporter transport fields only.
|
||||||
|
- **Env vars removed from `config.py`:** `grafana_url`, `prometheus_url`. (Grafana/Prometheus
|
||||||
|
URLs now live on service records.) `MANAGE_ENCRYPTION_KEY` is added as required.
|
||||||
|
- **Default widget seeding** is removed; a fresh install starts with no widgets. The user
|
||||||
|
adds Jellyfin/Backups widgets after configuring the corresponding services.
|
||||||
|
- **`docs/REQUIREMENTS.md` and `README.md`** updated to describe services, the
|
||||||
|
`MANAGE_ENCRYPTION_KEY` requirement, and the breaking upgrade note.
|
||||||
|
|
||||||
|
## 9. File-level plan
|
||||||
|
|
||||||
|
### Create (backend)
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `integrations/__init__.py` | Package marker. |
|
||||||
|
| `integrations/base.py` | `ServiceDefinition`, `WidgetKind`, `SecretField`, `ServiceConfigBase`. |
|
||||||
|
| `integrations/registry.py` | Closed `SERVICE_DEFINITIONS` + helpers. |
|
||||||
|
| `integrations/grafana.py`, `prometheus.py`, `jellyfin.py`, `nextcloud.py`, `ssh_tasks.py` | One module per service. |
|
||||||
|
| `services/secrets.py` | Fernet encrypt/decrypt + key validation. |
|
||||||
|
| `services/service_store.py` | CRUD for `services` table; decrypt-on-read for adapters. |
|
||||||
|
| `routers/services.py` | `/api/services*` endpoints. |
|
||||||
|
| `models/services.py` | Pydantic request/response models. |
|
||||||
|
|
||||||
|
### Modify (backend)
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|-------|--------|
|
||||||
|
| `services/settings_store.py` | `services` table; widget columns; drop machine app fields. |
|
||||||
|
| `widgets/sources.py` | Adapters take a `ServiceRecord`. |
|
||||||
|
| `widgets/registry.py` | Retired (metadata served by `integrations/registry.py`). |
|
||||||
|
| `routers/widgets.py` | Validate against service widget schema; resolve service on data fetch. |
|
||||||
|
| `config.py` | Remove `grafana_url`/`prometheus_url`; document `MANAGE_ENCRYPTION_KEY` (read in `secrets.py`). |
|
||||||
|
| `main.py` | Register `services_router`; validate encryption key on startup. |
|
||||||
|
| `dependencies.py` | Jellyfin/SSH resolution now goes via services, not machine app fields. |
|
||||||
|
|
||||||
|
### Create (frontend)
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `integrations/registry.ts` | Closed frontend service registry. |
|
||||||
|
| `api/services.ts`, `hooks/useServices.ts` | Services API + hooks. |
|
||||||
|
| `pages/ServicePage.tsx` | Generic `/services/:type/:id` page. |
|
||||||
|
| `integrations/components/*` | Per-service page components. |
|
||||||
|
|
||||||
|
### Modify (frontend)
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|-------|--------|
|
||||||
|
| `App.tsx` | Replace `/addons/:addonId` with `/services/:serviceType/:serviceId`. |
|
||||||
|
| `components/WidgetConfigDialog.tsx` | Service → widget-kind picker. |
|
||||||
|
| `widgets/registry.ts` | Retired; widgets derived from service registry. |
|
||||||
|
| `types/index.ts` | Service types; widget gains `service_id` + `widget_kind`. |
|
||||||
|
| `pages/Settings.tsx` | Remove machine Jellyfin/Jellyseerr fields. |
|
||||||
|
|
||||||
|
## 10. Slice boundaries (chained PRs)
|
||||||
|
|
||||||
|
Each slice keeps `pytest`, `ruff`, `npm run lint`, and `npm run build` green.
|
||||||
|
|
||||||
|
1. **Backend foundation** — encryption helper, `integrations/` base + 5 definitions +
|
||||||
|
registry, `services` table + store, `/api/services*` endpoints, tests. No widget
|
||||||
|
changes yet.
|
||||||
|
2. **Backend widget rebind** — add `service_id`/`widget_kind` to widgets, refactor
|
||||||
|
adapters to take a `ServiceRecord`, retire old `widgets/registry.py`, update data
|
||||||
|
endpoint.
|
||||||
|
3. **Frontend services runtime** — types, API, hooks, `integrations/registry.ts`,
|
||||||
|
service pages, route swap, remove addon pages.
|
||||||
|
4. **Frontend dashboard + settings rework** — service-based widget picker, drop machine
|
||||||
|
app fields from Settings, remove `grafana_url`/`prometheus_url` from config,
|
||||||
|
re-seed behavior, docs (`README.md`, `REQUIREMENTS.md`), changelog breaking-change
|
||||||
|
note.
|
||||||
|
|
||||||
|
Estimated total: ~2,000–2,400 changed lines across four PRs.
|
||||||
|
|
||||||
|
## 11. Decisions resolved
|
||||||
|
|
||||||
|
1. **Deleting a service that still has widgets** → **cascade delete.** The store deletes
|
||||||
|
every `dashboard_widgets` row referencing the service inside the same transaction as
|
||||||
|
the service delete. Simple and safe in SQLite; no 409 pre-check.
|
||||||
|
2. **`MANAGE_ENCRYPTION_KEY` dev default** → **always required.** No fallback, even when
|
||||||
|
`AUTH_ENABLED=false`. Startup fails fast if it is missing or not a valid Fernet key.
|
||||||
|
3. **SSH task runner shape** → **multi-instance, reusable tasks, persisted run history.**
|
||||||
|
See §12 for the full model.
|
||||||
|
|
||||||
|
## 12. SSH task runner model
|
||||||
|
|
||||||
|
The SSH task runner is the most involved service type. Instances absorb the SSH task
|
||||||
|
execution role currently held by machines; tasks stay global and reusable; every
|
||||||
|
invocation is logged.
|
||||||
|
|
||||||
|
### 12.1 Instances
|
||||||
|
|
||||||
|
- `service_type = "ssh_tasks"`.
|
||||||
|
- Each instance is an SSH endpoint: `host`, `port`, `username`, `ssh_key_id`, optional
|
||||||
|
`passphrase`. Connection config lives on the service record; the SSH key itself stays
|
||||||
|
in the existing saved-key registry (referenced by `ssh_key_id`).
|
||||||
|
- Multi-instance by design ("home server", "media box", …).
|
||||||
|
|
||||||
|
### 12.2 Tasks (global, reusable)
|
||||||
|
|
||||||
|
- Saved tasks remain a **global** registry (`name`, `task_type` shell/python, `content`,
|
||||||
|
`enabled`). A task is **not** owned by an instance.
|
||||||
|
- Each task gains `default_service_id` (replaces the old `default_machine_id`) — the
|
||||||
|
instance it targets by default. At run time the caller may override the target
|
||||||
|
instance.
|
||||||
|
- A task can therefore run against any instance; the link is captured per-run.
|
||||||
|
|
||||||
|
### 12.3 Run history (logs)
|
||||||
|
|
||||||
|
A new `service_task_runs` table records every invocation:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS service_task_runs (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
task_id TEXT NOT NULL,
|
||||||
|
service_id TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL, -- success | failure | timeout | error
|
||||||
|
exit_status INTEGER,
|
||||||
|
duration_ms INTEGER,
|
||||||
|
stdout_tail TEXT,
|
||||||
|
stderr_tail TEXT,
|
||||||
|
error TEXT,
|
||||||
|
created_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_service_task_runs_service ON service_task_runs(service_id, created_at DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_service_task_runs_task ON service_task_runs(task_id, created_at DESC);
|
||||||
|
```
|
||||||
|
|
||||||
|
- Populated by the SSH task adapter on every widget data fetch and by the Actions
|
||||||
|
runner on manual runs.
|
||||||
|
- Surfaced on the instance's service page as a log/history list, and on the task detail
|
||||||
|
as recent runs.
|
||||||
|
- Replaces the legacy `saved_task_runs` concept once the Actions page is rebuilt on
|
||||||
|
services (Slice 4 / a follow-up).
|
||||||
|
|
||||||
|
### 12.4 SSH task widget
|
||||||
|
|
||||||
|
Widget config for `ssh_tasks` becomes `{ task_id, service_id? }`:
|
||||||
|
|
||||||
|
- If `service_id` is omitted, the task's `default_service_id` is used.
|
||||||
|
- The adapter loads the task, resolves the instance, runs it, appends a
|
||||||
|
`service_task_runs` row, and returns the trimmed stdout/stderr/exit status.
|
||||||
|
|
||||||
|
### 12.5 Relationship to machines
|
||||||
|
|
||||||
|
- The SSH task execution role moves **out of machines** into `ssh_tasks` instances.
|
||||||
|
- Machines **keep** their role for the File Browser and node_exporter monitoring
|
||||||
|
transport in this change, to avoid also reworking Files/Monitoring here.
|
||||||
|
- Practical consequence: an SSH host used for both files and tasks may be defined twice
|
||||||
|
(once as a machine, once as an ssh_tasks instance) during the transition. Unifying
|
||||||
|
machines under services is an explicit **follow-up change**, not part of this one.
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
# Proposal: Runtime Service Registry
|
||||||
|
|
||||||
|
**Change:** `service-registry`
|
||||||
|
**Phase:** proposal
|
||||||
|
**Date:** 2026-06-19
|
||||||
|
**Status:** awaiting review (design only — no implementation yet)
|
||||||
|
|
||||||
|
## Context and problem
|
||||||
|
|
||||||
|
Phase 1 shipped a configurable dashboard widget system whose service URLs (Grafana,
|
||||||
|
Prometheus) and app credentials (Jellyfin, Jellyseerr) are driven by environment
|
||||||
|
variables and machine-level fields. This has three problems:
|
||||||
|
|
||||||
|
1. **Operators cannot change services without a redeploy.** Adding a second Grafana,
|
||||||
|
pointing Prometheus at a different host, or rotating a Jellyfin API key requires
|
||||||
|
editing env vars and restarting containers.
|
||||||
|
2. **Configuration is split across three places.** Service URLs live in env vars
|
||||||
|
(`GRAFANA_URL`, `PROMETHEUS_URL`), Jellyfin/Jellyseerr live on machine records, and
|
||||||
|
widget instances live in the widget table. There is no single "what is configured"
|
||||||
|
view.
|
||||||
|
3. **The widget registry is decoupled from the services it depends on.** A Grafana
|
||||||
|
widget does not know which Grafana instance it talks to; the widget config holds a
|
||||||
|
`dashboard_uid` while the base URL is global.
|
||||||
|
|
||||||
|
## Proposal
|
||||||
|
|
||||||
|
Introduce a **runtime service registry** persisted in the backend SQLite database:
|
||||||
|
|
||||||
|
- Each **service instance** (e.g. "Production Grafana", "Home Jellyfin") is a DB record
|
||||||
|
carrying its non-secret config and encrypted secret fields.
|
||||||
|
- **Service definitions** live as Python modules with Pydantic classes in the repo. Each
|
||||||
|
definition declares its config schema, its secret fields, and the **widget kinds** it
|
||||||
|
provides (with their own config schemas).
|
||||||
|
- **Service pages** at `/services/:serviceType/:serviceId` render the service-specific UI
|
||||||
|
and list the widgets that service can contribute to the dashboard. These replace the
|
||||||
|
existing addon pages.
|
||||||
|
- **Dashboard widgets** become service-bound: a widget instance references a `service_id`
|
||||||
|
and a `widget_kind` drawn from that service's definition.
|
||||||
|
- Machine records are reduced to **transport only** (SSH + node_exporter); the
|
||||||
|
machine-level Jellyfin/Jellyseerr app fields are removed.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- One source of truth for every external service the app talks to.
|
||||||
|
- Add/reconfigure/rotate a service from the UI with no redeploy.
|
||||||
|
- Multiple instances per service type (two Grafanas, two Jellyfins).
|
||||||
|
- Centralized, version-controlled service definitions that are easy to extend.
|
||||||
|
- Widgets discoverable per-service and individually addable to the dashboard.
|
||||||
|
- Secrets (API keys / tokens) encrypted at rest.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- **No general-purpose plugin/marketplace system.** Service definitions are closed,
|
||||||
|
compile-time code. Adding a brand-new service still requires a backend deploy and a
|
||||||
|
Python module.
|
||||||
|
- **No OAuth token exchange per service in this change.** Only API keys / tokens are
|
||||||
|
stored (encrypted). OAuth-proxy flows (e.g. Grafana behind Authentik) continue to be
|
||||||
|
handled externally.
|
||||||
|
- **No drag-and-drop dashboard layout, no grid, no per-user dashboards.** This change
|
||||||
|
keeps the existing single stacked-column dashboard model.
|
||||||
|
- **No in-app charting.** The thin-dashboard observability rule still holds; service
|
||||||
|
pages surface deep-links and metadata only.
|
||||||
|
- **No silent data migration.** Machine-level Jellyfin/Jellyseerr config is removed
|
||||||
|
without an automatic converter (see Decisions).
|
||||||
|
|
||||||
|
## Decisions (from grilling)
|
||||||
|
|
||||||
|
| Topic | Decision |
|
||||||
|
|-------|----------|
|
||||||
|
| Scope of services | All current services: Grafana, Prometheus, Jellyfin, Nextcloud, and the SSH task runner. Definitions centralized in repo. |
|
||||||
|
| Definition format | Python modules with Pydantic classes for service config and widget config, combined under each service definition. |
|
||||||
|
| Auth storage | API keys / tokens only, encrypted at rest. |
|
||||||
|
| Encryption key | Single env-provided master key (`MANAGE_ENCRYPTION_KEY`). |
|
||||||
|
| Machine app config | Services **replace** machine-level Jellyfin/Jellyseerr app config. Machines become SSH/monitoring transport only. |
|
||||||
|
| Migration | **Break backwards compatibility.** Users re-enter service config after upgrade; no automatic converter. |
|
||||||
|
| Multi-instance | Yes — multiple service records per service type. |
|
||||||
|
| Addon pages | Replaced by generic service pages at `/services/:serviceType/:serviceId`. |
|
||||||
|
| Widget binding | The service definition **owns** its widget config schemas. Widgets are instantiated from a service instance + a widget kind. |
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **Breaking upgrade.** Existing deployments lose their Jellyfin config and must re-enter
|
||||||
|
it. We must document this loudly in the changelog and README.
|
||||||
|
- **Encryption key management.** Losing `MANAGE_ENCRYPTION_KEY` makes all stored secrets
|
||||||
|
unrecoverable. Key rotation requires re-encrypting every service record.
|
||||||
|
- **Large surface area.** This change touches backend models, settings store, widget
|
||||||
|
registry, adapters, frontend routing, dashboard config UI, and docs. It must be split
|
||||||
|
into reviewable PRs (see `tasks.md`).
|
||||||
|
- **SSH task runner as a service** needs care: saved tasks already have their own
|
||||||
|
registry. The service record should hold connection/auth; the task registry stays.
|
||||||
|
- **Env vars are not fully eliminated.** The encryption key and core auth/OIDC settings
|
||||||
|
still require env vars; only service URLs/credentials move to the DB.
|
||||||
|
|
||||||
|
## Out of scope for this proposal
|
||||||
|
|
||||||
|
- Automatic migration tooling from machine app config to service records.
|
||||||
|
- Secret rotation UI or key-rotation workflow.
|
||||||
|
- Per-user or multi-dashboard support.
|
||||||
|
- Runtime/hot-reload of service definition files (definitions are loaded at startup).
|
||||||
@@ -0,0 +1,212 @@
|
|||||||
|
# Tasks: Runtime Service Registry
|
||||||
|
|
||||||
|
**Change:** `service-registry`
|
||||||
|
**Phase:** tasks
|
||||||
|
**Date:** 2026-06-19
|
||||||
|
|
||||||
|
## Review workload forecast
|
||||||
|
|
||||||
|
| Field | Value |
|
||||||
|
|-------|-------|
|
||||||
|
| Estimated changed lines | ~2,000–2,400 |
|
||||||
|
| 400-line budget risk | High |
|
||||||
|
| Chained PRs recommended | Yes (4 PRs) |
|
||||||
|
| Chain strategy | stacked-to-main |
|
||||||
|
|
||||||
|
```text
|
||||||
|
Decision needed before apply: Yes (see design §11 open questions)
|
||||||
|
Chained PRs recommended: Yes
|
||||||
|
Chain strategy: stacked-to-main
|
||||||
|
```
|
||||||
|
|
||||||
|
## Slice 1: Backend service foundation (no widget changes)
|
||||||
|
|
||||||
|
**Goal:** Persist service instances with encrypted secrets and expose CRUD + metadata.
|
||||||
|
|
||||||
|
- [ ] **1.1 Add encryption helper**
|
||||||
|
- Files: `backend/src/media_library_viewer_api/services/secrets.py` (new)
|
||||||
|
- Lines: ~60
|
||||||
|
- Details: Fernet-based `encrypt_secrets` / `decrypt_secrets` / `get_encryption_key`.
|
||||||
|
Raise on missing `MANAGE_ENCRYPTION_KEY`. Add `cryptography` dependency if missing.
|
||||||
|
- [ ] **1.2 Add integrations base classes**
|
||||||
|
- Files: `integrations/__init__.py`, `integrations/base.py` (new)
|
||||||
|
- Lines: ~80
|
||||||
|
- Details: `ServiceDefinition`, `WidgetKind`, `SecretField`, `ServiceConfigBase`.
|
||||||
|
- [ ] **1.3 Add five service definitions + registry**
|
||||||
|
- Files: `integrations/grafana.py`, `prometheus.py`, `jellyfin.py`, `nextcloud.py`,
|
||||||
|
`ssh_tasks.py`, `integrations/registry.py` (new)
|
||||||
|
- Lines: ~220
|
||||||
|
- Details: One `ServiceDefinition` per service with config schema, secret fields, and
|
||||||
|
widget kinds. `SERVICE_DEFINITIONS` + `get_service_definition` /
|
||||||
|
`get_widget_kind` helpers.
|
||||||
|
- [ ] **1.4 Add service store + `services` table**
|
||||||
|
- Files: `services/settings_store.py` (modify), `services/service_store.py` (new)
|
||||||
|
- Lines: ~120
|
||||||
|
- Details: `services` table in `init_schema`; CRUD helpers; decrypt-on-read for
|
||||||
|
adapters; "set" flags for the API without plaintext. **Cascade delete:** removing a
|
||||||
|
service deletes its widgets in the same transaction. Also add the
|
||||||
|
`service_task_runs` table (design §12.3) now so later slices can populate it.
|
||||||
|
- [ ] **1.5 Add service Pydantic models + router**
|
||||||
|
- Files: `models/services.py` (new), `routers/services.py` (new), `main.py` (modify)
|
||||||
|
- Lines: ~110
|
||||||
|
- Details: `GET /api/services/types`, `GET /api/services`, `POST/PUT/DELETE
|
||||||
|
/api/services/{id}`. Validate type, config, and secret schema against the definition.
|
||||||
|
- [ ] **1.6 Validate encryption key on startup**
|
||||||
|
- Files: `auth.py` or `main.py` lifespan (modify)
|
||||||
|
- Lines: ~10
|
||||||
|
- Details: Extend startup validation to require `MANAGE_ENCRYPTION_KEY`.
|
||||||
|
- [ ] **1.7 Add backend tests**
|
||||||
|
- Files: `backend/tests/test_services.py` (new)
|
||||||
|
- Lines: ~140
|
||||||
|
- Details: Registry contents, CRUD round-trip, secret encryption/decryption,
|
||||||
|
unknown service type → 422, missing/invalid encryption key → startup error,
|
||||||
|
cascade-delete removes a service's widgets.
|
||||||
|
- [ ] **1.8 Verify**
|
||||||
|
- Run: `cd backend && .venv/bin/ruff check . && PYTHONPATH=src .venv/bin/python -m pytest`
|
||||||
|
|
||||||
|
**Slice 1 total:** ~720 changed lines (smallest coherent backend foundation).
|
||||||
|
|
||||||
|
## Slice 2: Backend widget rebind to services
|
||||||
|
|
||||||
|
**Goal:** Widgets reference a service instance + widget kind; adapters resolve services.
|
||||||
|
|
||||||
|
- [ ] **2.1 Add widget columns + migrate table**
|
||||||
|
- Files: `services/settings_store.py` (modify)
|
||||||
|
- Lines: ~40
|
||||||
|
- Details: Add `service_id`, `widget_kind` to `dashboard_widgets`; keep `widget_type`
|
||||||
|
as `{service_type}.{kind}` during transition; drop `addon_id`.
|
||||||
|
- [ ] **2.2 Refactor source adapters**
|
||||||
|
- Files: `widgets/sources.py` (modify)
|
||||||
|
- Lines: ~160
|
||||||
|
- Details: Each adapter takes `(service: ServiceRecord, widget_kind, config)`.
|
||||||
|
`SOURCE_ADAPTERS` keyed by `service_type`. Jellyfin/Grafana/Prometheus/SSH adapters
|
||||||
|
resolve connection from the service record. The SSH adapter resolves the task +
|
||||||
|
instance, runs it, and **appends a `service_task_runs` row** (design §12.3).
|
||||||
|
- [ ] **2.3 Retire old widget registry**
|
||||||
|
- Files: `widgets/registry.py` (delete or hollow out), `widgets/__init__.py`
|
||||||
|
- Lines: ~-60
|
||||||
|
- Details: Widget metadata now comes from `integrations/registry.py`.
|
||||||
|
- [ ] **2.4 Update widgets router + models**
|
||||||
|
- Files: `routers/widgets.py`, `models/widgets.py` (modify)
|
||||||
|
- Lines: ~90
|
||||||
|
- Details: Validation uses the service definition's widget schema; data endpoint
|
||||||
|
loads service, builds `ServiceRecord`, calls adapter.
|
||||||
|
- [ ] **2.5 Update widget tests**
|
||||||
|
- Files: `backend/tests/test_widgets.py` (modify)
|
||||||
|
- Lines: ~120
|
||||||
|
- Details: Rewrite adapter/data tests around service instances; cover
|
||||||
|
service-missing, wrong-kind, and encrypted-secret resolution.
|
||||||
|
- [ ] **2.6 Verify**
|
||||||
|
- Run: `cd backend && .venv/bin/ruff check . && PYTHONPATH=src .venv/bin/python -m pytest`
|
||||||
|
|
||||||
|
**Slice 2 total:** ~330 changed lines.
|
||||||
|
|
||||||
|
## Slice 3: Frontend services runtime
|
||||||
|
|
||||||
|
**Goal:** Service types/API/hooks, frontend service registry, service pages, route swap.
|
||||||
|
|
||||||
|
- [ ] **3.1 Add service types**
|
||||||
|
- Files: `frontend/src/types/index.ts` (modify)
|
||||||
|
- Lines: ~50
|
||||||
|
- Details: `ServiceInstance`, `ServiceInstanceInput`, `ServiceTypeInfo`,
|
||||||
|
`ServiceWidgetKind`. Widget gains `service_id`, `widget_kind`.
|
||||||
|
- [ ] **3.2 Add services API + hooks**
|
||||||
|
- Files: `frontend/src/api/services.ts`, `frontend/src/hooks/useServices.ts` (new)
|
||||||
|
- Lines: ~110
|
||||||
|
- Details: Fetch/create/update/delete service instances and types.
|
||||||
|
- [ ] **3.3 Add frontend service registry**
|
||||||
|
- Files: `frontend/src/integrations/registry.ts` (new)
|
||||||
|
- Lines: ~120
|
||||||
|
- Details: Closed registry mirroring backend: config fields, secret fields
|
||||||
|
(`secret: true`), widget kinds, service page components.
|
||||||
|
- [ ] **3.4 Add service page + components**
|
||||||
|
- Files: `frontend/src/pages/ServicePage.tsx`, `frontend/src/integrations/components/*`
|
||||||
|
(new)
|
||||||
|
- Lines: ~180
|
||||||
|
- Details: Generic page dispatches by service type; renders config editor + widget
|
||||||
|
kinds. Add per-service components (Grafana, Prometheus, Jellyfin, Nextcloud,
|
||||||
|
SSH tasks).
|
||||||
|
- [ ] **3.5 Swap routes; remove addon pages**
|
||||||
|
- Files: `frontend/src/App.tsx`, `frontend/src/pages/AddonPage.tsx`,
|
||||||
|
`frontend/src/addons/*` (modify/delete)
|
||||||
|
- Lines: ~-40 net
|
||||||
|
- Details: `/services/:serviceType/:serviceId`; redirect old `/addons/*` to the
|
||||||
|
default service of that type.
|
||||||
|
- [ ] **3.6 Add frontend registry test**
|
||||||
|
- Files: `frontend/src/integrations/registry.test.ts` (new)
|
||||||
|
- Lines: ~40
|
||||||
|
- Details: Assert all five service types and their widget kinds.
|
||||||
|
- [ ] **3.7 Verify**
|
||||||
|
- Run: `cd frontend && npm run lint && npm run build && npm run test -- src/integrations/registry.test.ts`
|
||||||
|
|
||||||
|
**Slice 3 total:** ~460 changed lines.
|
||||||
|
|
||||||
|
## Slice 4: Dashboard picker, settings rework, cleanup, docs
|
||||||
|
|
||||||
|
**Goal:** End-to-end service-based dashboard; remove legacy machine app config + env vars.
|
||||||
|
|
||||||
|
- [ ] **4.1 Rework widget config dialog**
|
||||||
|
- Files: `frontend/src/components/WidgetConfigDialog.tsx` (modify)
|
||||||
|
- Lines: ~120
|
||||||
|
- Details: "Add widget" = pick service → pick widget kind → configure. Widget cards
|
||||||
|
show parent service name.
|
||||||
|
- [ ] **4.2 Update widget components to service model**
|
||||||
|
- Files: `frontend/src/widgets/*` (modify)
|
||||||
|
- Lines: ~120
|
||||||
|
- Details: Components read `widget_kind`; data shapes unchanged but sourced from the
|
||||||
|
service adapter. SSH task widget shows last run status from `service_task_runs`.
|
||||||
|
- [ ] **4.3 Remove machine Jellyfin/Jellyseerr fields**
|
||||||
|
- Files: `frontend/src/pages/Settings.tsx`, `frontend/src/types/index.ts`
|
||||||
|
(modify)
|
||||||
|
- Lines: ~-60
|
||||||
|
- Details: Machines are SSH/monitoring transport only.
|
||||||
|
- [ ] **4.4 Remove grafana_url / prometheus_url from backend config**
|
||||||
|
- Files: `backend/src/media_library_viewer_api/config.py`,
|
||||||
|
`docker-compose.yml`, `docker-compose.dev.yml`, `.env.example`
|
||||||
|
- Lines: ~-10
|
||||||
|
- Details: URLs now live on service records. Add `MANAGE_ENCRYPTION_KEY` to compose
|
||||||
|
- `.env.example`.
|
||||||
|
- [ ] **4.5 Stop default widget seeding**
|
||||||
|
- Files: `services/settings_store.py` (modify)
|
||||||
|
- Lines: ~-20
|
||||||
|
- Details: Fresh installs start with no widgets; user adds them after configuring
|
||||||
|
services.
|
||||||
|
- [ ] **4.6 Docs + changelog**
|
||||||
|
- Files: `docs/REQUIREMENTS.md`, `README.md`, `docs/CHANGELOG.md` (new or modify)
|
||||||
|
- Lines: ~80
|
||||||
|
- Details: Service registry section; `MANAGE_ENCRYPTION_KEY` requirement; breaking
|
||||||
|
upgrade note (re-enter Jellyfin config).
|
||||||
|
- [ ] **4.7 Verify full stack**
|
||||||
|
- Run: backend `ruff` + `pytest`; frontend `lint` + `build` + `test`.
|
||||||
|
|
||||||
|
**Slice 4 total:** ~330 changed lines.
|
||||||
|
|
||||||
|
## Integration and acceptance
|
||||||
|
|
||||||
|
- [ ] **5.1 Backend full test run** — `PYTHONPATH=src pytest`, all green.
|
||||||
|
- [ ] **5.2 Frontend full build/lint/test** — `npm run lint && npm run build && npm run test`.
|
||||||
|
- [ ] **5.3 Manual dev-stack check** — `docker compose -f docker-compose.dev.yml up --build`:
|
||||||
|
- Create a Grafana service from the UI; verify the dashboard link widget works.
|
||||||
|
- Create a Jellyfin service; verify the activity widget resolves it.
|
||||||
|
- Delete a service with widgets → widgets are cascade-deleted and the service is gone.
|
||||||
|
- Restart the stack; secrets remain usable (key stable).
|
||||||
|
- Missing `MANAGE_ENCRYPTION_KEY` → backend refuses to start.
|
||||||
|
- SSH task runner: define two instances, run the same reusable task against each,
|
||||||
|
and see both runs in the instance's history log.
|
||||||
|
|
||||||
|
## Guards
|
||||||
|
|
||||||
|
```text
|
||||||
|
Decision needed before apply: No (design §11 resolved)
|
||||||
|
Chained PRs recommended: Yes
|
||||||
|
Chain strategy: stacked-to-main
|
||||||
|
400-line budget risk: High
|
||||||
|
```
|
||||||
|
|
||||||
|
## Explicit follow-ups (out of scope for this change)
|
||||||
|
|
||||||
|
- Rebuild the Actions page UI on top of services (global reusable tasks +
|
||||||
|
`default_service_id`), replacing the current machine-based saved-task runner.
|
||||||
|
- Unify machines under services so an SSH host is defined once (today machines still
|
||||||
|
own File Browser + node_exporter transport; see design §12.5).
|
||||||
|
- Key rotation / re-encrypt workflow for `MANAGE_ENCRYPTION_KEY`.
|
||||||
Reference in New Issue
Block a user