From bc389ae3f711c0370afb2063dec6cef84d2106d4 Mon Sep 17 00:00:00 2001 From: Developer Date: Thu, 9 Jul 2026 07:45:55 +0000 Subject: [PATCH] spec(service-storage-harness): reconcile proposal post Change A + question round Resolve Q1 (reuse Change A's PrometheusChartWidget renderer via InService data path), Q2 (totals = item count, not transfer bytes), Q3 (active = downloading/uploading), Q4 (N instances), Q5 (username/password cookie auth). Remove TBD/stale thin-dashboard caveats. --- .../service-storage-harness/proposal.md | 148 ++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 openspec/changes/service-storage-harness/proposal.md diff --git a/openspec/changes/service-storage-harness/proposal.md b/openspec/changes/service-storage-harness/proposal.md new file mode 100644 index 0000000..85631a1 --- /dev/null +++ b/openspec/changes/service-storage-harness/proposal.md @@ -0,0 +1,148 @@ +# SDD Proposal: Service Storage Harness (with qBittorrent widgets + MediaIndex migration) + +**Change:** `service-storage-harness` +**Phase:** proposal +**Date:** 2026-07-08 + +## 1. Problem / Why Now + +Two things are happening at once, and this change addresses both: + +1. **Feature request — qBittorrent widgets.** The operator wants at-a-glance qBittorrent visibility on the Manage dashboard: active downloads/uploads, total bytes transferred, and a download/upload speed indicator. qBittorrent is currently a blind spot — it is not modeled in the service registry and exposes no widgets. + +2. **Platform gap — services have no owned persistence.** Manage already has a service registry (machines, Jellyfin, Grafana, Prometheus, Alertmanager, Jellyseerr, ssh_tasks) and a configurable widget system. But when a service needs to *remember operational data over time*, there is no shared answer. `MediaIndex` is a one-off: it owns its own `media_index.db`, runs its own schema, and is built by a dedicated subprocess worker. It is a "special snowflake." Rather than add a *second* snowflake for qBittorrent speed history, we extract the shared lifecycle into a small **`ServiceDataHarness`** and prove it with qBittorrent, then migrate `MediaIndex` onto it so the pattern has two evidence-based consumers. + +The two are bundled because the second service (qBittorrent) is the one that justifies generalizing from the first (`MediaIndex`) — n=2 is what makes the abstraction worth its cost. + +## 2. Target Users and Situations + +- **Primary users:** Homelab operators running qBittorrent alongside Manage, who want download/upload activity visible without opening the qBittorrent UI. +- **Workflow moments:** + - Glance at the dashboard: "is a download running, how fast, how much has been transferred?" + - Decide whether qBittorrent is healthy without leaving Manage. + - Browse the Jellyfin media catalog (unchanged UX) — which now runs on the same storage harness, validating the abstraction. +- **Urgency:** Medium. The feature is valuable but not breaking; the platform refactor is opportunistic (do it now while only two consumers exist, before a third snowflake appears). + +## 3. Product Outcome + +After this change, an authenticated user can: + +- Register one or more **qBittorrent service instances** in the existing Services UI (URL + username + password, stored as encrypted secrets — same posture as Grafana/Prometheus). +- Place three discrete **qBittorrent widget kinds** on the dashboard: + 1. **Totals tile** — count of currently-listed torrents (total items in the qBittorrent list, which may exceed the active count because rate/connection limits leave some torrents non-transferring). NOT cumulative bytes transferred. + 2. **Active torrents list** — torrents whose state is `downloading` or `uploading`. + 3. **Speed chart** — live download/upload speed over a short rolling window, rendered with the **`PrometheusChartWidget` recharts renderer established by the `prometheus-direct-charting` change** (fed from the `QbittorrentSampleStore` via an InService-style data path returning the same `{series}` shape). No hand-rolled SVG sparkline; the thin-dashboard rule was already repealed for recharts charting by the foundational change. +- Continue using the Jellyfin **Media page** exactly as before; its underlying storage moves onto the harness transparently and additionally becomes **multi-instance capable** (scoped per Jellyfin service). + +## 4. Scope Boundaries and Non-Goals + +### In scope + +- **`ServiceDataHarness`** — a general lifecycle layer: per-concern DB files, per-integration schema migrations, `service_id` scoping of all tables, and cascade-delete when a service instance is removed. +- **qBittorrent integration** — new `integrations/qbittorrent.py` (config + secret schema + widget kinds), new `clients/qbittorrent.py` (Web API client, cookie login), a `QbittorrentSampleStore` (speed samples), widget source adapter(s) in `widgets/sources.py`, registry entry. +- **qBittorrent frontend** — three widget components under `frontend/src/widgets/`, binding in `integrations/registry.ts`, types, API client functions. +- **MediaIndex migration** — `MediaIndex` registered with the harness, `media_items` scoped by `service_id`, subprocess worker updated. The `media_index.db` file location is unchanged (only gains a column + harness registration) to minimize churn on a load-bearing feature. +- **Cascade-delete wiring** — removing a service instance cleans up its owned data in every harness-managed table. + +### Non-goals (explicitly out of scope) + +- **Torrent management UI** — no add/pause/delete/recheck/priority UI. Read-only visibility only. +- **A generic time-series database.** The harness owns *lifecycle*, not a generic `(service_id, key, ts, value)` table. Each integration owns its own schema and operations. +- **A generic CRUD/ORM layer.** Stores keep bespoke operations (`append/window` vs `replace_all/query`); only the lifecycle is shared. +- **Per-user or per-tenant storage partitioning.** Storage is scoped by `service_id` only. +- **Re-indexing/migrating existing media data.** The `media_index.db` file stays in place; the migration is a schema column add + harness registration, not a data move. +- **WebSocket / push updates.** Polling via existing `useWidgetData(widgetId, refreshIntervalMs)` is sufficient. +- **Transfer-byte totals.** The totals widget counts torrent *items*, not cumulative bytes uploaded/downloaded (per the §8 Q2 resolution). Byte totals are out of scope. +- **qBittorrent Prometheus exporter.** Not built; the speed chart is sourced from the local `QbittorrentSampleStore`, not Prometheus. + +## 5. High-Level Approach + +### 5.1 Architectural decisions (locked during grilling) + +| Decision | Outcome | +|---|---| +| **Storage tech** | SQLite. Low volume; proven in stack. | +| **Abstraction level** | `ServiceDataHarness` owns the *lifecycle* (DB filename, migrations, `service_id` scoping, cascade-delete). Each integration owns its *operations* in a bespoke Store. General where shared; bespoke where not. | +| **MediaIndex migration** | Included in this change, but **sequenced**: harness + qBit proven first; MediaIndex folded in after. Inside one change, never two simultaneous risks. | +| **DB topology** | Per-concern DB files. `media_index.db` keeps its file (gains `service_id` + registration); new `qbittorrent.db`. | + +### 5.2 Backend + +1. **`ServiceDataHarness`** (new, `services/service_data.py` or similar): + - A registry of *concerns*: each integration declares a DB filename, an ordered list of migration SQL statements, the tables it owns, and a `service_id` column convention. + - On startup: runs pending migrations per concern DB. + - On service deletion: cascades — for each concern table, `DELETE FROM WHERE service_id = ?`. + - Provides connection management per concern DB (separate connections → bulk media writes don't share a writer with chatty qBit appends). + +2. **qBittorrent client** (`clients/qbittorrent.py`): + - Login via `/api/v2/auth/login` → cookie session; reuse cookie across calls; re-login on 403. + - Endpoints used: `/api/v2/transfer/info` (global totals), `/api/v2/sync/maindata` (active torrents + current speeds), `/api/v2/torrents/info` (filtered lists if needed). + - Timeout + reachability handling consistent with `JellyfinClient`/`GrafanaWidgetSource`. + +3. **`QbittorrentSampleStore`** (new): + - Table `qbittorrent_speed_samples(service_id TEXT, ts INTEGER, dl_speed INTEGER, up_speed INTEGER)` in `qbittorrent.db`. + - Operations: `append(service_id, ts, dl, up)`, `window(service_id, since_ts)`, `prune(service_id, older_than_ts)`. + - Pruning runs on each append (cap retention to the configured window, e.g. 120 samples). + +4. **Widget source adapter** — `QbittorrentWidgetSource` in `widgets/sources.py` implementing the existing `WidgetSource.fetch(service, widget_kind, config)` contract. For the speed kind, `fetch` appends a sample to the store and returns the current window for rendering. + +5. **Integration registration** — add `qbittorrent` to `integrations/registry.py` with config schema (base URL, timeout) and secret schema (username, password); declare widget kinds. + +6. **MediaIndex migration** — register `MediaIndex` as a concern; add `service_id` column (backfill existing rows with the default/local Jellyfin service id); update `replace_items`/`query`/worker to scope by `service_id`; route through the harness connection. + +### 5.3 Frontend + +1. **Three widget components** under `frontend/src/widgets/`: + - `QbittorrentTotalsWidget.tsx` — numeric tiles for all-time bytes (reuses `MetricCard`/`SectionCard`). + - `QbittorrentActiveTorrentsWidget.tsx` — list/table of active torrents (reuses the already-migrated `DataTable` + `@tanstack/react-table`; **no new DataGrid migration risk**). + - `QbittorrentSpeedWidget.tsx` — visual form decided by §8 Q1 (numeric tiles + delta, or sparkline if an exception is granted, or a PrometheusMetricWidget binding if the exporter path is chosen). +2. **Registry binding** — add a `qbittorrent` entry to `SERVICE_REGISTRY` in `frontend/src/integrations/registry.ts` mapping the three widget kinds to components + config schemas. +3. **Types + API client** — `frontend/src/types/index.ts` and `frontend/src/api/client.ts` gain qBittorrent-aware widget kinds only (data flows through the existing `useWidgetData` polling; no new endpoints beyond widget CRUD). + +### 5.4 Type contracts + +- Backend Pydantic models for qBittorrent config/secret schema in `integrations/qbittorrent.py`. +- Frontend TypeScript interfaces for qBittorrent widget payloads. +- Harness has no data-shape types of its own (it is lifecycle-only) — keeping the seam clean. + +## 6. Success Criteria / Acceptance Criteria + +1. A user can register a qBittorrent service instance (URL + username + password) and the three widget kinds appear as bindable on the dashboard. +2. Totals tile shows all-time bytes from `transferInfo`; active list shows downloading/uploading torrents; speed indicator reflects current rates. +3. `ServiceDataHarness` runs migrations on startup and cascades deletes across qBittorrent samples *and* media items when a service is removed. +4. `MediaIndex` continues to power the Media page identically (all existing Media tests green) and is now scoped by `service_id`. +5. Removing a Jellyfin service removes only that service's media rows; removing a qBittorrent service removes only that service's speed samples. +6. A misconfigured/unreachable qBittorrent instance degrades gracefully per-widget (error state), the rest of the dashboard renders. +7. Backend tests (`pytest`) and frontend `npm run build` + `npm run lint` stay green. +8. No secrets land in widget `config_json`; qBittorrent password is encrypted via the existing Fernet path. +9. No new charting dependency is added unless §8 Q1 grants an explicit exception. + +## 7. Risks and Mitigations + +| Risk | Mitigation | +|------|------------| +| **Compound risk: new abstraction × load-bearing refactor.** A wrong harness shape breaks the Media page. | Sequence inside the change: harness + qBit land and pass tests first; MediaIndex migrates only after the harness shape is settled in code. | +| **Harness over-generalization.** Building a meta-framework from n=2. | Keep the harness lifecycle-only; do NOT add a generic value table or generic CRUD. Each store keeps bespoke operations. Extract further only when a third shape appears. | +| **Thin-dashboard rule collision (§8 Q1).** In-app speed graph violates the stated "no in-app charting" rule. | Resolve in the question round before spec; default to the rule-compliant numeric-tile + Grafana deep-link option unless an explicit exception is granted. | +| **qBittorrent auth lifecycle.** Cookie expiry / 403 handling. | Re-login transparently on 403; short request timeout; surface persistent auth failure as widget error state. | +| **MediaIndex backfill correctness.** Adding `service_id` to existing rows. | Backfill all existing rows to the default/local Jellyfin service id; migration is additive; existing Media tests must pass unchanged. | +| **Review budget (>400 changed lines).** Bundled scope is large. | Slice into chained PRs (Slice 1: harness + qBit; Slice 2: MediaIndex migration). Each slice leaves `npm run build` + `npm run lint` + `pytest` green. | +| **DataGrid migration (config rule callout).** | Not a risk here: the active-torrents list reuses the already-migrated `DataTable` (`@tanstack/react-table`). No new DataGrid migration is introduced. The key technical risks are the harness abstraction and the MediaIndex refactor, above. | + +## 8. Resolved Questions (question round complete) + +All five product/semantic questions resolved during grilling + the Change A (`prometheus-direct-charting`) lifecycle: + +- **Q1 — Speed visualization.** RESOLVED: the thin-dashboard rule was repealed by Change A; in-app recharts charting is now sanctioned. The qBit speed widget reuses the `PrometheusChartWidget` recharts renderer, fed from `QbittorrentSampleStore.window()` via an InService data path returning the same `{series}` shape. No SVG sparkline, no Grafana dependency. +- **Q2 — "Totals" semantics.** RESOLVED: count of currently-listed torrent items (total in the qBittorrent list), which may exceed the active count due to rate/connection limits. NOT transfer bytes, NOT per-session counters. +- **Q3 — "Active" definition.** RESOLVED: torrents in state `downloading` or `uploading` only. +- **Q4 — Instances.** RESOLVED: support N qBittorrent instances, each a service row, independently scoped by `service_id`. +- **Q5 — Auth model.** RESOLVED: username/password login → cookie session, encrypted via Fernet. No reverse-proxy no-auth flag in this pass. + +## 9. Future Phases + +1. **Third service consumer** — when a service with a genuinely new storage shape arrives, reconsider promoting the harness toward a broader abstraction (evidence-based, n=3). +2. **Torrent management** — add/pause/delete actions (would require write endpoints + confirmation UX). +3. **Storage admin UI** — surface harness-managed table sizes and a "clear cached data" action per service. +4. **Reverse-proxy / no-auth qBittorrent flag** — for setups behind Authentik where native login is bypassed. +5. **Transfer-byte totals** — if desired later, add a separate widget using `transferInfo.globalUploaded/Downloaded`.