manage/openspec/changes/service-storage-harness/proposal.md

# SDD Proposal: Service Storage Harness (with qBittorrent widgets + MediaIndex migration)

**Change:** `service-storage-harness`
**Phase:** proposal
**Date:** 2026-07-08

## 1. Problem / Why Now

Two things are happening at once, and this change addresses both:

1. **Feature request — qBittorrent widgets.** The operator wants at-a-glance qBittorrent visibility on the Manage dashboard: active downloads/uploads, total bytes transferred, and a download/upload speed indicator. qBittorrent is currently a blind spot — it is not modeled in the service registry and exposes no widgets.

2. **Platform gap — services have no owned persistence.** Manage already has a service registry (machines, Jellyfin, Grafana, Prometheus, Alertmanager, Jellyseerr, ssh_tasks) and a configurable widget system. But when a service needs to *remember operational data over time*, there is no shared answer. `MediaIndex` is a one-off: it owns its own `media_index.db`, runs its own schema, and is built by a dedicated subprocess worker. It is a "special snowflake." Rather than add a *second* snowflake for qBittorrent speed history, we extract the shared lifecycle into a small **`ServiceDataHarness`** and prove it with qBittorrent, then migrate `MediaIndex` onto it so the pattern has two evidence-based consumers.

The two are bundled because the second service (qBittorrent) is the one that justifies generalizing from the first (`MediaIndex`) — n=2 is what makes the abstraction worth its cost.

## 2. Target Users and Situations

- **Primary users:** Homelab operators running qBittorrent alongside Manage, who want download/upload activity visible without opening the qBittorrent UI.
- **Workflow moments:**
  - Glance at the dashboard: "is a download running, how fast, how much has been transferred?"
  - Decide whether qBittorrent is healthy without leaving Manage.
  - Browse the Jellyfin media catalog (unchanged UX) — which now runs on the same storage harness, validating the abstraction.
- **Urgency:** Medium. The feature is valuable but not breaking; the platform refactor is opportunistic (do it now while only two consumers exist, before a third snowflake appears).

## 3. Product Outcome

After this change, an authenticated user can:

- Register one or more **qBittorrent service instances** in the existing Services UI (URL + username + password, stored as encrypted secrets — same posture as Grafana/Prometheus).
- Place three discrete **qBittorrent widget kinds** on the dashboard:
  1. **Totals tile** — count of currently-listed torrents (total items in the qBittorrent list, which may exceed the active count because rate/connection limits leave some torrents non-transferring). NOT cumulative bytes transferred.
  2. **Active torrents list** — torrents whose state is `downloading` or `uploading`.
  3. **Speed chart** — live download/upload speed over a short rolling window, rendered with the **`PrometheusChartWidget` recharts renderer established by the `prometheus-direct-charting` change** (fed from the `QbittorrentSampleStore` via an InService-style data path returning the same `{series}` shape). No hand-rolled SVG sparkline; the thin-dashboard rule was already repealed for recharts charting by the foundational change.
- Continue using the Jellyfin **Media page** exactly as before; its underlying storage moves onto the harness transparently and additionally becomes **multi-instance capable** (scoped per Jellyfin service).

## 4. Scope Boundaries and Non-Goals

### In scope

- **`ServiceDataHarness`** — a general lifecycle layer: per-concern DB files, per-integration schema migrations, `service_id` scoping of all tables, and cascade-delete when a service instance is removed.
- **qBittorrent integration** — new `integrations/qbittorrent.py` (config + secret schema + widget kinds), new `clients/qbittorrent.py` (Web API client, cookie login), a `QbittorrentSampleStore` (speed samples), widget source adapter(s) in `widgets/sources.py`, registry entry.
- **qBittorrent frontend** — three widget components under `frontend/src/widgets/`, binding in `integrations/registry.ts`, types, API client functions.
- **MediaIndex migration** — `MediaIndex` registered with the harness, `media_items` scoped by `service_id`, subprocess worker updated. The `media_index.db` file location is unchanged (only gains a column + harness registration) to minimize churn on a load-bearing feature.
- **Cascade-delete wiring** — removing a service instance cleans up its owned data in every harness-managed table.

### Non-goals (explicitly out of scope)

- **Torrent management UI** — no add/pause/delete/recheck/priority UI. Read-only visibility only.
- **A generic time-series database.** The harness owns *lifecycle*, not a generic `(service_id, key, ts, value)` table. Each integration owns its own schema and operations.
- **A generic CRUD/ORM layer.** Stores keep bespoke operations (`append/window` vs `replace_all/query`); only the lifecycle is shared.
- **Per-user or per-tenant storage partitioning.** Storage is scoped by `service_id` only.
- **Re-indexing/migrating existing media data.** The `media_index.db` file stays in place; the migration is a schema column add + harness registration, not a data move.
- **WebSocket / push updates.** Polling via existing `useWidgetData(widgetId, refreshIntervalMs)` is sufficient.
- **Transfer-byte totals.** The totals widget counts torrent *items*, not cumulative bytes uploaded/downloaded (per the §8 Q2 resolution). Byte totals are out of scope.
- **qBittorrent Prometheus exporter.** Not built; the speed chart is sourced from the local `QbittorrentSampleStore`, not Prometheus.

## 5. High-Level Approach

### 5.1 Architectural decisions (locked during grilling)

| Decision | Outcome |
|---|---|
| **Storage tech** | SQLite. Low volume; proven in stack. |
| **Abstraction level** | `ServiceDataHarness` owns the *lifecycle* (DB filename, migrations, `service_id` scoping, cascade-delete). Each integration owns its *operations* in a bespoke Store. General where shared; bespoke where not. |
| **MediaIndex migration** | Included in this change, but **sequenced**: harness + qBit proven first; MediaIndex folded in after. Inside one change, never two simultaneous risks. |
| **DB topology** | Per-concern DB files. `media_index.db` keeps its file (gains `service_id` + registration); new `qbittorrent.db`. |

### 5.2 Backend

1. **`ServiceDataHarness`** (new, `services/service_data.py` or similar):
   - A registry of *concerns*: each integration declares a DB filename, an ordered list of migration SQL statements, the tables it owns, and a `service_id` column convention.
   - On startup: runs pending migrations per concern DB.
   - On service deletion: cascades — for each concern table, `DELETE FROM <table> WHERE service_id = ?`.
   - Provides connection management per concern DB (separate connections → bulk media writes don't share a writer with chatty qBit appends).

2. **qBittorrent client** (`clients/qbittorrent.py`):
   - Login via `/api/v2/auth/login` → cookie session; reuse cookie across calls; re-login on 403.
   - Endpoints used: `/api/v2/transfer/info` (global totals), `/api/v2/sync/maindata` (active torrents + current speeds), `/api/v2/torrents/info` (filtered lists if needed).
   - Timeout + reachability handling consistent with `JellyfinClient`/`GrafanaWidgetSource`.

3. **`QbittorrentSampleStore`** (new):
   - Table `qbittorrent_speed_samples(service_id TEXT, ts INTEGER, dl_speed INTEGER, up_speed INTEGER)` in `qbittorrent.db`.
   - Operations: `append(service_id, ts, dl, up)`, `window(service_id, since_ts)`, `prune(service_id, older_than_ts)`.
   - Pruning runs on each append (cap retention to the configured window, e.g. 120 samples).

4. **Widget source adapter** — `QbittorrentWidgetSource` in `widgets/sources.py` implementing the existing `WidgetSource.fetch(service, widget_kind, config)` contract. For the speed kind, `fetch` appends a sample to the store and returns the current window for rendering.

5. **Integration registration** — add `qbittorrent` to `integrations/registry.py` with config schema (base URL, timeout) and secret schema (username, password); declare widget kinds.

6. **MediaIndex migration** — register `MediaIndex` as a concern; add `service_id` column (backfill existing rows with the default/local Jellyfin service id); update `replace_items`/`query`/worker to scope by `service_id`; route through the harness connection.

### 5.3 Frontend

1. **Three widget components** under `frontend/src/widgets/`:
   - `QbittorrentTotalsWidget.tsx` — numeric tiles for all-time bytes (reuses `MetricCard`/`SectionCard`).
   - `QbittorrentActiveTorrentsWidget.tsx` — list/table of active torrents (reuses the already-migrated `DataTable` + `@tanstack/react-table`; **no new DataGrid migration risk**).
   - `QbittorrentSpeedWidget.tsx` — visual form decided by §8 Q1 (numeric tiles + delta, or sparkline if an exception is granted, or a PrometheusMetricWidget binding if the exporter path is chosen).
2. **Registry binding** — add a `qbittorrent` entry to `SERVICE_REGISTRY` in `frontend/src/integrations/registry.ts` mapping the three widget kinds to components + config schemas.
3. **Types + API client** — `frontend/src/types/index.ts` and `frontend/src/api/client.ts` gain qBittorrent-aware widget kinds only (data flows through the existing `useWidgetData` polling; no new endpoints beyond widget CRUD).

### 5.4 Type contracts

- Backend Pydantic models for qBittorrent config/secret schema in `integrations/qbittorrent.py`.
- Frontend TypeScript interfaces for qBittorrent widget payloads.
- Harness has no data-shape types of its own (it is lifecycle-only) — keeping the seam clean.

## 6. Success Criteria / Acceptance Criteria

1. A user can register a qBittorrent service instance (URL + username + password) and the three widget kinds appear as bindable on the dashboard.
2. Totals tile shows all-time bytes from `transferInfo`; active list shows downloading/uploading torrents; speed indicator reflects current rates.
3. `ServiceDataHarness` runs migrations on startup and cascades deletes across qBittorrent samples *and* media items when a service is removed.
4. `MediaIndex` continues to power the Media page identically (all existing Media tests green) and is now scoped by `service_id`.
5. Removing a Jellyfin service removes only that service's media rows; removing a qBittorrent service removes only that service's speed samples.
6. A misconfigured/unreachable qBittorrent instance degrades gracefully per-widget (error state), the rest of the dashboard renders.
7. Backend tests (`pytest`) and frontend `npm run build` + `npm run lint` stay green.
8. No secrets land in widget `config_json`; qBittorrent password is encrypted via the existing Fernet path.
9. No new charting dependency is added unless §8 Q1 grants an explicit exception.

## 7. Risks and Mitigations

| Risk | Mitigation |
|------|------------|
| **Compound risk: new abstraction × load-bearing refactor.** A wrong harness shape breaks the Media page. | Sequence inside the change: harness + qBit land and pass tests first; MediaIndex migrates only after the harness shape is settled in code. |
| **Harness over-generalization.** Building a meta-framework from n=2. | Keep the harness lifecycle-only; do NOT add a generic value table or generic CRUD. Each store keeps bespoke operations. Extract further only when a third shape appears. |
| **Thin-dashboard rule collision (§8 Q1).** In-app speed graph violates the stated "no in-app charting" rule. | Resolve in the question round before spec; default to the rule-compliant numeric-tile + Grafana deep-link option unless an explicit exception is granted. |
| **qBittorrent auth lifecycle.** Cookie expiry / 403 handling. | Re-login transparently on 403; short request timeout; surface persistent auth failure as widget error state. |
| **MediaIndex backfill correctness.** Adding `service_id` to existing rows. | Backfill all existing rows to the default/local Jellyfin service id; migration is additive; existing Media tests must pass unchanged. |
| **Review budget (>400 changed lines).** Bundled scope is large. | Slice into chained PRs (Slice 1: harness + qBit; Slice 2: MediaIndex migration). Each slice leaves `npm run build` + `npm run lint` + `pytest` green. |
| **DataGrid migration (config rule callout).** | Not a risk here: the active-torrents list reuses the already-migrated `DataTable` (`@tanstack/react-table`). No new DataGrid migration is introduced. The key technical risks are the harness abstraction and the MediaIndex refactor, above. |

## 8. Resolved Questions (question round complete)

All five product/semantic questions resolved during grilling + the Change A (`prometheus-direct-charting`) lifecycle:

- **Q1 — Speed visualization.** RESOLVED: the thin-dashboard rule was repealed by Change A; in-app recharts charting is now sanctioned. The qBit speed widget reuses the `PrometheusChartWidget` recharts renderer, fed from `QbittorrentSampleStore.window()` via an InService data path returning the same `{series}` shape. No SVG sparkline, no Grafana dependency.
- **Q2 — "Totals" semantics.** RESOLVED: count of currently-listed torrent items (total in the qBittorrent list), which may exceed the active count due to rate/connection limits. NOT transfer bytes, NOT per-session counters.
- **Q3 — "Active" definition.** RESOLVED: torrents in state `downloading` or `uploading` only.
- **Q4 — Instances.** RESOLVED: support N qBittorrent instances, each a service row, independently scoped by `service_id`.
- **Q5 — Auth model.** RESOLVED: username/password login → cookie session, encrypted via Fernet. No reverse-proxy no-auth flag in this pass.

## 9. Future Phases

1. **Third service consumer** — when a service with a genuinely new storage shape arrives, reconsider promoting the harness toward a broader abstraction (evidence-based, n=3).
2. **Torrent management** — add/pause/delete actions (would require write endpoints + confirmation UX).
3. **Storage admin UI** — surface harness-managed table sizes and a "clear cached data" action per service.
4. **Reverse-proxy / no-auth qBittorrent flag** — for setups behind Authentik where native login is bypassed.
5. **Transfer-byte totals** — if desired later, add a separate widget using `transferInfo.globalUploaded/Downloaded`.