manage/openspec/changes/service-storage-harness/proposal.md

SDD Proposal: Service Storage Harness (with qBittorrent widgets + MediaIndex migration)

Change: service-storage-harness Phase: proposal Date: 2026-07-08

1. Problem / Why Now

Two things are happening at once, and this change addresses both:

1. Feature request — qBittorrent widgets. The operator wants at-a-glance qBittorrent visibility on the Manage dashboard: active downloads/uploads, total bytes transferred, and a download/upload speed indicator. qBittorrent is currently a blind spot — it is not modeled in the service registry and exposes no widgets.

2. Platform gap — services have no owned persistence. Manage already has a service registry (machines, Jellyfin, Grafana, Prometheus, Alertmanager, Jellyseerr, ssh_tasks) and a configurable widget system. But when a service needs to remember operational data over time, there is no shared answer. MediaIndex is a one-off: it owns its own media_index.db, runs its own schema, and is built by a dedicated subprocess worker. It is a "special snowflake." Rather than add a second snowflake for qBittorrent speed history, we extract the shared lifecycle into a small ServiceDataHarness and prove it with qBittorrent, then migrate MediaIndex onto it so the pattern has two evidence-based consumers.

The two are bundled because the second service (qBittorrent) is the one that justifies generalizing from the first (MediaIndex) — n=2 is what makes the abstraction worth its cost.

2. Target Users and Situations

• Primary users: Homelab operators running qBittorrent alongside Manage, who want download/upload activity visible without opening the qBittorrent UI.
• Workflow moments:
  • Glance at the dashboard: "is a download running, how fast, how much has been transferred?"
  • Decide whether qBittorrent is healthy without leaving Manage.
  • Browse the Jellyfin media catalog (unchanged UX) — which now runs on the same storage harness, validating the abstraction.
• Urgency: Medium. The feature is valuable but not breaking; the platform refactor is opportunistic (do it now while only two consumers exist, before a third snowflake appears).

3. Product Outcome

After this change, an authenticated user can:

• Register one or more qBittorrent service instances in the existing Services UI (URL + username + password, stored as encrypted secrets — same posture as Grafana/Prometheus).
• Place three discrete qBittorrent widget kinds on the dashboard:
  1. Totals tile — count of currently-listed torrents (total items in the qBittorrent list, which may exceed the active count because rate/connection limits leave some torrents non-transferring). NOT cumulative bytes transferred.
  2. Active torrents list — torrents whose state is downloading or uploading.
  3. Speed chart — live download/upload speed over a short rolling window, rendered with the PrometheusChartWidget recharts renderer established by the prometheus-direct-charting change (fed from the QbittorrentSampleStore via an InService-style data path returning the same {series} shape). No hand-rolled SVG sparkline; the thin-dashboard rule was already repealed for recharts charting by the foundational change.
• Continue using the Jellyfin Media page exactly as before; its underlying storage moves onto the harness transparently and additionally becomes multi-instance capable (scoped per Jellyfin service).

4. Scope Boundaries and Non-Goals

In scope

• ServiceDataHarness — a general lifecycle layer: per-concern DB files, per-integration schema migrations, service_id scoping of all tables, and cascade-delete when a service instance is removed.
• qBittorrent integration — new integrations/qbittorrent.py (config + secret schema + widget kinds), new clients/qbittorrent.py (Web API client, cookie login), a QbittorrentSampleStore (speed samples), widget source adapter(s) in widgets/sources.py, registry entry.
• qBittorrent frontend — three widget components under frontend/src/widgets/, binding in integrations/registry.ts, types, API client functions.
• MediaIndex migration — MediaIndex registered with the harness, media_items scoped by service_id, subprocess worker updated. The media_index.db file location is unchanged (only gains a column + harness registration) to minimize churn on a load-bearing feature.
• Cascade-delete wiring — removing a service instance cleans up its owned data in every harness-managed table.

Non-goals (explicitly out of scope)

• Torrent management UI — no add/pause/delete/recheck/priority UI. Read-only visibility only.
• A generic time-series database. The harness owns lifecycle, not a generic (service_id, key, ts, value) table. Each integration owns its own schema and operations.
• A generic CRUD/ORM layer. Stores keep bespoke operations (append/window vs replace_all/query); only the lifecycle is shared.
• Per-user or per-tenant storage partitioning. Storage is scoped by service_id only.
• Re-indexing/migrating existing media data. The media_index.db file stays in place; the migration is a schema column add + harness registration, not a data move.
• WebSocket / push updates. Polling via existing useWidgetData(widgetId, refreshIntervalMs) is sufficient.
• Transfer-byte totals. The totals widget counts torrent items, not cumulative bytes uploaded/downloaded (per the §8 Q2 resolution). Byte totals are out of scope.
• qBittorrent Prometheus exporter. Not built; the speed chart is sourced from the local QbittorrentSampleStore, not Prometheus.

5. High-Level Approach

5.1 Architectural decisions (locked during grilling)

Decision	Outcome
Storage tech	SQLite. Low volume; proven in stack.
Abstraction level	ServiceDataHarness owns the lifecycle (DB filename, migrations, service_id scoping, cascade-delete). Each integration owns its operations in a bespoke Store. General where shared; bespoke where not.
MediaIndex migration	Included in this change, but sequenced: harness + qBit proven first; MediaIndex folded in after. Inside one change, never two simultaneous risks.
DB topology	Per-concern DB files. media_index.db keeps its file (gains service_id + registration); new qbittorrent.db.

5.2 Backend

1. ServiceDataHarness (new, services/service_data.py or similar):

   • A registry of concerns: each integration declares a DB filename, an ordered list of migration SQL statements, the tables it owns, and a service_id column convention.
   • On startup: runs pending migrations per concern DB.
   • On service deletion: cascades — for each concern table, DELETE FROM <table> WHERE service_id = ?.
   • Provides connection management per concern DB (separate connections → bulk media writes don't share a writer with chatty qBit appends).

2. qBittorrent client (clients/qbittorrent.py):

   • Login via /api/v2/auth/login → cookie session; reuse cookie across calls; re-login on 403.
   • Endpoints used: /api/v2/transfer/info (global totals), /api/v2/sync/maindata (active torrents + current speeds), /api/v2/torrents/info (filtered lists if needed).
   • Timeout + reachability handling consistent with JellyfinClient/GrafanaWidgetSource.

3. QbittorrentSampleStore (new):

   • Table qbittorrent_speed_samples(service_id TEXT, ts INTEGER, dl_speed INTEGER, up_speed INTEGER) in qbittorrent.db.
   • Operations: append(service_id, ts, dl, up), window(service_id, since_ts), prune(service_id, older_than_ts).
   • Pruning runs on each append (cap retention to the configured window, e.g. 120 samples).

4. Widget source adapter — QbittorrentWidgetSource in widgets/sources.py implementing the existing WidgetSource.fetch(service, widget_kind, config) contract. For the speed kind, fetch appends a sample to the store and returns the current window for rendering.

5. Integration registration — add qbittorrent to integrations/registry.py with config schema (base URL, timeout) and secret schema (username, password); declare widget kinds.

6. MediaIndex migration — register MediaIndex as a concern; add service_id column (backfill existing rows with the default/local Jellyfin service id); update replace_items/query/worker to scope by service_id; route through the harness connection.

5.3 Frontend

1. Three widget components under frontend/src/widgets/:
   • QbittorrentTotalsWidget.tsx — numeric tiles for all-time bytes (reuses MetricCard/SectionCard).
   • QbittorrentActiveTorrentsWidget.tsx — list/table of active torrents (reuses the already-migrated DataTable + @tanstack/react-table; no new DataGrid migration risk).
   • QbittorrentSpeedWidget.tsx — visual form decided by §8 Q1 (numeric tiles + delta, or sparkline if an exception is granted, or a PrometheusMetricWidget binding if the exporter path is chosen).
2. Registry binding — add a qbittorrent entry to SERVICE_REGISTRY in frontend/src/integrations/registry.ts mapping the three widget kinds to components + config schemas.
3. Types + API client — frontend/src/types/index.ts and frontend/src/api/client.ts gain qBittorrent-aware widget kinds only (data flows through the existing useWidgetData polling; no new endpoints beyond widget CRUD).

5.4 Type contracts

• Backend Pydantic models for qBittorrent config/secret schema in integrations/qbittorrent.py.
• Frontend TypeScript interfaces for qBittorrent widget payloads.
• Harness has no data-shape types of its own (it is lifecycle-only) — keeping the seam clean.

6. Success Criteria / Acceptance Criteria

1. A user can register a qBittorrent service instance (URL + username + password) and the three widget kinds appear as bindable on the dashboard.
2. Totals tile shows all-time bytes from transferInfo; active list shows downloading/uploading torrents; speed indicator reflects current rates.
3. ServiceDataHarness runs migrations on startup and cascades deletes across qBittorrent samples and media items when a service is removed.
4. MediaIndex continues to power the Media page identically (all existing Media tests green) and is now scoped by service_id.
5. Removing a Jellyfin service removes only that service's media rows; removing a qBittorrent service removes only that service's speed samples.
6. A misconfigured/unreachable qBittorrent instance degrades gracefully per-widget (error state), the rest of the dashboard renders.
7. Backend tests (pytest) and frontend npm run build + npm run lint stay green.
8. No secrets land in widget config_json; qBittorrent password is encrypted via the existing Fernet path.
9. No new charting dependency is added unless §8 Q1 grants an explicit exception.

7. Risks and Mitigations

Risk	Mitigation
Compound risk: new abstraction × load-bearing refactor. A wrong harness shape breaks the Media page.	Sequence inside the change: harness + qBit land and pass tests first; MediaIndex migrates only after the harness shape is settled in code.
Harness over-generalization. Building a meta-framework from n=2.	Keep the harness lifecycle-only; do NOT add a generic value table or generic CRUD. Each store keeps bespoke operations. Extract further only when a third shape appears.
Thin-dashboard rule collision (§8 Q1). In-app speed graph violates the stated "no in-app charting" rule.	Resolve in the question round before spec; default to the rule-compliant numeric-tile + Grafana deep-link option unless an explicit exception is granted.
qBittorrent auth lifecycle. Cookie expiry / 403 handling.	Re-login transparently on 403; short request timeout; surface persistent auth failure as widget error state.
MediaIndex backfill correctness. Adding service_id to existing rows.	Backfill all existing rows to the default/local Jellyfin service id; migration is additive; existing Media tests must pass unchanged.
Review budget (>400 changed lines). Bundled scope is large.	Slice into chained PRs (Slice 1: harness + qBit; Slice 2: MediaIndex migration). Each slice leaves npm run build + npm run lint + pytest green.
DataGrid migration (config rule callout).	Not a risk here: the active-torrents list reuses the already-migrated DataTable (@tanstack/react-table). No new DataGrid migration is introduced. The key technical risks are the harness abstraction and the MediaIndex refactor, above.

8. Resolved Questions (question round complete)

All five product/semantic questions resolved during grilling + the Change A (prometheus-direct-charting) lifecycle:

• Q1 — Speed visualization. RESOLVED: the thin-dashboard rule was repealed by Change A; in-app recharts charting is now sanctioned. The qBit speed widget reuses the PrometheusChartWidget recharts renderer, fed from QbittorrentSampleStore.window() via an InService data path returning the same {series} shape. No SVG sparkline, no Grafana dependency.
• Q2 — "Totals" semantics. RESOLVED: count of currently-listed torrent items (total in the qBittorrent list), which may exceed the active count due to rate/connection limits. NOT transfer bytes, NOT per-session counters.
• Q3 — "Active" definition. RESOLVED: torrents in state downloading or uploading only.
• Q4 — Instances. RESOLVED: support N qBittorrent instances, each a service row, independently scoped by service_id.
• Q5 — Auth model. RESOLVED: username/password login → cookie session, encrypted via Fernet. No reverse-proxy no-auth flag in this pass.

9. Future Phases

1. Third service consumer — when a service with a genuinely new storage shape arrives, reconsider promoting the harness toward a broader abstraction (evidence-based, n=3).
2. Torrent management — add/pause/delete actions (would require write endpoints + confirmation UX).
3. Storage admin UI — surface harness-managed table sizes and a "clear cached data" action per service.
4. Reverse-proxy / no-auth qBittorrent flag — for setups behind Authentik where native login is bypassed.
5. Transfer-byte totals — if desired later, add a separate widget using transferInfo.globalUploaded/Downloaded.