spec(service-storage-harness): add design
10 design decisions incl: lifecycle-only harness, per-concern DB files, QbittorrentSampleStore (120-sample cap), qBit cookie client, 3 widget kinds (totals=item count, active=DL/UL state, speed=reuses Change A renderer via shared LineSeriesChart extract), MediaIndex +service_id backfill, scoped replace_items (fixes latent global-clear bug), best-effort cascade-delete. 4-slice plan. 3 source findings (worker already threads service_id).
This commit is contained in:
Developer
@@ -0,0 +1,863 @@
|
||||
# SDD Design: Service Storage Harness (with qBittorrent widgets + MediaIndex migration)
|
||||
|
||||
**Change:** `service-storage-harness`
|
||||
**Phase:** design
|
||||
**Date:** 2026-07-09
|
||||
|
||||
> Grounded in `proposal.md` (Q1–Q5 resolved) and the already-archived
|
||||
> `prometheus-direct-charting` change. No source changes in this phase.
|
||||
|
||||
## 0. Source findings (read before anything else)
|
||||
|
||||
The proposal was written against a stale project map. Reading actual source
|
||||
surfaced three findings that shape the design. Trust source, not the map.
|
||||
|
||||
### 0.1 The media worker ALREADY threads `service_id` — but it is NOT persisted in rows
|
||||
|
||||
`workers/media_index_worker.py` already accepts `--service-id` via argparse
|
||||
(`run_build(final_index_path, staging_index_path, service_id="")`), and
|
||||
`_resolve_jellyfin(service_id)` resolves the Jellyfin client + user_id from the
|
||||
settings store. The media router's `_start_worker(index, service_id)` and
|
||||
`post_build_index` already pass `jellyfin_service_id` through to the worker.
|
||||
|
||||
**But:** `service_id` is used ONLY to pick the Jellyfin connection. It is never
|
||||
stored in the `media_items` table (which has no `service_id` column). The
|
||||
migration must close this gap: `replace_items` must scope its DELETE+INSERT by
|
||||
`service_id`, and `query` must filter by it. The plumbing to get `service_id`
|
||||
into the worker already exists — only the storage layer is missing.
|
||||
|
||||
### 0.2 `media_items` is cleared globally on every build
|
||||
|
||||
`replace_items` does `DELETE FROM media_items` (no WHERE clause). This means a
|
||||
build for Jellyfin instance A wipes instance B's rows. After migration this
|
||||
becomes `DELETE FROM media_items WHERE service_id = ?`.
|
||||
|
||||
### 0.3 The `delete_service` cascade already exists — the harness hooks into it
|
||||
|
||||
`SettingsStore.delete_service` already cascade-deletes `dashboard_widgets WHERE
|
||||
service_id = ?` (with a PRAGMA-guarded column check). The harness cascade-delete
|
||||
hooks into the same place: after the service row is deleted, iterate registered
|
||||
concerns and delete that `service_id` from each owned table.
|
||||
|
||||
### 0.4 DB path is a module constant, not in config.py
|
||||
|
||||
`DEFAULT_INDEX_PATH = Path(".cache/media_library_viewer/media_index.sqlite")` in
|
||||
`media_index_impl.py`. The file stays at this path (per the locked topology
|
||||
decision). The harness must not move it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ ServiceDataHarness │
|
||||
│ (services/service_data.py — LIFECYCLE ONLY) │
|
||||
│ │
|
||||
│ • register_concern(db_filename, migrations[], tables[], │
|
||||
│ service_id_column="service_id") │
|
||||
│ • run_migrations() — on startup, per concern DB │
|
||||
│ • cascade_delete(service_id) — iterate concerns, DELETE rows │
|
||||
│ • connect(db_filename) → sqlite3.Connection (per-concern) │
|
||||
└──────────────┬───────────────────────────┬───────────────────────┘
|
||||
│ │
|
||||
┌──────────▼──────────┐ ┌─────────▼──────────────┐
|
||||
│ QbittorrentStore │ │ MediaIndex │
|
||||
│ (services/ │ │ (services/ │
|
||||
│ qbittorrent_store) │ │ media_index_impl) │
|
||||
│ │ │ │
|
||||
│ qbittorrent.db │ │ media_index.sqlite │
|
||||
│ └ qbittorrent_ │ │ └ media_items │
|
||||
│ speed_samples │ │ (+ service_id col) │
|
||||
│ (service_id, ts, │ │ └ index_metadata │
|
||||
│ dl_speed, │ │ │
|
||||
│ up_speed) │ │ bespoke: replace_items │
|
||||
│ │ │ (scoped), query │
|
||||
│ bespoke: append, │ │ (scoped), status │
|
||||
│ window, prune │ │ │
|
||||
└──────────────────────┘ └────────────────────────┘
|
||||
│
|
||||
┌──────────▼──────────┐
|
||||
│ QbittorrentClient │
|
||||
│ (clients/ │
|
||||
│ qbittorrent) │
|
||||
│ │
|
||||
│ login → cookie │
|
||||
│ sync/maindata │
|
||||
│ (totals + active + │
|
||||
│ speeds) │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
**Key constraints carried from the proposal:**
|
||||
|
||||
- Harness is lifecycle-only: migrations, service_id scoping, cascade-delete. No generic value table, no generic CRUD (D2).
|
||||
- Per-concern DB files: `media_index.sqlite` stays put; new `qbittorrent.db` (D4).
|
||||
- MediaIndex migration is sequenced after harness + qBit are proven (D3).
|
||||
- qBit speed chart reuses the `PrometheusChartWidget` recharts renderer fed from an InService data path returning `{series}` (Q1).
|
||||
- Totals = count of listed items, NOT transfer bytes (Q2).
|
||||
- Active = state downloading|uploading (Q3). N instances (Q4). Username/password → cookie (Q5).
|
||||
|
||||
---
|
||||
|
||||
## 2. Backend design
|
||||
|
||||
### 2.1 `ServiceDataHarness` (`services/service_data.py`)
|
||||
|
||||
A lifecycle-only registry of storage concerns. Each concern declares its own DB
|
||||
filename, ordered migrations, owned tables, and the column used for service
|
||||
scoping.
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class StorageConcern:
|
||||
"""A per-integration storage namespace registered with the harness."""
|
||||
|
||||
concern_key: str # e.g. "qbittorrent", "media_index"
|
||||
db_filename: str # e.g. "qbittorrent.db", "media_index.sqlite"
|
||||
migrations: list[str] # ordered CREATE/ALTER statements (idempotent)
|
||||
tables: list[str] # tables owned by this concern (for cascade)
|
||||
service_id_column: str = "service_id"
|
||||
|
||||
|
||||
class ServiceDataHarness:
|
||||
"""Lifecycle-only registry of per-concern storage.
|
||||
|
||||
Owns: DB provisioning, per-concern migrations, service_id cascade-delete.
|
||||
Does NOT own: data operations (each store keeps bespoke append/window/query/etc.).
|
||||
"""
|
||||
|
||||
def __init__(self, base_dir: Path) -> None:
|
||||
self._base_dir = Path(base_dir)
|
||||
self._concerns: dict[str, StorageConcern] = {}
|
||||
|
||||
def register(self, concern: StorageConcern) -> None:
|
||||
"""Register a storage concern. Called at module import / startup."""
|
||||
self._concerns[concern.concern_key] = concern
|
||||
|
||||
def db_path(self, concern_key: str) -> Path:
|
||||
"""Return the absolute path to a concern's DB file."""
|
||||
concern = self._concerns[concern_key]
|
||||
return self._base_dir / concern.db_filename
|
||||
|
||||
def connect(self, concern_key: str) -> sqlite3.Connection:
|
||||
"""Open a WAL-mode connection to a concern's DB."""
|
||||
path = self.db_path(concern_key)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(path, timeout=30)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.execute("PRAGMA busy_timeout=30000")
|
||||
return conn
|
||||
|
||||
def run_migrations(self) -> None:
|
||||
"""Run pending migrations for every registered concern."""
|
||||
for concern in self._concerns.values():
|
||||
path = self.db_path(concern.concern_key)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sqlite3.connect(path, timeout=30) as conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
conn.executescript(";".join(concern.migrations))
|
||||
|
||||
def cascade_delete(self, service_id: str) -> None:
|
||||
"""Delete all rows for a service_id across every concern's tables.
|
||||
|
||||
Called from SettingsStore.delete_service after the service row is removed.
|
||||
"""
|
||||
for concern in self._concerns.values():
|
||||
col = concern.service_id_column
|
||||
with sqlite3.connect(self.db_path(concern.concern_key), timeout=30) as conn:
|
||||
for table in concern.tables:
|
||||
cols = {row[1] for row in conn.execute(f"PRAGMA table_info({table})").fetchall()}
|
||||
if col in cols:
|
||||
conn.execute(f"DELETE FROM {table} WHERE {col} = ?", (service_id,))
|
||||
```
|
||||
|
||||
**Module-level singleton + registration:**
|
||||
|
||||
```python
|
||||
_HARNESS: ServiceDataHarness | None = None
|
||||
|
||||
def get_service_data_harness() -> ServiceDataHarness:
|
||||
global _HARNESS
|
||||
if _HARNESS is None:
|
||||
base_dir = Path(os.environ.get("BACKEND_CACHE_DIR", ".cache/media_library_viewer"))
|
||||
_HARNESS = ServiceDataHarness(base_dir)
|
||||
# Register built-in concerns (each store module calls register on import)
|
||||
_HARNESS.register(QBITTORRENT_CONCERN)
|
||||
_HARNESS.register(MEDIA_INDEX_CONCERN)
|
||||
_HARNESS.run_migrations()
|
||||
return _HARNESS
|
||||
```
|
||||
|
||||
**Startup hook:** called from `main.py` lifespan alongside `get_settings_store().ensure_defaults()`. The harness is lazy-initialized on first access (like `SettingsStore`), so tests can override the base dir via env.
|
||||
|
||||
### 2.2 `QbittorrentSampleStore` (`services/qbittorrent_store.py`)
|
||||
|
||||
Speed-sample storage for the qBit speed widget. Registered as a concern with the
|
||||
harness.
|
||||
|
||||
```python
|
||||
QBITTORRENT_CONCERN = StorageConcern(
|
||||
concern_key="qbittorrent",
|
||||
db_filename="qbittorrent.db",
|
||||
migrations=[
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS qbittorrent_speed_samples (
|
||||
service_id TEXT NOT NULL,
|
||||
ts INTEGER NOT NULL,
|
||||
dl_speed INTEGER NOT NULL DEFAULT 0,
|
||||
up_speed INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_qbit_samples_service_ts
|
||||
ON qbittorrent_speed_samples(service_id, ts);
|
||||
"""
|
||||
],
|
||||
tables=["qbittorrent_speed_samples"],
|
||||
)
|
||||
|
||||
MAX_SAMPLES = 120 # ~2 min at 1s poll, ~4 min at 2s poll
|
||||
|
||||
|
||||
class QbittorrentSampleStore:
|
||||
"""Bespoke speed-sample store for qBittorrent widgets."""
|
||||
|
||||
def __init__(self, harness: ServiceDataHarness | None = None) -> None:
|
||||
self._harness = harness or get_service_data_harness()
|
||||
|
||||
def append(self, service_id: str, ts: int, dl_speed: int, up_speed: int) -> None:
|
||||
"""Append a sample and prune old entries beyond MAX_SAMPLES."""
|
||||
with self._harness.connect("qbittorrent") as conn:
|
||||
conn.execute(
|
||||
"INSERT INTO qbittorrent_speed_samples (service_id, ts, dl_speed, up_speed) VALUES (?, ?, ?, ?)",
|
||||
(service_id, ts, dl_speed, up_speed),
|
||||
)
|
||||
# Prune: keep only the most recent MAX_SAMPLES rows for this service
|
||||
conn.execute(
|
||||
"""DELETE FROM qbittorrent_speed_samples
|
||||
WHERE service_id = ? AND ts NOT IN (
|
||||
SELECT ts FROM qbittorrent_speed_samples
|
||||
WHERE service_id = ?
|
||||
ORDER BY ts DESC LIMIT ?
|
||||
)""",
|
||||
(service_id, service_id, MAX_SAMPLES),
|
||||
)
|
||||
|
||||
def window(self, service_id: str, since_ts: int | None = None) -> list[dict[str, Any]]:
|
||||
"""Return all samples for a service since a timestamp (or all if None)."""
|
||||
with self._harness.connect("qbittorrent") as conn:
|
||||
if since_ts is not None:
|
||||
rows = conn.execute(
|
||||
"SELECT ts, dl_speed, up_speed FROM qbittorrent_speed_samples WHERE service_id = ? AND ts >= ? ORDER BY ts ASC",
|
||||
(service_id, since_ts),
|
||||
).fetchall()
|
||||
else:
|
||||
rows = conn.execute(
|
||||
"SELECT ts, dl_speed, up_speed FROM qbittorrent_speed_samples WHERE service_id = ? ORDER BY ts ASC",
|
||||
(service_id,),
|
||||
).fetchall()
|
||||
return [{"ts": r[0], "dl_speed": r[1], "up_speed": r[2]} for r in rows]
|
||||
```
|
||||
|
||||
### 2.3 `QbittorrentClient` (`clients/qbittorrent.py`)
|
||||
|
||||
Cookie-session HTTP client modeled on `JellyfinClient`'s session pattern.
|
||||
|
||||
```python
|
||||
class QbittorrentClient:
|
||||
"""Minimal qBittorrent Web API client (read-only: sync/maindata only)."""
|
||||
|
||||
def __init__(self, base_url: str, username: str, password: str, timeout: int = 10) -> None:
|
||||
self.base_url = base_url.rstrip("/")
|
||||
if not self.base_url.endswith("/api/v2"):
|
||||
self.base_url += "/api/v2"
|
||||
self._username = username
|
||||
self._password = password
|
||||
self._timeout = timeout
|
||||
self._session = requests.Session()
|
||||
self._logged_in = False
|
||||
|
||||
def _login(self) -> None:
|
||||
"""POST username/password to /auth/login; store the SID cookie."""
|
||||
resp = self._session.post(
|
||||
f"{self.base_url}/auth/login",
|
||||
data={"username": self._username, "password": self._password},
|
||||
timeout=self._timeout,
|
||||
headers={"Referer": self.base_url},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
if resp.text.strip() != "Ok.":
|
||||
raise RuntimeError(f"qBittorrent login failed: {resp.text.strip()}")
|
||||
self._logged_in = True
|
||||
|
||||
def _get(self, path: str, **params: Any) -> dict[str, Any]:
|
||||
"""GET with auto re-login on 403."""
|
||||
if not self._logged_in:
|
||||
self._login()
|
||||
url = f"{self.base_url}{path}"
|
||||
resp = self._session.get(url, params=params, timeout=self._timeout)
|
||||
if resp.status_code == 403:
|
||||
self._logged_in = False
|
||||
self._login()
|
||||
resp = self._session.get(url, params=params, timeout=self._timeout)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
def maindata(self) -> dict[str, Any]:
|
||||
"""Fetch /sync/maindata — returns server_state + torrents dict.
|
||||
|
||||
server_state contains: dl_info_speed (bytes/s), up_info_speed (bytes/s), etc.
|
||||
torrents is a dict of {hash: {name, state, progress, ...}}.
|
||||
"""
|
||||
return self._get("/sync/maindata")
|
||||
```
|
||||
|
||||
**Endpoints used (all from `/sync/maindata` — single call covers all three widgets):**
|
||||
|
||||
| Widget | Data extracted from `maindata()` |
|
||||
|---|---|
|
||||
| **totals** | `len(response["torrents"])` — count of all listed torrents |
|
||||
| **active** | `filter(t for t in response["torrents"].values() if t["state"] in {"downloading","uploading"})` |
|
||||
| **speed** | `response["server_state"]["dl_info_speed"]` + `["up_info_speed"]` (current instant speed, appended to store) |
|
||||
|
||||
**Note:** `/api/v2/transfer/info` is NOT needed (totals = item count per Q2, not transfer bytes).
|
||||
|
||||
### 2.4 Widget source adapter — `QbittorrentWidgetSource`
|
||||
|
||||
Lives in `widgets/sources.py`, implements `WidgetSource.fetch(service, widget_kind, config)`.
|
||||
Resolves the client inline from `ServiceRecord` (like `PrometheusWidgetSource` does — no new
|
||||
dependency-injection helper needed).
|
||||
|
||||
```python
|
||||
class QbittorrentWidgetSource:
|
||||
"""Fetch qBittorrent data for totals, active, and speed widgets."""
|
||||
|
||||
async def fetch(self, service: ServiceRecord | None, widget_kind: str, config: dict[str, Any]) -> dict[str, Any]:
|
||||
try:
|
||||
if service is None:
|
||||
return {"error": "qBittorrent widget is missing its service"}
|
||||
base_url = str(service.config.get("base_url") or "")
|
||||
username = str(service.secrets.get("username") or "")
|
||||
password = str(service.secrets.get("password") or "")
|
||||
timeout = int(service.config.get("timeout_seconds") or 10)
|
||||
if not base_url or not username or not password:
|
||||
return {"error": "qBittorrent service is missing base_url, username, or password"}
|
||||
|
||||
client = QbittorrentClient(base_url, username, password, timeout)
|
||||
data = await asyncio.wait_for(asyncio.to_thread(client.maindata), timeout=timeout)
|
||||
server_state = data.get("server_state", {})
|
||||
torrents = data.get("torrents", {})
|
||||
|
||||
if widget_kind == "totals":
|
||||
# Q2: count of listed items, broken down by state
|
||||
by_state: dict[str, int] = {}
|
||||
for t in torrents.values():
|
||||
state = str(t.get("state", "unknown"))
|
||||
by_state[state] = by_state.get(state, 0) + 1
|
||||
return {"total": len(torrents), "by_state": by_state}
|
||||
|
||||
if widget_kind == "active":
|
||||
# Q3: downloading or uploading only
|
||||
active = [
|
||||
{"name": t.get("name"), "state": t.get("state"),
|
||||
"size": t.get("size"), "progress": t.get("progress"),
|
||||
"dl_speed": t.get("dlspeed"), "up_speed": t.get("upspeed")}
|
||||
for t in torrents.values()
|
||||
if str(t.get("state", "")) in {"downloading", "uploading"}
|
||||
]
|
||||
return {"torrents": active}
|
||||
|
||||
if widget_kind == "speed":
|
||||
# Q1: append sample + return window as {series} shape
|
||||
# matching PrometheusChartWidget's expected format
|
||||
dl = int(server_state.get("dl_info_speed", 0))
|
||||
up = int(server_state.get("up_info_speed", 0))
|
||||
ts = int(time.time())
|
||||
store = QbittorrentSampleStore()
|
||||
store.append(service.id, ts, dl, up)
|
||||
samples = store.window(service.id)
|
||||
series = [
|
||||
{"label": "download",
|
||||
"points": [{"t": s["ts"] * 1000, "v": s["dl_speed"]} for s in samples]},
|
||||
{"label": "upload",
|
||||
"points": [{"t": s["ts"] * 1000, "v": s["up_speed"]} for s in samples]},
|
||||
]
|
||||
return {"series": series}
|
||||
|
||||
return {"error": f"Unknown qBittorrent widget kind: {widget_kind}"}
|
||||
except asyncio.TimeoutError:
|
||||
return {"error": "qBittorrent data fetch timed out"}
|
||||
except Exception as exc:
|
||||
logger.exception("qbittorrent adapter failed")
|
||||
return {"error": f"qBittorrent fetch failed: {exc}"}
|
||||
```
|
||||
|
||||
**Registered in `SERVICE_ADAPTERS`:**
|
||||
|
||||
```python
|
||||
SERVICE_ADAPTERS: dict[str, WidgetSource] = {
|
||||
"prometheus": PrometheusWidgetSource(),
|
||||
"qbittorrent": QbittorrentWidgetSource(), # NEW
|
||||
"alertmanager": AlertmanagerWidgetSource(),
|
||||
"jellyfin": JellyfinWidgetSource(),
|
||||
"ssh_tasks": SshTaskWidgetSource(),
|
||||
}
|
||||
```
|
||||
|
||||
### 2.5 Integration registration — `integrations/qbittorrent.py`
|
||||
|
||||
Models config + secret schema on `prometheus.py`.
|
||||
|
||||
```python
|
||||
class QbittorrentConfig(ServiceConfigBase):
|
||||
base_url: ServiceBaseUrl
|
||||
timeout_seconds: int = 10
|
||||
|
||||
|
||||
# No per-widget config needed for any of the three kinds
|
||||
# (all derive from the service connection).
|
||||
class QbittorrentWidgetConfig(WidgetConfigBase):
|
||||
pass
|
||||
|
||||
|
||||
DEFINITION = ServiceDefinition(
|
||||
service_type="qbittorrent",
|
||||
name="qBittorrent",
|
||||
description="Torrent client activity, speeds, and item counts.",
|
||||
config_model=QbittorrentConfig,
|
||||
secret_fields=[
|
||||
SecretField(key="username", label="Username", required=True),
|
||||
SecretField(key="password", label="Password", required=True, helper="Stored encrypted"),
|
||||
],
|
||||
widget_kinds=[
|
||||
widget_kind(kind="totals", name="Totals",
|
||||
description="Count of all listed torrents, broken down by state.",
|
||||
model_cls=QbittorrentWidgetConfig, default_config={},
|
||||
refresh_interval_ms=30_000),
|
||||
widget_kind(kind="active", name="Active torrents",
|
||||
description="Torrents currently downloading or uploading.",
|
||||
model_cls=QbittorrentWidgetConfig, default_config={},
|
||||
refresh_interval_ms=15_000),
|
||||
widget_kind(kind="speed", name="Speed chart",
|
||||
description="Live download/upload speed over a short window.",
|
||||
model_cls=QbittorrentWidgetConfig, default_config={},
|
||||
refresh_interval_ms=5_000),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
**Registered in `integrations/registry.py`:**
|
||||
|
||||
```python
|
||||
from media_library_viewer_api.integrations.qbittorrent import DEFINITION as QBITTORRENT
|
||||
SERVICE_DEFINITIONS["qbittorrent"] = QBITTORRENT
|
||||
```
|
||||
|
||||
### 2.6 Cascade-delete wiring
|
||||
|
||||
`SettingsStore.delete_service` gains a harness call after the service row is
|
||||
deleted:
|
||||
|
||||
```python
|
||||
def delete_service(self, service_id: str) -> None:
|
||||
self.init_schema()
|
||||
with self.connect() as conn:
|
||||
# existing widget cascade ...
|
||||
conn.execute("DELETE FROM services WHERE id = ?", (service_id,))
|
||||
# NEW: cascade-delete harness-managed data
|
||||
try:
|
||||
from media_library_viewer_api.services.service_data import get_service_data_harness
|
||||
get_service_data_harness().cascade_delete(service_id)
|
||||
except Exception:
|
||||
logger.exception("harness cascade-delete failed for service %s", service_id)
|
||||
```
|
||||
|
||||
The try/except guard prevents a harness failure from blocking service deletion
|
||||
(data cleanup is best-effort; the service row is already gone).
|
||||
|
||||
---
|
||||
|
||||
## 3. MediaIndex migration design (load-bearing)
|
||||
|
||||
### 3.1 Schema migration: add `service_id` column
|
||||
|
||||
The `init_schema` method's `CREATE TABLE IF NOT EXISTS` gains the new column.
|
||||
For existing databases, a migration adds it:
|
||||
|
||||
```sql
|
||||
-- In MEDIA_INDEX_CONCERN.migrations (runs via harness on startup):
|
||||
-- The init_schema already creates the table for new installs WITH service_id.
|
||||
-- This migration handles existing DBs that lack the column.
|
||||
|
||||
-- media_index_impl.py init_schema: add service_id to the CREATE TABLE columns.
|
||||
-- Harness migration (runs on existing DBs):
|
||||
ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT '';
|
||||
```
|
||||
|
||||
The `DEFAULT ''` backfills all existing rows to empty string (the legacy
|
||||
sentinel — see §3.3 for how this is resolved).
|
||||
|
||||
### 3.2 `replace_items` — scoped delete + insert
|
||||
|
||||
```python
|
||||
def replace_items(self, rows: Iterable[dict[str, Any]], service_id: str = "") -> int:
|
||||
self.init_schema()
|
||||
row_list = list(rows)
|
||||
# ... columns list gains "service_id" ...
|
||||
with self.connect() as conn:
|
||||
# SCOPED: only delete this service's rows
|
||||
conn.execute("DELETE FROM media_items WHERE service_id = ?", (service_id,))
|
||||
conn.executemany(
|
||||
f"INSERT OR REPLACE INTO media_items ({','.join(columns)}) VALUES ({placeholders})",
|
||||
[[service_id] + [row.get(column) for column in columns_without_service_id] for row in row_list],
|
||||
)
|
||||
# ... metadata ...
|
||||
return len(row_list)
|
||||
```
|
||||
|
||||
### 3.3 `query` — scoped filter
|
||||
|
||||
```python
|
||||
def query(self, service_id: str = "", ...) -> tuple[list[dict[str, Any]], int]:
|
||||
# ... existing where clauses ...
|
||||
# Add service_id filter: if non-empty, scope; if empty (legacy), show all
|
||||
# (backward-compatibility for the period before multi-instance is wired in UI)
|
||||
if service_id:
|
||||
where.append("service_id = ?")
|
||||
params.append(service_id)
|
||||
# ... rest unchanged ...
|
||||
```
|
||||
|
||||
**Backfill semantics:** existing rows get `service_id = ''` (empty string). When
|
||||
`service_id` is empty string in the query, the filter is skipped, so the Media
|
||||
page shows all items (backward-compatible behavior). When a specific Jellyfin
|
||||
service triggers a rebuild, `replace_items(rows, service_id=that_service)` scopes
|
||||
the delete + insert. New builds set the real service_id; legacy rows remain
|
||||
visible until a rebuild replaces them.
|
||||
|
||||
### 3.4 `build_media_index` — thread service_id
|
||||
|
||||
`build_media_index` already receives no `service_id` today. Add it as a
|
||||
parameter and pass it through to `replace_items`:
|
||||
|
||||
```python
|
||||
def build_media_index(
|
||||
client, user_id, libraries, index=None, page_size=500,
|
||||
media_root="", fallback_prefix="",
|
||||
progress_callback=None, should_cancel=None,
|
||||
service_id: str = "", # NEW
|
||||
) -> int:
|
||||
# ... existing logic ...
|
||||
# Pass service_id to replace_items:
|
||||
processed_total = index.replace_items(normalized_rows, service_id=service_id)
|
||||
```
|
||||
|
||||
### 3.5 Worker — already threads `service_id`, just pass it to `build_media_index`
|
||||
|
||||
The worker's `run_build(final_index_path, staging_index_path, service_id="")`
|
||||
already receives `service_id` from argparse. The only change: pass it to
|
||||
`build_media_index(..., service_id=service_id)`.
|
||||
|
||||
### 3.6 Media router — thread `service_id` into `query_media`
|
||||
|
||||
`query_media` already resolves `client: JellyfinClient = Depends(get_jellyfin_client)`.
|
||||
The service_id is available via request query param (the dependency layer resolves
|
||||
it from `?jellyfin_service_id=...`). Add it to the query call:
|
||||
|
||||
```python
|
||||
@router.get("/query")
|
||||
def query_media(
|
||||
...,
|
||||
jellyfin_service_id: str | None = None, # already available pattern
|
||||
index: MediaIndex = Depends(get_media_index),
|
||||
) -> dict[str, Any]:
|
||||
...
|
||||
rows, total = index.query(service_id=jellyfin_service_id or "", ...)
|
||||
```
|
||||
|
||||
### 3.7 MediaIndex concern registration
|
||||
|
||||
```python
|
||||
MEDIA_INDEX_CONCERN = StorageConcern(
|
||||
concern_key="media_index",
|
||||
db_filename="media_index.sqlite", # SAME FILE, unchanged path
|
||||
migrations=[
|
||||
"ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT ''",
|
||||
# This ALTER is idempotent-safe: init_schema creates the table WITH
|
||||
# service_id for new installs; this migration adds it to existing DBs.
|
||||
# SQLite ALTER TABLE ADD COLUMN is a no-op if the column already exists
|
||||
# (we guard with a PRAGMA check in run_migrations, or catch the error).
|
||||
],
|
||||
tables=["media_items"],
|
||||
)
|
||||
```
|
||||
|
||||
**IMPORTANT — ALTER TABLE idempotency:** SQLite raises an error if the column
|
||||
already exists. The `run_migrations` method should catch this per-statement or
|
||||
pre-check via `PRAGMA table_info`. Design choice: wrap each migration in a
|
||||
try/except for "duplicate column name" errors:
|
||||
|
||||
```python
|
||||
def run_migrations(self) -> None:
|
||||
for concern in self._concerns.values():
|
||||
path = self.db_path(concern.concern_key)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with sqlite3.connect(path, timeout=30) as conn:
|
||||
conn.execute("PRAGMA journal_mode=WAL")
|
||||
for stmt in concern.migrations:
|
||||
try:
|
||||
conn.executescript(stmt)
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "duplicate column name" not in str(exc).lower():
|
||||
raise
|
||||
```
|
||||
|
||||
### 3.8 File location confirmation
|
||||
|
||||
`DEFAULT_INDEX_PATH = Path(".cache/media_library_viewer/media_index.sqlite")` —
|
||||
unchanged. The harness `base_dir` defaults to `Path(".cache/media_library_viewer")`
|
||||
(the same parent). The `db_filename = "media_index.sqlite"` matches. No data move.
|
||||
|
||||
---
|
||||
|
||||
## 4. Frontend design
|
||||
|
||||
### 4.1 Three widget components
|
||||
|
||||
All three live in `frontend/src/widgets/`, modeled on existing patterns
|
||||
(`AlertmanagerAlertsWidget`, `PrometheusChartWidget`, `MetricCard`).
|
||||
|
||||
**`QbittorrentTotalsWidget.tsx`** — MetricCard-style count tile:
|
||||
|
||||
```tsx
|
||||
// Renders {total: number, by_state: {...}} from useWidgetData.
|
||||
// Uses SectionCard + numeric display (like BackupsWidget / MetricCard).
|
||||
// Shows total count prominently + state breakdown badges.
|
||||
```
|
||||
|
||||
**`QbittorrentActiveTorrentsWidget.tsx`** — active torrents list:
|
||||
|
||||
```tsx
|
||||
// Renders {torrents: [{name, state, size, progress, dl_speed, up_speed}]}.
|
||||
// Uses SectionCard + a compact list/table (DataTable or manual Table rows).
|
||||
// Shows name, state badge, progress bar, speeds. Max 10 rows with scroll.
|
||||
```
|
||||
|
||||
**`QbittorrentSpeedWidget.tsx`** — speed chart reusing Change A's renderer:
|
||||
|
||||
```tsx
|
||||
// Renders {series: [{label, points:[{t,v}]}]} — IDENTICAL to PrometheusChartWidget.
|
||||
// Two options:
|
||||
// (a) Import PrometheusChartWidget directly and pass props (if its props accept
|
||||
// the series externally rather than via useWidgetData).
|
||||
// (b) Extract the recharts rendering into a shared <LineSeriesChart series={...} />
|
||||
// component that both PrometheusChartWidget and QbittorrentSpeedWidget use.
|
||||
//
|
||||
// RECOMMENDED: option (b) — extract a shared LineSeriesChart component (~40 lines)
|
||||
// into frontend/src/components/LineSeriesChart.tsx. Both widgets call useWidgetData
|
||||
// independently (different refresh intervals) but share the renderer.
|
||||
```
|
||||
|
||||
**`LineSeriesChart.tsx`** (shared renderer extraction):
|
||||
|
||||
```tsx
|
||||
// Extracts: mergeSeries, formatTime, CHART_COLORS, and the <ResponsiveContainer>
|
||||
// + <LineChart> JSX from PrometheusChartWidget.
|
||||
// Props: { series: ChartSeries[], height?: number }
|
||||
// PrometheusChartWidget becomes a thin wrapper: useWidgetData → <LineSeriesChart series={data.data.series} />
|
||||
// QbittorrentSpeedWidget: same pattern, different refresh interval (5s vs 60s).
|
||||
```
|
||||
|
||||
This keeps the recharts rendering in ONE place (no duplication) while letting each
|
||||
widget own its polling lifecycle.
|
||||
|
||||
### 4.2 Frontend registry binding (`integrations/registry.ts`)
|
||||
|
||||
```typescript
|
||||
qbittorrent: {
|
||||
serviceType: "qbittorrent",
|
||||
name: "qBittorrent",
|
||||
description: "Torrent client activity, speeds, and item counts.",
|
||||
widgets: [
|
||||
{ kind: "totals", name: "Totals", description: "...",
|
||||
refreshIntervalMs: 30_000, defaultConfig: {},
|
||||
configSchema: { type: "object", properties: {}, required: [] },
|
||||
component: QbittorrentTotalsWidget },
|
||||
{ kind: "active", name: "Active torrents", description: "...",
|
||||
refreshIntervalMs: 15_000, defaultConfig: {},
|
||||
configSchema: { type: "object", properties: {}, required: [] },
|
||||
component: QbittorrentActiveTorrentsWidget },
|
||||
{ kind: "speed", name: "Speed chart", description: "...",
|
||||
refreshIntervalMs: 5_000, defaultConfig: {},
|
||||
configSchema: { type: "object", properties: {}, required: [] },
|
||||
component: QbittorrentSpeedWidget },
|
||||
],
|
||||
},
|
||||
```
|
||||
|
||||
### 4.3 Types
|
||||
|
||||
No new TypeScript types needed for widget payloads — data flows through the
|
||||
existing `WidgetDataResponse` + `useWidgetData` polling. The series shape
|
||||
(`{series:[{label,points:[{t,v}]}]}`) is already used by `PrometheusChartWidget`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Tests
|
||||
|
||||
### 5.1 Backend tests
|
||||
|
||||
| Test file | Coverage |
|
||||
|---|---|
|
||||
| `backend/tests/test_service_data.py` (NEW) | Harness: register, run_migrations (creates tables), cascade_delete (removes rows by service_id), migration idempotency (ALTER doesn't crash on re-run) |
|
||||
| `backend/tests/test_qbittorrent_store.py` (NEW) | Store: append + prune (MAX_SAMPLES cap), window (returns samples in order), service_id isolation (two services don't cross-contaminate) |
|
||||
| `backend/tests/test_qbittorrent_client.py` (NEW) | Client: login flow (POST /auth/login → Ok.), cookie reuse, 403 → re-login, maindata parsing, timeout handling |
|
||||
| `backend/tests/test_widgets.py` (extend) | `QbittorrentWidgetSource`: totals (counts all torrents), active (filters state), speed (appends sample + returns {series}), missing-service error, timeout error |
|
||||
|
||||
### 5.2 Frontend tests
|
||||
|
||||
| Test file | Coverage |
|
||||
|---|---|
|
||||
| `QbittorrentTotalsWidget.test.tsx` (NEW) | Loading skeleton, error alert, rendered count + state badges |
|
||||
| `QbittorrentActiveTorrentsWidget.test.tsx` (NEW) | Loading, error, rendered torrent rows |
|
||||
| `QbittorrentSpeedWidget.test.tsx` (NEW) | Loading, error, rendered chart (series present) |
|
||||
| `LineSeriesChart.test.tsx` (NEW) | Renders lines from series data, empty state |
|
||||
|
||||
### 5.3 Existing MediaIndex tests must stay green
|
||||
|
||||
`backend/tests/test_media_index.py` exercises `replace_items`, `query`, `status`.
|
||||
After migration, these call with the new `service_id=""` default (backward-compatible).
|
||||
The tests pass unchanged because empty-string service_id shows all rows.
|
||||
|
||||
---
|
||||
|
||||
## 6. Slice plan (for tasks.md)
|
||||
|
||||
Four slices, each ≤400 changed lines, each leaving `pytest` + `npm run build` +
|
||||
`npm run lint` green. Slices 1–2 prove the harness; Slice 3 migrates MediaIndex;
|
||||
Slice 4 wires cascade-delete end-to-end.
|
||||
|
||||
### Slice 1: Harness + QbittorrentSampleStore + QbittorrentClient + integration (~280–350 lines)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `backend/src/media_library_viewer_api/services/service_data.py` (NEW — harness + StorageConcern)
|
||||
- `backend/src/media_library_viewer_api/services/qbittorrent_store.py` (NEW — store + concern)
|
||||
- `backend/src/media_library_viewer_api/clients/qbittorrent.py` (NEW — client)
|
||||
- `backend/src/media_library_viewer_api/integrations/qbittorrent.py` (NEW — definition)
|
||||
- `backend/src/media_library_viewer_api/integrations/registry.py` (MODIFY — add qbittorrent)
|
||||
- `backend/src/media_library_viewer_api/main.py` (MODIFY — call harness init in lifespan)
|
||||
- `backend/tests/test_service_data.py` (NEW)
|
||||
- `backend/tests/test_qbittorrent_store.py` (NEW)
|
||||
- `backend/tests/test_qbittorrent_client.py` (NEW)
|
||||
|
||||
**Exit gate:** harness creates tables + runs migrations + cascade_delete works in
|
||||
tests. qBit client login/maindata tested with mocked HTTP. No frontend changes yet.
|
||||
|
||||
### Slice 2: Widget adapter + frontend widgets + registry binding (~320–400 lines)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `backend/src/media_library_viewer_api/widgets/sources.py` (MODIFY — add QbittorrentWidgetSource + SERVICE_ADAPTERS entry)
|
||||
- `backend/tests/test_widgets.py` (EXTEND — qBit adapter tests)
|
||||
- `frontend/src/components/LineSeriesChart.tsx` (NEW — shared renderer extracted from PrometheusChartWidget)
|
||||
- `frontend/src/widgets/PrometheusChartWidget.tsx` (MODIFY — use LineSeriesChart)
|
||||
- `frontend/src/widgets/QbittorrentTotalsWidget.tsx` (NEW)
|
||||
- `frontend/src/widgets/QbittorrentActiveTorrentsWidget.tsx` (NEW)
|
||||
- `frontend/src/widgets/QbittorrentSpeedWidget.tsx` (NEW)
|
||||
- `frontend/src/widgets/index.ts` (MODIFY — barrel exports)
|
||||
- `frontend/src/integrations/registry.ts` (MODIFY — add qbittorrent binding)
|
||||
- `frontend/src/widgets/__tests__/QbittorrentTotalsWidget.test.tsx` (NEW)
|
||||
- `frontend/src/widgets/__tests__/QbittorrentActiveTorrentsWidget.test.tsx` (NEW)
|
||||
- `frontend/src/widgets/__tests__/QbittorrentSpeedWidget.test.tsx` (NEW)
|
||||
- `frontend/src/components/__tests__/LineSeriesChart.test.tsx` (NEW)
|
||||
|
||||
**Exit gate:** qBit widgets render loading/error/data states; speed widget shows
|
||||
a recharts line chart from {series}; LineSeriesChart is shared with PrometheusChartWidget.
|
||||
|
||||
### Slice 3: MediaIndex migration onto harness (~200–280 lines)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `backend/src/media_library_viewer_api/services/service_data.py` (MODIFY — register MEDIA_INDEX_CONCERN)
|
||||
- `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY — add service_id to schema, replace_items, query)
|
||||
- `backend/src/media_library_viewer_api/services/media_index.py` (re-export unchanged)
|
||||
- `backend/src/media_library_viewer_api/workers/media_index_worker.py` (MODIFY — pass service_id to build_media_index)
|
||||
- `backend/src/media_library_viewer_api/routers/media.py` (MODIFY — thread jellyfin_service_id into query_media)
|
||||
- `backend/tests/test_media_index.py` (EXTEND — service_id-scoped replace_items + query)
|
||||
|
||||
**Exit gate:** all existing MediaIndex tests pass unchanged (empty-string default);
|
||||
new tests verify scoped delete/insert/query. Media page works identically.
|
||||
|
||||
### Slice 4: Cascade-delete end-to-end + integration test (~80–120 lines)
|
||||
|
||||
**Files:**
|
||||
|
||||
- `backend/src/media_library_viewer_api/services/settings_store.py` (MODIFY — call harness.cascade_delete in delete_service)
|
||||
- `backend/tests/test_services.py` (EXTEND — verify cascade-delete removes qBit samples when service is deleted)
|
||||
|
||||
**Exit gate:** deleting a qBittorrent service removes its speed samples; deleting
|
||||
a Jellyfin service removes its media items.
|
||||
|
||||
---
|
||||
|
||||
## 7. Key design decisions summary
|
||||
|
||||
| # | Decision | Rationale |
|
||||
|---|---|---|
|
||||
| D1 | Harness is lifecycle-only (migrations, service_id, cascade) | n=2 justifies abstraction, but data-ops differ wildly (append/window vs replace_all/query). Generalize only the shared lifecycle. |
|
||||
| D2 | Per-concern DB files via harness `base_dir` + `db_filename` | media_index.sqlite stays put (no data move); qbittorrent.db is new. Separate writers. |
|
||||
| D3 | `run_migrations` catches "duplicate column name" per-statement | SQLite ALTER TABLE ADD COLUMN is not idempotent; migrations must not crash on re-run. |
|
||||
| D4 | Speed widget extracts a shared `LineSeriesChart` component | Both PrometheusChartWidget and QbittorrentSpeedWidget need the same recharts renderer; extract it once (DRY) rather than duplicating or tightly coupling. |
|
||||
| D5 | qBit adapter resolves client inline from ServiceRecord (like PrometheusWidgetSource) | No new dependency-injection helper; the adapter gets config+secrets from the service record. |
|
||||
| D6 | MediaIndex backfill uses `service_id = ''` (empty string) sentinel | Backward-compatible: existing tests + Media page work unchanged; empty-string queries skip the filter (show all). Real service_ids overwrite on next rebuild. |
|
||||
| D7 | `replace_items` deletes `WHERE service_id = ?` instead of all | Fixes the existing global-clear bug (building for one Jellyfin wipes others). |
|
||||
| D8 | Cascade-delete is best-effort (try/except) in delete_service | Harness failure must not block service deletion; data cleanup is non-critical. |
|
||||
| D9 | Speed sample timestamps multiplied by 1000 for frontend | PrometheusChartWidget expects `t` in milliseconds (JS epoch); SQLite stores Unix seconds. |
|
||||
| D10 | MAX_SAMPLES = 120 cap with per-append prune | ~2 min history at 1s poll; bounded DB growth; single-statement prune. |
|
||||
|
||||
---
|
||||
|
||||
## 8. Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| **MediaIndex migration breaks existing tests.** | `service_id` defaults to `""`; empty-string queries skip the filter → all rows visible. Existing tests pass unchanged. |
|
||||
| **ALTER TABLE fails on fresh installs** where init_schema already created the column. | `run_migrations` catches "duplicate column name" errors per-statement. |
|
||||
| **Harness lazy-init hides migration failures.** | `run_migrations` runs on first access (startup lifespan); failures raise (not swallowed) except the expected duplicate-column case. |
|
||||
| **Speed widget poll interval (5s) too aggressive.** | 5s is the default; configurable via widget config if needed. MAX_SAMPLES cap bounds storage. |
|
||||
| **qBit cookie expiry between polls.** | Client re-logins on 403 transparently; persistent auth failure surfaces as widget error state. |
|
||||
| **LineSeriesChart extraction breaks PrometheusChartWidget.** | Slice 2 includes tests for both; extraction is mechanical (move JSX + helpers, pass series as prop). |
|
||||
| **Review budget (>400 lines).** | Four slices, each ≤400 lines. Slices 1–2 are independently shippable (qBit works without MediaIndex migration). |
|
||||
|
||||
---
|
||||
|
||||
## 9. Data flow diagrams
|
||||
|
||||
### 9.1 Speed widget data flow (InService path)
|
||||
|
||||
```
|
||||
Dashboard poll (useWidgetData, 5s)
|
||||
└► GET /api/widgets/instances/{id}/data
|
||||
└► QbittorrentWidgetSource.fetch(service, "speed", {})
|
||||
├► QbittorrentClient.maindata()
|
||||
│ └► /api/v2/sync/maindata → {server_state:{dl_info_speed, up_info_speed}}
|
||||
├► QbittorrentSampleStore.append(service.id, ts, dl, up)
|
||||
│ └► INSERT + prune (keep 120)
|
||||
├► QbittorrentSampleStore.window(service.id)
|
||||
│ └► SELECT ts, dl_speed, up_speed → [{ts, dl_speed, up_speed}]
|
||||
└► return {series: [{label:"download", points:[{t,v}]}, {label:"upload", points:[{t,v}]}]}
|
||||
└► Frontend: QbittorrentSpeedWidget → <LineSeriesChart series={...} />
|
||||
```
|
||||
|
||||
### 9.2 Cascade-delete data flow
|
||||
|
||||
```
|
||||
DELETE /api/services/instances/{id}
|
||||
└► SettingsStore.delete_service(id)
|
||||
├► DELETE FROM dashboard_widgets WHERE service_id = ?
|
||||
├► DELETE FROM services WHERE id = ?
|
||||
└► ServiceDataHarness.cascade_delete(id)
|
||||
├► DELETE FROM qbittorrent_speed_samples WHERE service_id = ?
|
||||
└► DELETE FROM media_items WHERE service_id = ?
|
||||
```
|
||||
Reference in New Issue
Block a user