Compare commits

...

21 Commits

Author SHA1 Message Date
Developer 5cb5e79032 chore(service-storage-harness): archive verified+synced change
Move to openspec/changes/archive/2026-07-09-service-storage-harness/
(history preserved via rename detection). 9 artifacts: proposal/spec/design/
tasks/apply-progress/verify-report/sync-report/archive-report + delta spec.
Canonical openspec/specs/service-storage/ remains. Native status engine
discrepancy (ambiguous change selection) disregarded per parent verification.
2026-07-09 09:47:05 +00:00
Developer c9201a004c spec(service-storage-harness): sync into canonical service-storage domain
New canonical openspec/specs/service-storage/spec.md (28 reqs SS-101..128).
Change-side delta + sync-report. web-ui + prometheus-charting canonicals
untouched.
2026-07-09 09:37:22 +00:00
Developer 40a7ac80d3 spec(service-storage-harness): verify + reconcile tracking
Write apply-progress.md, tick all 35 tasks, add verify-report.md (28/28
SS-101..128 PASS). Gates green: 322 pytest, ruff clean, FE build+lint 0
errors, PrometheusChartWidget extraction 4/4 non-regressive. No blocking
code findings. Process note: slice 2 was +644 lines over 400-line budget
(additive, no scope creep; retrospectively extract+widgets could split).
2026-07-09 09:30:52 +00:00
Developer c9404f0794 spec(service-storage-harness): add spec (28 requirements SS-101..128)
Covers harness lifecycle, QbittorrentSampleStore, QbittorrentClient,
3 widget kinds (totals=item count, active=DL/UL, speed=LineSeriesChart
reuse), MediaIndex migration (+service_id, scoped replace_items bug fix,
backward-compat query), cascade-delete wiring, test/build greenness.
2026-07-09 09:15:32 +00:00
Developer 75c949ad25 feat(service-storage-harness): slice 4 — cascade-delete wiring + integration test
Wire ServiceDataHarness.cascade_delete into SettingsStore.delete_service
(best-effort try/except, logs on failure). Fix migration runner to also
catch 'no such table' on fresh DBs (ALTER TABLE before init_schema).
Integration test proves end-to-end cascade across both concerns (qBit
samples + media items) with multi-instance preservation.

Backend: 322 pytest pass, ruff clean.
2026-07-09 09:11:42 +00:00
Developer c87f398e37 feat(service-storage-harness): slice 3 — migrate MediaIndex onto harness (scoped replace_items, +service_id)
Register MediaIndex as a harness concern with ALTER TABLE migration to add
service_id column (idempotent). Scope replace_items by service_id (FIXES latent
global-clear bug where building for one Jellyfin wiped another's rows). Scope
query by service_id (empty-string = all rows, backward-compat). Thread
service_id through build_media_index + worker + query_media router. New
regression test proves scoped replace preserves other services' rows.

Backend: 321 pytest pass, ruff clean. Frontend: build green.
2026-07-09 09:00:31 +00:00
Developer 1fb12b8a0a feat(service-storage-harness): slice 2 — qbit widgets + LineSeriesChart extract 2026-07-09 08:49:20 +00:00
Developer e7bd0afdd1 feat(service-storage-harness): slice 1 — harness + qbit store + client + integration
ServiceDataHarness (services/service_data.py): lifecycle-only registry of
per-concern storage — DB provisioning, idempotent migrations (ALTER TABLE
duplicate-column-name caught per-statement), cascade_delete(service_id).
QbittorrentSampleStore: append/window/prune (MAX_SAMPLES=120) in
qbittorrent.db. QbittorrentClient: cookie-login Web API client (403 re-login,
/sync/maindata). Integration registered with 3 widget kinds (totals/active/
speed). Harness initialized in main.py lifespan.

Backend: 314 pytest pass (21 new), ruff clean.
2026-07-09 08:24:08 +00:00
Developer 9a251db23c spec(service-storage-harness): add tasks (4 slices, each <=400 lines)
S1 harness+store+client+integration; S2 widget adapter+3 FE widgets+
LineSeriesChart extract; S3 MediaIndex +service_id migration (fixes latent
global-clear bug); S4 cascade-delete wiring. Each slice leaves pytest/npm
build/npm lint green. Risk flags on LineSeriesChart extract + MediaIndex migration.
2026-07-09 08:02:31 +00:00
Developer cff082c7a1 spec(service-storage-harness): add design
10 design decisions incl: lifecycle-only harness, per-concern DB files,
QbittorrentSampleStore (120-sample cap), qBit cookie client, 3 widget kinds
(totals=item count, active=DL/UL state, speed=reuses Change A renderer via
shared LineSeriesChart extract), MediaIndex +service_id backfill, scoped
replace_items (fixes latent global-clear bug), best-effort cascade-delete.
4-slice plan. 3 source findings (worker already threads service_id).
2026-07-09 07:55:59 +00:00
Developer bc389ae3f7 spec(service-storage-harness): reconcile proposal post Change A + question round
Resolve Q1 (reuse Change A's PrometheusChartWidget renderer via InService
data path), Q2 (totals = item count, not transfer bytes), Q3 (active =
downloading/uploading), Q4 (N instances), Q5 (username/password cookie
auth). Remove TBD/stale thin-dashboard caveats.
2026-07-09 07:45:55 +00:00
Developer 7efc06a629 chore(prometheus-direct-charting): archive verified+synced change
Move to openspec/changes/archive/2026-07-08-prometheus-direct-charting/
(git mv, history preserved). 9 artifacts: proposal/spec/design/tasks/
apply-progress/verify-report/sync-report/archive-report + delta spec.
Canonical openspec/specs/prometheus-charting/ remains.
2026-07-08 23:01:35 +00:00
Developer 3e77075171 spec(prometheus-direct-charting): sync into canonical prometheus-charting domain
New canonical domain openspec/specs/prometheus-charting/spec.md with all
27 requirements (SC-101..127) as the durable post-change contract. Change-side
delta specs/prometheus-charting/spec.md + sync-report.md. web-ui canonical
untouched (different concern).
2026-07-08 22:57:29 +00:00
Developer 7440603cdb spec(prometheus-direct-charting): verify + close SC-125 + reconcile tracking
Add loading-state tests to the three Prometheus widget test files
(closes SC-125 PARTIAL). Write apply-progress.md, tick all 39 tasks,
add verify-report.md (26/27 PASS, 1 PARTIAL->PASS). All gates green:
293 pytest, ruff clean, npm build+lint 0 errors. No blocking findings.
2026-07-08 22:50:30 +00:00
Developer 67ca0fc3bc feat(prometheus-direct-charting): slice 3 — remove grafana + config rewrite + changelog
Remove the entire Grafana surface: integrations/grafana.py, GrafanaWidgetSource
(+ _fetch_chart, now redundant since prometheus chart exists), GrafanaLinkWidget,
LinksTab, get_grafana_status endpoint, useGrafanaStatus hook, GrafanaStatus type,
fetchGrafanaStatus client fn, registry/nav/tab entries (FE+BE). Rewrite
config.yaml thin-dashboard rule to match reality (recharts is sanctioned for
Prometheus-backed series). CHANGELOG migration note added.

Backend: 293 pytest pass, ruff clean. Frontend: build+lint green (0 errors).
SC-115/116 grep-clean (only prometheus_range.py migration comments + Dashboard.test
shortcut fixture remain — both spec-allowed).
2026-07-08 22:35:06 +00:00
Developer 65bae95e3c feat(prometheus-direct-charting): slice 2 — gauge + mean widgets
Add gauge widget (recharts RadialBarChart with configurable threshold
bands, scalar-only per SC-111) and mean widget (client-side average over
range-query window, scalar-only per SC-114). Extract shared _instant_query
helper from the metric path; _fetch_gauge and _fetch_mean dispatch in
PrometheusWidgetSource.fetch(). Both new widget kinds declared in
integrations/prometheus.py and frontend registry.

Backend: 305 pytest pass, ruff clean. Frontend: 136 vitest pass, build+lint green.
2026-07-08 22:10:11 +00:00
Developer 5dad98231f feat(prometheus-direct-charting): slice 1 — prom range query + chart rebrand
Add PrometheusWidgetSource._fetch_chart hitting /api/v1/query_range directly
(SC-101..104). New shared helpers in widgets/prometheus_range.py:
step_for_window (window preset -> step, ~200pts) and normalize_prometheus_matrix
(extracted label/dedup rule, retargeted at Prom matrix, robust to malformed
data). Chart widget kind moved grafana->prometheus in both registries;
GrafanaChartWidget renamed -> PrometheusChartWidget (git mv, recharts body
preserved). Grafana binding/service untouched (removed in slice 3).

Backend: 298 pytest pass, ruff clean. Frontend: 128 vitest pass, build+lint green.
2026-07-08 21:55:02 +00:00
Developer d906b0392b spec(prometheus-direct-charting): add tasks (3 slices, each <=400 lines)
S1 adds Prom chart (range query + rebrand, no grafana removal); S2 adds
gauge+mean; S3 removes grafana + config rewrite + changelog. Each slice
leaves pytest/npm build/npm lint green. Review-workload forecast per slice.
2026-07-08 21:33:16 +00:00
Developer 9b7415080b spec(prometheus-direct-charting): patch SC-116/118 for service-tabs refactor
ObservabilityPage.tsx was refactored into service-tabs/; update removal
criteria to name real targets and whitelist the Dashboard.test fixture
shortcut-label collision.
2026-07-08 21:29:09 +00:00
Developer 78e273efe8 spec(prometheus-direct-charting): add design
5 locked design decisions: step derivation formula (max(15, round(s/200))),
shared normalize_prometheus_matrix helper in widgets/prometheus_range.py,
recharts RadialBarChart gauge w/ threshold bands, mean via client-side
avg over query_range, 3-slice plan each <=400 lines.
Source-verified: ObservabilityPage refactored into service-tabs/ (map stale).
2026-07-08 21:28:02 +00:00
Developer b7e5ca3cbc spec(prometheus-direct-charting): add proposal + spec
Drop Grafana as chart middleman; query Prometheus directly via
/api/v1/query_range. Rebrand GrafanaChartWidget -> PrometheusChartWidget,
add gauge + mean widget kinds, remove Grafana surface, rewrite stale
thin-dashboard rule in config.yaml. 27 acceptance requirements (SC-101..127).
2026-07-08 21:21:14 +00:00
75 changed files with 7526 additions and 1079 deletions
+18
View File
@@ -4,6 +4,24 @@ All notable changes to Manage. Breaking changes are marked with **BREAKING**.
## [Unreleased]
### Added — Direct Prometheus charting
- **Prometheus is now the direct source for in-app charts.** New widget kinds
on the `prometheus` service: `chart` (multi-series line chart via recharts,
backed by `/api/v1/query_range`), `gauge` (instant scalar with configurable
threshold bands), and `mean` (client-side average over a time window).
### **BREAKING** — Grafana service type removed
- The `grafana` service type, Grafana link widget, Grafana chart widget, and
`GET /api/monitoring/grafana-status` endpoint were **removed**. Manage now
queries Prometheus directly for all chart data.
- **Migration:** Delete any existing Grafana service instances and create
Prometheus service instances instead (pointing at your Prometheus URL). Any
configured `grafana/chart` widgets must be recreated as `prometheus/chart`
widgets. Grafana link widgets are gone — use Prometheus chart/metric widgets
instead.
### Added — Observability service registry
- **Alertmanager is now a service type.** Configure Alertmanager, Grafana, and
@@ -0,0 +1,80 @@
"""Minimal qBittorrent Web API client (read-only: sync/maindata only).
Modeled on :class:`~media_library_viewer_api.clients.jellyfin.JellyfinClient`'s
session pattern. Authentication uses username/password login which stores an
SID cookie in the requests session. The client re-logins transparently on 403.
"""
from __future__ import annotations
import logging
from typing import Any
import requests
logger = logging.getLogger(__name__)
class QbittorrentClient:
"""Small wrapper around the qBittorrent Web API.
Only the endpoints needed by the dashboard widgets are implemented
(currently just ``/sync/maindata``). All calls share a single
:class:`requests.Session` that carries the login cookie.
"""
def __init__(self, base_url: str, username: str, password: str, timeout: int = 10) -> None:
if not base_url:
raise ValueError("qBittorrent base_url is required")
if not username:
raise ValueError("qBittorrent username is required")
self.base_url = base_url.rstrip("/")
if not self.base_url.endswith("/api/v2"):
self.base_url += "/api/v2"
self._username = username
self._password = password
self.timeout = timeout
self._session = requests.Session()
self._logged_in = False
def _login(self) -> None:
"""POST username/password to ``/auth/login``; store the SID cookie.
qBittorrent returns the plain text ``"Ok."`` on success. The
``Referer`` header is required by some qBittorrent CSRF protections.
"""
resp = self._session.post(
f"{self.base_url}/auth/login",
data={"username": self._username, "password": self._password},
timeout=self.timeout,
headers={"Referer": self.base_url},
)
resp.raise_for_status()
if resp.text.strip() != "Ok.":
raise RuntimeError(f"qBittorrent login failed: {resp.text.strip()}")
self._logged_in = True
logger.info("qBittorrent login successful for %s", self.base_url)
def _get(self, path: str, **params: Any) -> dict[str, Any]:
"""GET an endpoint with auto-login on first call and re-login on 403."""
if not self._logged_in:
self._login()
url = f"{self.base_url}{path}"
resp = self._session.get(url, params=params, timeout=self.timeout)
if resp.status_code == 403:
logger.debug("qBittorrent 403 on %s, re-logging in", path)
self._logged_in = False
self._login()
resp = self._session.get(url, params=params, timeout=self.timeout)
resp.raise_for_status()
return resp.json()
def maindata(self) -> dict[str, Any]:
"""Fetch ``/sync/maindata``.
Returns a dict with ``server_state`` (containing ``dl_info_speed``,
``up_info_speed``, etc.) and ``torrents`` (a dict of
``{hash: {name, state, progress, size, dlspeed, upspeed, ...}}``).
"""
return self._get("/sync/maindata")
@@ -1,7 +1,7 @@
"""Base classes for service integrations.
A *service definition* is a closed, compile-time description of an external service
the app can talk to (Grafana, Jellyfin, …). Each definition declares:
the app can talk to (Jellyfin, Prometheus, …). Each definition declares:
* its non-secret ``config_schema`` (derived from a Pydantic model),
* the secret fields it accepts (API keys / tokens),
@@ -24,7 +24,7 @@ from pydantic import BaseModel, BeforeValidator, Field
def _validate_service_base_url(value: Any) -> str:
"""Require an absolute http(s) URL for service ``base_url`` fields.
Relative hosts (e.g. ``grafana.example.com``) break downstream HTTP clients
Relative hosts (e.g. ``example.com``) break downstream HTTP clients
because ``requests`` treats them as relative paths, so we fail fast with a
clear error instead of letting the call silently malfunction.
"""
@@ -1,73 +0,0 @@
"""Grafana service definition."""
from __future__ import annotations
from media_library_viewer_api.integrations.base import (
SecretField,
ServiceBaseUrl,
ServiceConfigBase,
ServiceDefinition,
WidgetConfigBase,
widget_kind,
)
class GrafanaConfig(ServiceConfigBase):
"""Non-secret Grafana connection config."""
base_url: ServiceBaseUrl
timeout_seconds: int = 5
class GrafanaLinkWidgetConfig(WidgetConfigBase):
"""Deep-link to a Grafana dashboard or panel."""
dashboard_uid: str
panel_id: int | None = None
class GrafanaChartWidgetConfig(WidgetConfigBase):
"""Render a time-series chart from a Grafana datasource query."""
datasource_uid: str = "prometheus"
query: str = ""
from_ts: str = "now-1h"
to_ts: str = "now"
interval_ms: int = 30_000
max_data_points: int = 100
DEFINITION = ServiceDefinition(
service_type="grafana",
name="Grafana",
description="Dashboards, metrics, and logs.",
config_model=GrafanaConfig,
secret_fields=[
SecretField(key="api_key", label="API key", helper="Service account token (optional)"),
],
widget_kinds=[
widget_kind(
kind="link",
name="Dashboard link",
description="Deep-link to a Grafana dashboard or panel.",
model_cls=GrafanaLinkWidgetConfig,
default_config={"dashboard_uid": ""},
refresh_interval_ms=0,
),
widget_kind(
kind="chart",
name="Chart",
description="Live time-series chart from a Grafana datasource query.",
model_cls=GrafanaChartWidgetConfig,
default_config={
"datasource_uid": "prometheus",
"query": "",
"from_ts": "now-1h",
"to_ts": "now",
"interval_ms": 30_000,
"max_data_points": 100,
},
refresh_interval_ms=60_000,
),
],
)
@@ -25,6 +25,32 @@ class PrometheusMetricWidgetConfig(WidgetConfigBase):
promql: str
class PrometheusChartWidgetConfig(WidgetConfigBase):
"""A PromQL range query rendered as a multi-series line chart (SC-101..SC-104)."""
promql: str
window: str = "1h" # one of 1h / 6h / 24h / 7d (see WINDOW_PRESETS)
class PrometheusGaugeWidgetConfig(WidgetConfigBase):
"""A PromQL instant query rendered as a gauge with optional threshold bands (SC-109..SC-111)."""
promql: str
warn_at: float | None = None
crit_at: float | None = None
min: float | None = None
max: float | None = None
unit: str | None = None
class PrometheusMeanWidgetConfig(WidgetConfigBase):
"""A PromQL range query averaged client-side into a single value (SC-112..SC-114)."""
promql: str
window: str = "1h" # one of 1h / 6h / 24h / 7d (see WINDOW_PRESETS)
unit: str | None = None
DEFINITION = ServiceDefinition(
service_type="prometheus",
name="Prometheus",
@@ -42,5 +68,29 @@ DEFINITION = ServiceDefinition(
default_config={"promql": ""},
refresh_interval_ms=30_000,
),
widget_kind(
kind="chart",
name="Chart",
description="Multi-series line chart from a PromQL range query.",
model_cls=PrometheusChartWidgetConfig,
default_config={"promql": "", "window": "1h"},
refresh_interval_ms=60_000,
),
widget_kind(
kind="gauge",
name="Gauge",
description="Instant query rendered as a gauge with optional threshold bands.",
model_cls=PrometheusGaugeWidgetConfig,
default_config={"promql": ""},
refresh_interval_ms=30_000,
),
widget_kind(
kind="mean",
name="Mean",
description="Average value of a PromQL query over a time window.",
model_cls=PrometheusMeanWidgetConfig,
default_config={"promql": "", "window": "1h"},
refresh_interval_ms=60_000,
),
],
)
@@ -0,0 +1,68 @@
"""qBittorrent service definition.
Declares the config model (base URL + timeout), secret fields (username +
password), and three widget kinds (totals, active, speed). Models on
:mod:`media_library_viewer_api.integrations.prometheus`.
"""
from __future__ import annotations
from media_library_viewer_api.integrations.base import (
SecretField,
ServiceBaseUrl,
ServiceConfigBase,
ServiceDefinition,
WidgetConfigBase,
widget_kind,
)
class QbittorrentConfig(ServiceConfigBase):
"""Non-secret qBittorrent connection config."""
base_url: ServiceBaseUrl
timeout_seconds: int = 10
class QbittorrentWidgetConfig(WidgetConfigBase):
"""Per-widget config (empty — all three kinds derive from the service connection)."""
pass
DEFINITION = ServiceDefinition(
service_type="qbittorrent",
name="qBittorrent",
description="Torrent client activity, speeds, and item counts.",
config_model=QbittorrentConfig,
secret_fields=[
SecretField(key="username", label="Username", required=True),
SecretField(key="password", label="Password", required=True, helper="Stored encrypted"),
],
widget_kinds=[
widget_kind(
kind="totals",
name="Totals",
description="Count of all listed torrents, broken down by state.",
model_cls=QbittorrentWidgetConfig,
default_config={},
refresh_interval_ms=30_000,
),
widget_kind(
kind="active",
name="Active torrents",
description="Torrents currently downloading or uploading.",
model_cls=QbittorrentWidgetConfig,
default_config={},
refresh_interval_ms=15_000,
),
widget_kind(
kind="speed",
name="Speed chart",
description="Live download/upload speed over a short window.",
model_cls=QbittorrentWidgetConfig,
default_config={},
refresh_interval_ms=5_000,
),
],
)
@@ -10,18 +10,18 @@ from media_library_viewer_api.integrations.alertmanager import DEFINITION as ALE
from media_library_viewer_api.integrations.authentik import DEFINITION as AUTHENTIK
from media_library_viewer_api.integrations.backups import DEFINITION as BACKUPS
from media_library_viewer_api.integrations.base import ServiceDefinition, WidgetKind
from media_library_viewer_api.integrations.grafana import DEFINITION as GRAFANA
from media_library_viewer_api.integrations.jellyfin import DEFINITION as JELLYFIN
from media_library_viewer_api.integrations.nextcloud import DEFINITION as NEXTCLOUD
from media_library_viewer_api.integrations.prometheus import DEFINITION as PROMETHEUS
from media_library_viewer_api.integrations.qbittorrent import DEFINITION as QBITTORRENT
from media_library_viewer_api.integrations.ssh_tasks import DEFINITION as SSH_TASKS
SERVICE_DEFINITIONS: dict[str, ServiceDefinition] = {
GRAFANA.service_type: GRAFANA,
PROMETHEUS.service_type: PROMETHEUS,
ALERTMANAGER.service_type: ALERTMANAGER,
JELLYFIN.service_type: JELLYFIN,
NEXTCLOUD.service_type: NEXTCLOUD,
QBITTORRENT.service_type: QBITTORRENT,
SSH_TASKS.service_type: SSH_TASKS,
BACKUPS.service_type: BACKUPS,
AUTHENTIK.service_type: AUTHENTIK,
@@ -52,6 +52,12 @@ async def lifespan(app: FastAPI):
get_settings_store().ensure_defaults()
except Exception:
logger.exception("Failed to seed default settings during startup")
try:
from media_library_viewer_api.services.service_data import get_service_data_harness
get_service_data_harness()
except Exception:
logger.exception("Failed to initialize service data harness during startup")
mail_queue = get_mail_queue()
backup_poller = get_backup_poller()
mail_queue.start()
@@ -2,7 +2,7 @@
Widgets are either:
* **service-bound** — reference a ``service_id`` and a ``widget_kind`` declared
by that service's definition (Grafana link, Prometheus metric, Jellyfin
by that service's definition (Prometheus metric, Jellyfin
activity, SSH task output); or
* **built-in** — ``service_id`` is null and ``widget_kind`` is one of the
service-less kinds (backups, static).
@@ -282,6 +282,7 @@ def query_media(
sort_order: str = Query("Ascending", description="Ascending or Descending"),
limit: int = Query(100, ge=1, le=1000),
offset: int = Query(0, ge=0),
jellyfin_service_id: str | None = None,
client: JellyfinClient = Depends(get_jellyfin_client),
user_id: str = Depends(get_user_id),
index: MediaIndex = Depends(get_media_index),
@@ -316,6 +317,7 @@ def query_media(
sort_order=sort_order,
limit=limit,
offset=offset,
service_id=jellyfin_service_id or "",
)
logger.info("Media query returned total=%s rows=%s", total, len(rows))
@@ -1,6 +1,6 @@
"""Monitoring router — observability service status.
Observability components (Alertmanager, Grafana, Prometheus) are resolved from
Observability components (Alertmanager, Prometheus) are resolved from
the service registry, not environment variables. The endpoints pick the first
enabled instance of a type when no ``service_id`` is given, and return graceful
"not configured" / "unreachable" payloads so the UI always renders a health card.
@@ -172,29 +172,6 @@ def get_alertmanager_status(
}
@router.get("/grafana-status")
def get_grafana_status(
service_id: str | None = None,
store: SettingsStore = Depends(get_settings_store),
) -> dict[str, Any]:
"""Probe a Grafana service instance's ``/api/health`` endpoint."""
service = resolve_service_record(store, "grafana", service_id)
if service is None:
return _status_response(None, error="no_service_configured")
try:
response = requests.get(
f"{_base_url(service)}/api/health",
headers=_auth_headers(service),
timeout=_timeout(service, 5),
)
response.raise_for_status()
data = response.json()
except Exception:
logger.exception("Failed to fetch Grafana status")
return _status_response(service, error="grafana_unreachable")
return _status_response(service, version=data.get("version", ""))
@router.get("/prometheus-status")
def get_prometheus_status(
service_id: str | None = None,
@@ -17,6 +17,7 @@ from typing import Any, Callable, Iterable
from media_library_viewer_api.clients.jellyfin import JellyfinClient
from media_library_viewer_api.domain.media import display_media_row, normalize_media_item
from media_library_viewer_api.path_utils import resolve_remote_media_path
from media_library_viewer_api.services.service_data import StorageConcern
logger = logging.getLogger(__name__)
@@ -25,6 +26,20 @@ logger = logging.getLogger(__name__)
DEFAULT_INDEX_PATH = Path(".cache/media_library_viewer/media_index.sqlite")
MEDIA_TYPES = "Movie,Episode,Video"
# Harness concern registration: the media_index DB is scoped by service_id so
# multiple Jellyfin instances can coexist. The ALTER TABLE migration adds the
# service_id column to existing DBs; init_schema adds it for fresh installs.
# The harness run_migrations catches "duplicate column name" on re-runs.
MEDIA_INDEX_CONCERN = StorageConcern(
concern_key="media_index",
db_filename=DEFAULT_INDEX_PATH.name,
migrations=[
"ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT ''",
],
tables=["media_items"],
service_id_column="service_id",
)
# Only values from this whitelist are interpolated into ORDER BY. User-selected
# sort keys map to these known SQL snippets to avoid SQL injection.
SORT_COLUMNS = {
@@ -135,7 +150,8 @@ class MediaIndex:
date_added_ts INTEGER,
path TEXT,
library_id TEXT,
library_name TEXT
library_name TEXT,
service_id TEXT NOT NULL DEFAULT ''
);
CREATE TABLE IF NOT EXISTS index_metadata (
key TEXT PRIMARY KEY,
@@ -167,8 +183,13 @@ class MediaIndex:
(key, str(value)),
)
def replace_items(self, rows: Iterable[dict[str, Any]]) -> int:
"""Atomically replace indexed media rows with a freshly built set."""
def replace_items(self, rows: Iterable[dict[str, Any]], service_id: str = "") -> int:
"""Atomically replace indexed media rows with a freshly built set.
Scoped by ``service_id``: only rows belonging to this service are
deleted before the new batch is inserted. This means building for one
Jellyfin instance no longer wipes another instance's rows.
"""
self.init_schema()
row_list = list(rows)
columns = [
@@ -194,13 +215,14 @@ class MediaIndex:
"path",
"library_id",
"library_name",
"service_id",
]
placeholders = ",".join(["?"] * len(columns))
with self.connect() as conn:
conn.execute("DELETE FROM media_items")
conn.execute("DELETE FROM media_items WHERE service_id = ?", (service_id,))
conn.executemany(
f"INSERT OR REPLACE INTO media_items ({','.join(columns)}) VALUES ({placeholders})",
[[row.get(column) for column in columns] for row in row_list],
[[row.get(column) if column != "service_id" else service_id for column in columns] for row in row_list],
)
conn.execute(
"INSERT OR REPLACE INTO index_metadata (key, value) VALUES ('updated_at', ?)",
@@ -287,8 +309,14 @@ class MediaIndex:
sort_order: str = "Ascending",
limit: int = 100,
offset: int = 0,
service_id: str = "",
) -> tuple[list[dict[str, Any]], int]:
"""Query indexed media with full-index filters, sorting, and pagination."""
"""Query indexed media with full-index filters, sorting, and pagination.
When ``service_id`` is non-empty, only rows matching that service are
returned. When empty (the default), all rows are returned (backward-
compatible with callers that are not multi-instance aware).
"""
self.init_schema()
where = []
params: list[Any] = []
@@ -309,6 +337,9 @@ class MediaIndex:
where.append("hdr = 1")
elif hdr_filter == "SDR/unknown only":
where.append("(hdr IS NULL OR hdr = 0)")
if service_id:
where.append("service_id = ?")
params.append(service_id)
where_sql = " WHERE " + " AND ".join(where) if where else ""
sort_sql = SORT_COLUMNS.get(sort_key, SORT_COLUMNS["title"])
@@ -338,6 +369,7 @@ def build_media_index(
fallback_prefix: str = "",
progress_callback: Callable[[dict[str, Any]], None] | None = None,
should_cancel: Callable[[], bool] | None = None,
service_id: str = "",
) -> int:
"""Fetch Jellyfin pages for all selected libraries and rebuild the index."""
index = index or MediaIndex()
@@ -453,7 +485,7 @@ def build_media_index(
logger.info("Media index finalizing rows=%s", len(normalized_rows))
emit("finalizing", "Writing index to disk")
ensure_not_cancelled()
count = index.replace_items(normalized_rows)
count = index.replace_items(normalized_rows, service_id=service_id)
duration = time.perf_counter() - started_at
index.set_metadata("build_duration_seconds", f"{duration:.3f}")
processed_total = count
@@ -0,0 +1,91 @@
"""Speed-sample storage for qBittorrent widgets.
This module defines the storage concern for qBittorrent speed data and a
bespoke store with ``append``/``window`` operations. It is registered with the
:class:`~media_library_viewer_api.services.service_data.ServiceDataHarness` as
the first real consumer of the harness lifecycle layer.
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING, Any
from media_library_viewer_api.services.service_data import StorageConcern
if TYPE_CHECKING:
from media_library_viewer_api.services.service_data import ServiceDataHarness
logger = logging.getLogger(__name__)
QBITTORRENT_CONCERN = StorageConcern(
concern_key="qbittorrent",
db_filename="qbittorrent.db",
migrations=[
"""
CREATE TABLE IF NOT EXISTS qbittorrent_speed_samples (
service_id TEXT NOT NULL,
ts INTEGER NOT NULL,
dl_speed INTEGER NOT NULL DEFAULT 0,
up_speed INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_qbit_samples_service_ts
ON qbittorrent_speed_samples(service_id, ts);
"""
],
tables=["qbittorrent_speed_samples"],
)
#: Maximum samples kept per service (~2 min at 1 s poll, ~4 min at 2 s poll).
MAX_SAMPLES = 120
class QbittorrentSampleStore:
"""Bespoke speed-sample store for qBittorrent widgets.
Each ``append`` inserts a new sample and prunes entries beyond
:data:`MAX_SAMPLES`, keeping only the most recent rows for the given
``service_id``.
"""
def __init__(self, harness: ServiceDataHarness | None = None) -> None:
if harness is None:
from media_library_viewer_api.services.service_data import get_service_data_harness
harness = get_service_data_harness()
self._harness = harness
def append(self, service_id: str, ts: int, dl_speed: int, up_speed: int) -> None:
"""Append a sample and prune old entries beyond ``MAX_SAMPLES``."""
with self._harness.connect("qbittorrent") as conn:
conn.execute(
"INSERT INTO qbittorrent_speed_samples (service_id, ts, dl_speed, up_speed) VALUES (?, ?, ?, ?)",
(service_id, ts, dl_speed, up_speed),
)
conn.execute(
"DELETE FROM qbittorrent_speed_samples "
"WHERE service_id = ? AND ts NOT IN ("
" SELECT ts FROM qbittorrent_speed_samples"
" WHERE service_id = ?"
" ORDER BY ts DESC LIMIT ?"
")",
(service_id, service_id, MAX_SAMPLES),
)
conn.commit()
def window(self, service_id: str, since_ts: int | None = None) -> list[dict[str, Any]]:
"""Return all samples for a service since a timestamp (or all if ``None``)."""
with self._harness.connect("qbittorrent") as conn:
if since_ts is not None:
rows = conn.execute(
"SELECT ts, dl_speed, up_speed FROM qbittorrent_speed_samples "
"WHERE service_id = ? AND ts >= ? ORDER BY ts ASC",
(service_id, since_ts),
).fetchall()
else:
rows = conn.execute(
"SELECT ts, dl_speed, up_speed FROM qbittorrent_speed_samples WHERE service_id = ? ORDER BY ts ASC",
(service_id,),
).fetchall()
return [{"ts": r[0], "dl_speed": r[1], "up_speed": r[2]} for r in rows]
@@ -0,0 +1,158 @@
"""Lifecycle-only storage harness for service-owned persistent data.
This module owns the *lifecycle* of per-concern SQLite databases: provisioning,
schema migrations, and cascade-delete when a service instance is removed. It
does **not** own data operations — each integration implements its own Store
with bespoke operations (``append``/``window``, ``replace_items``/``query``,
etc.). This keeps the general interface narrow (lifecycle) and the specific
interfaces rich (per-integration operations).
Each storage *concern* is registered with a :class:`StorageConcern` dataclass
declaring its DB filename, ordered migration statements, owned tables, and the
column used for service scoping.
"""
from __future__ import annotations
import logging
import os
import sqlite3
from dataclasses import dataclass, field
from pathlib import Path
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class StorageConcern:
"""A per-integration storage namespace registered with the harness."""
concern_key: str # e.g. "qbittorrent", "media_index"
db_filename: str # e.g. "qbittorrent.db", "media_index.sqlite"
migrations: list[str] # ordered CREATE/ALTER statements (idempotent via IF NOT EXISTS or ALTER-catch)
tables: list[str] = field(default_factory=list) # tables owned by this concern (for cascade)
service_id_column: str = "service_id"
class ServiceDataHarness:
"""Lifecycle-only registry of per-concern storage.
Owns:
* DB provisioning (per-concern SQLite files under ``base_dir``).
* Schema migrations (run on first access via :meth:`run_migrations`).
* ``service_id`` cascade-delete when a service instance is removed.
Does **not** own:
* Data operations — each store keeps bespoke append/window/query/etc.
* A generic value table or generic CRUD layer.
"""
def __init__(self, base_dir: Path | str) -> None:
self._base_dir = Path(base_dir)
self._concerns: dict[str, StorageConcern] = {}
@property
def base_dir(self) -> Path:
return self._base_dir
def register(self, concern: StorageConcern) -> None:
"""Register a storage concern. Called at startup / on first access."""
self._concerns[concern.concern_key] = concern
def db_path(self, concern_key: str) -> Path:
"""Return the absolute path to a concern's DB file."""
concern = self._concerns[concern_key]
return self._base_dir / concern.db_filename
def connect(self, concern_key: str) -> sqlite3.Connection:
"""Open a WAL-mode connection to a concern's DB."""
path = self.db_path(concern_key)
path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path, timeout=30)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=30000")
return conn
def run_migrations(self) -> None:
"""Run pending migrations for every registered concern.
Each migration string is split into individual statements (by ``;``)
and executed individually. ``ALTER TABLE ... ADD COLUMN`` statements
that fail with "duplicate column name" are silently skipped, making
migrations idempotent across re-runs and fresh installs where
``init_schema`` may have already created the column.
"""
for concern in self._concerns.values():
path = self.db_path(concern.concern_key)
path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path, timeout=30)
try:
conn.execute("PRAGMA journal_mode=WAL")
for migration_sql in concern.migrations:
statements = [s.strip() for s in migration_sql.split(";") if s.strip()]
for stmt in statements:
try:
conn.execute(stmt)
except sqlite3.OperationalError as exc:
lowered = str(exc).lower()
if "duplicate column name" in lowered or "no such table" in lowered:
logger.debug("Skipping migration (already applied or table absent): %s", stmt[:80])
else:
raise
conn.commit()
finally:
conn.close()
def cascade_delete(self, service_id: str) -> None:
"""Delete all rows for a ``service_id`` across every concern's tables.
Called from :meth:`SettingsStore.delete_service` after the service row
is removed. Best-effort: callers wrap in try/except so a harness
failure does not block service deletion.
"""
for concern in self._concerns.values():
col = concern.service_id_column
path = self.db_path(concern.concern_key)
if not path.exists():
continue
with sqlite3.connect(path, timeout=30) as conn:
for table in concern.tables:
cols = {row[1] for row in conn.execute(f"PRAGMA table_info({table})").fetchall()}
if col in cols:
conn.execute(f"DELETE FROM {table} WHERE {col} = ?", (service_id,))
# ---------------------------------------------------------------------------
# Module-level singleton
# ---------------------------------------------------------------------------
_HARNESS: ServiceDataHarness | None = None
def get_service_data_harness() -> ServiceDataHarness:
"""Return the process-wide harness singleton, initializing it on first call.
Lazy registration of built-in concerns happens here (local imports avoid
circular dependencies). Migrations are run immediately after registration.
"""
global _HARNESS
if _HARNESS is None:
base_dir = Path(os.environ.get("BACKEND_CACHE_DIR", ".cache/media_library_viewer"))
_HARNESS = ServiceDataHarness(base_dir)
# Register built-in concerns (lazy import avoids circular dependency).
from media_library_viewer_api.services.qbittorrent_store import QBITTORRENT_CONCERN
_HARNESS.register(QBITTORRENT_CONCERN)
from media_library_viewer_api.services.media_index_impl import MEDIA_INDEX_CONCERN
_HARNESS.register(MEDIA_INDEX_CONCERN)
_HARNESS.run_migrations()
return _HARNESS
def reset_service_data_harness() -> None:
"""Reset the singleton (for testing)."""
global _HARNESS
_HARNESS = None
@@ -81,7 +81,7 @@ class SettingsStore:
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_monitoring_machines_mode ON monitoring_machines(mode)")
# The legacy SSH-scraping monitor (MonitoringPoller) was decommissioned;
# metrics now live in Prometheus/node_exporter/Grafana. Drop the orphan
# metrics now live in Prometheus/node_exporter. Drop the orphan
# table on startup so existing databases get a clean slate.
conn.execute("DROP TABLE IF EXISTS monitoring_machine_actions")
conn.execute(
@@ -1740,7 +1740,7 @@ class SettingsStore:
return self.get_service(service["id"]) or service
def delete_service(self, service_id: str) -> None:
"""Delete a service and cascade-delete widgets referencing it."""
"""Delete a service and cascade-delete widgets + harness data."""
self.init_schema()
with self.connect() as conn:
# The service_id column on dashboard_widgets is added in a later
@@ -1753,6 +1753,15 @@ class SettingsStore:
)
conn.execute("DELETE FROM services WHERE id = ?", (service_id,))
# Cascade-delete harness-managed data (best-effort: the service row is
# already removed; data cleanup must not block service deletion).
try:
from media_library_viewer_api.services.service_data import get_service_data_harness
get_service_data_harness().cascade_delete(service_id)
except Exception:
logger.exception("Failed to cascade-delete harness data for service %s", service_id)
def record_service_task_run(self, payload: dict[str, Any]) -> dict[str, Any]:
"""Append a service task run history row."""
self.init_schema()
@@ -0,0 +1,97 @@
"""Shared helpers for Prometheus range queries.
These two pieces were called out by the spec's downstream-notes as needing a
home: the step-derivation function (SC-104) and the series normalization helper
(SC-102). Keeping them in their own module makes them unit-testable in isolation
and reusable by the chart and mean widget paths (and, later, the in-service data
path of the service-storage-harness change) without ``sources.py`` growing
unbounded.
``normalize_prometheus_matrix`` is a direct extraction of the metric-label →
readable-label rule that previously lived inside the Grafana datasource-proxy
path, retargeted at the native Prometheus ``/api/v1/query_range`` matrix shape so
users migrating a ``grafana/chart`` widget to ``prometheus/chart`` see identical
labels.
"""
from __future__ import annotations
from typing import Any
#: Window presets (SC-108, SC-112). Users pick one of these rather than typing
#: raw ``from``/``to``/``step`` values. Values are window lengths in seconds.
WINDOW_PRESETS: dict[str, int] = {
"1h": 3_600,
"6h": 21_600,
"24h": 86_400,
"7d": 604_800,
}
#: Sentinel values Prometheus serialises for non-finite floats; map these to
#: ``None`` so the frontend renderer can skip them via ``connectNulls``.
_NON_NUMERIC = (None, "NaN", "+Inf", "-Inf")
def step_for_window(window_seconds: int, target_points: int = 200) -> int:
"""Derive a scrape ``step`` for a window that yields ~``target_points`` samples.
Clamped to a minimum of 15 seconds so Prometheus does not reject
sub-15s resolutions on high-cardinality queries. The spec (SC-104) requires
the resulting point count to land in the 100300 band; with
``target_points=200`` every preset yields 200 points.
"""
return max(15, round(window_seconds / target_points))
def normalize_prometheus_matrix(result: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""Turn a Prometheus ``/api/v1/query_range`` ``data.result`` matrix into the
``{label, points:[{t:int, v:float|None}]}`` series shape the frontend chart
renderer consumes.
Label rule (matches the removed Grafana path so labels are stable on
migration):
1. Drop ``__name__`` (and any other ``__``-prefixed) metric labels.
2. If labels remain, join them as ``k=v k=v`` (sorted for determinism).
3. Else fall back to ``"value"``.
4. Dedup label collisions with a `` (n)`` suffix.
"""
series: list[dict[str, Any]] = []
seen: dict[str, int] = {}
for entry in result:
metric = entry.get("metric") or {}
values = entry.get("values") or []
parts = [f"{k}={v}" for k, v in sorted(metric.items()) if not str(k).startswith("__")]
label = " ".join(parts) if parts else "value"
if label in seen:
seen[label] += 1
label = f"{label} ({seen[label]})"
else:
seen[label] = 0
points: list[dict[str, Any]] = []
for ts, raw in values:
t = _safe_int(ts)
if t is None:
# Drop samples whose timestamp is unusable rather than raising.
continue
points.append({"t": t, "v": _safe_float(raw)})
series.append({"label": label, "points": points})
return series
def _safe_float(raw: Any) -> float | None:
"""Best-effort float conversion; Prometheus sentinels and junk → ``None``."""
if raw in _NON_NUMERIC:
return None
try:
return float(raw)
except (TypeError, ValueError):
return None
def _safe_int(ts: Any) -> int | None:
"""Best-effort int conversion for a Prometheus timestamp."""
try:
return int(float(ts))
except (TypeError, ValueError):
return None
@@ -12,19 +12,27 @@ from __future__ import annotations
import asyncio
import logging
import time
from dataclasses import dataclass, field
from typing import Any, Protocol
import requests
from media_library_viewer_api.clients.jellyfin import JellyfinClient
from media_library_viewer_api.clients.qbittorrent import QbittorrentClient
from media_library_viewer_api.domain.dashboard import (
_map_sessions_to_activity_rows,
build_backup_dashboard_summary,
)
from media_library_viewer_api.integrations.alertmanager import summarize_alerts
from media_library_viewer_api.services.qbittorrent_store import QbittorrentSampleStore
from media_library_viewer_api.services.settings_store import SettingsStore, get_settings_store
from media_library_viewer_api.services.task_runner import run_saved_task
from media_library_viewer_api.widgets.prometheus_range import (
WINDOW_PRESETS,
normalize_prometheus_matrix,
step_for_window,
)
logger = logging.getLogger(__name__)
@@ -96,120 +104,8 @@ class StaticWidgetSource:
# ---------------------------------------------------------------------------
class GrafanaWidgetSource:
"""Build a Grafana deep-link or query datasource for a chart."""
async def fetch(self, service: ServiceRecord | None, widget_kind: str, config: dict[str, Any]) -> dict[str, Any]:
try:
if service is None:
return {"error": "Grafana widget is missing its service"}
base_url = str(service.config.get("base_url") or "").rstrip("/")
api_key = str(service.secrets.get("api_key") or "")
timeout = int(service.config.get("timeout_seconds") or 10)
if widget_kind == "chart":
return await self._fetch_chart(base_url, api_key, timeout, config)
# Default: deep-link
dashboard_uid = config.get("dashboard_uid")
if not dashboard_uid:
return {"error": "dashboard_uid is required"}
url = f"{base_url}/d/{dashboard_uid}"
panel_id = config.get("panel_id")
if panel_id is not None:
url = f"{url}?viewPanel={panel_id}"
return {"url": url}
except Exception as exc:
logger.exception("grafana adapter failed")
return {"error": f"Grafana link failed: {exc}"}
async def _fetch_chart(self, base_url: str, api_key: str, timeout: int, config: dict[str, Any]) -> dict[str, Any]:
"""Query Grafana /api/ds/query and normalize to {series: [...]}."""
if not api_key:
return {"error": "Grafana api_key is required for chart queries"}
query = config.get("query", "")
if not query:
return {"error": "query is required"}
datasource_uid = config.get("datasource_uid", "prometheus")
body = {
"queries": [
{
"datasource": {"uid": datasource_uid, "type": "prometheus"},
"expr": query,
"format": "time_series",
"intervalMs": int(config.get("interval_ms", 30_000)),
"maxDataPoints": int(config.get("max_data_points", 100)),
"refId": "A",
}
],
"from": config.get("from_ts", "now-1h"),
"to": config.get("to_ts", "now"),
}
def _do_post() -> dict[str, Any]:
resp = requests.post(
f"{base_url}/api/ds/query",
json=body,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
timeout=timeout,
)
resp.raise_for_status()
return resp.json()
try:
raw = await asyncio.wait_for(asyncio.to_thread(_do_post), timeout=timeout)
except asyncio.TimeoutError:
return {"error": "Grafana query timed out"}
except requests.RequestException as exc:
return {"error": f"Grafana query failed: {exc}"}
# Normalize Grafana's /api/ds/query response into series.
series: list[dict[str, Any]] = []
results = raw.get("results", {})
seen_labels: dict[str, int] = {}
for ref_id, ref_data in results.items():
for frame in ref_data.get("frames", []):
values = frame.get("data", {}).get("values", [])
if len(values) < 2:
continue
timestamps = values[0]
vals = values[1]
# Derive a meaningful series label from the frame metadata.
# Prometheus frames carry metric labels in schema.fields[-1].labels.
fields = frame.get("schema", {}).get("fields", [])
value_field = fields[-1] if fields else {}
# Prefer displayName (explicitly set in Grafana), then Prometheus
# labels (e.g. {instance: "server:9100", mode: "iowait"}), then
# the field name as a last resort.
display_name = value_field.get("config", {}).get("displayName") or value_field.get("displayName")
frame_labels = value_field.get("labels") or {}
if display_name:
label = str(display_name)
elif frame_labels:
# Build a readable label from the Prometheus labels, excluding
# redundant ones like __name__.
parts = [f"{k}={v}" for k, v in sorted(frame_labels.items()) if not k.startswith("__")]
label = " ".join(parts) if parts else "value"
else:
label = value_field.get("name", "value")
# Ensure unique labels when multiple series share the same name.
if label in seen_labels:
seen_labels[label] += 1
label = f"{label} ({seen_labels[label]})"
else:
seen_labels[label] = 0
points = [{"t": int(t), "v": float(v) if v is not None else None} for t, v in zip(timestamps, vals)]
series.append({"label": label, "points": points})
return {"series": series}
class PrometheusWidgetSource:
"""Run a PromQL instant query against a Prometheus service."""
"""Run PromQL queries against a Prometheus service (instant + range)."""
async def fetch(self, service: ServiceRecord | None, widget_kind: str, config: dict[str, Any]) -> dict[str, Any]:
try:
@@ -217,14 +113,63 @@ class PrometheusWidgetSource:
return {"error": "Prometheus widget is missing its service"}
base_url = str(service.config.get("base_url") or "").rstrip("/")
timeout = int(service.config.get("timeout_seconds") or 10)
promql = config.get("promql")
if not promql:
return {"error": "promql is required"}
url = f"{base_url}/api/v1/query"
if widget_kind == "chart":
return await self._fetch_chart(base_url, timeout, config)
if widget_kind == "gauge":
return await self._fetch_gauge(base_url, timeout, config)
if widget_kind == "mean":
return await self._fetch_mean(base_url, timeout, config)
# Default: instant-query metric path (unchanged).
raw = await self._instant_query(base_url, timeout, config.get("promql", ""))
return raw
except Exception as exc:
logger.exception("prometheus adapter failed")
return {"error": f"Prometheus query failed: {exc}"}
async def _range_query(self, base_url: str, timeout: int, promql: str, window: int) -> dict[str, Any]:
"""Run a Prometheus ``/api/v1/query_range`` over a window (seconds).
Shared by the ``chart`` (SC-101) and ``mean`` widget kinds. Returns
``{"matrix": result}`` on success or ``{"error": str}`` (never raises,
per SC-103).
"""
step = step_for_window(window)
end = int(time.time())
start = end - window
try:
response = await asyncio.wait_for(
asyncio.to_thread(
requests.get,
url,
f"{base_url}/api/v1/query_range",
params={"query": promql, "start": start, "end": end, "step": step},
timeout=timeout,
),
timeout=timeout,
)
response.raise_for_status()
payload = response.json()
except asyncio.TimeoutError:
return {"error": "Prometheus query timed out"}
except requests.RequestException as exc:
logger.exception("prometheus range query failed")
return {"error": f"Prometheus query failed: {exc}"}
result = payload.get("data", {}).get("result", [])
return {"matrix": result}
async def _instant_query(self, base_url: str, timeout: int, promql: str) -> dict[str, Any]:
"""Run a Prometheus ``/api/v1/query`` instant query.
Shared by the ``metric`` and ``gauge`` widget kinds. Returns
``{"result": data}`` on success or ``{"error": str}`` (never raises,
per SC-103).
"""
if not promql:
return {"error": "promql is required"}
try:
response = await asyncio.wait_for(
asyncio.to_thread(
requests.get,
f"{base_url}/api/v1/query",
params={"query": promql},
timeout=timeout,
),
@@ -232,15 +177,80 @@ class PrometheusWidgetSource:
)
response.raise_for_status()
payload = response.json()
return {"result": payload.get("data", {})}
except asyncio.TimeoutError:
return {"error": "Widget data fetch timed out"}
return {"error": "Prometheus query timed out"}
except requests.RequestException as exc:
logger.exception("prometheus adapter failed")
return {"error": f"Prometheus query failed: {exc}"}
except Exception as exc:
logger.exception("prometheus adapter failed")
logger.exception("prometheus instant query failed")
return {"error": f"Prometheus query failed: {exc}"}
return {"result": payload.get("data", {})}
async def _fetch_chart(self, base_url: str, timeout: int, config: dict[str, Any]) -> dict[str, Any]:
"""Range query → ``{series}`` for the chart widget (SC-101..SC-104)."""
promql = config.get("promql")
if not promql:
return {"error": "promql is required"}
window = WINDOW_PRESETS.get(config.get("window", "1h"), WINDOW_PRESETS["1h"])
raw = await self._range_query(base_url, timeout, promql, window)
if "error" in raw:
return raw
return {"series": normalize_prometheus_matrix(raw["matrix"])}
async def _fetch_gauge(self, base_url: str, timeout: int, config: dict[str, Any]) -> dict[str, Any]:
"""Instant query → scalar for the gauge widget (SC-109, SC-110, SC-111).
Scalar-only: a multi-series query returns an error (SC-111). Threshold
config (``warn_at``/``crit_at``/``min``/``max``/``unit``) is passed
through for the frontend renderer.
"""
raw = await self._instant_query(base_url, timeout, config.get("promql") or "")
if "error" in raw:
return raw
result = raw["result"].get("result", [])
if len(result) != 1:
return {"error": "Gauge requires a single-series query; refine your PromQL"}
try:
value = float(result[0]["value"][1])
except (KeyError, IndexError, ValueError, TypeError):
return {"error": "Gauge query returned no scalar value"}
return {
"value": value,
"warn_at": config.get("warn_at"),
"crit_at": config.get("crit_at"),
"min": config.get("min"),
"max": config.get("max"),
"unit": config.get("unit"),
}
async def _fetch_mean(self, base_url: str, timeout: int, config: dict[str, Any]) -> dict[str, Any]:
"""Range query → client-side mean for the mean widget (SC-112..SC-114).
Runs ``query_range`` over the configured window preset, averages all
non-null numeric samples of the single series, and returns a scalar.
Scalar-only: a multi-series query returns an error (SC-114).
"""
promql = config.get("promql")
if not promql:
return {"error": "promql is required"}
window = WINDOW_PRESETS.get(config.get("window", "1h"), WINDOW_PRESETS["1h"])
raw = await self._range_query(base_url, timeout, promql, window)
if "error" in raw:
return raw
result = raw["matrix"]
if len(result) != 1:
return {"error": "Mean requires a single-series query; refine your PromQL"}
points = result[0].get("values") or []
nums: list[float] = []
for _, v in points:
if v in (None, "NaN", "+Inf", "-Inf"):
continue
try:
nums.append(float(v))
except (TypeError, ValueError):
continue
if not nums:
return {"error": "Mean query returned no numeric samples in the window"}
mean = sum(nums) / len(nums)
return {"value": mean, "unit": config.get("unit")}
class AlertmanagerWidgetSource:
@@ -360,13 +370,75 @@ def _record_timeout(service: ServiceRecord | None, config: dict[str, Any], timeo
logger.exception("failed to record ssh task timeout")
class QbittorrentWidgetSource:
"""Fetch qBittorrent data for totals, active, and speed widgets."""
async def fetch(self, service: ServiceRecord | None, widget_kind: str, config: dict[str, Any]) -> dict[str, Any]:
try:
if service is None:
return {"error": "qBittorrent widget is missing its service"}
base_url = str(service.config.get("base_url") or "")
username = str(service.secrets.get("username") or "")
password = str(service.secrets.get("password") or "")
timeout = int(service.config.get("timeout_seconds") or 10)
if not base_url or not username or not password:
return {"error": "qBittorrent service is missing base_url, username, or password"}
client = QbittorrentClient(base_url, username, password, timeout)
data = await asyncio.wait_for(asyncio.to_thread(client.maindata), timeout=timeout)
server_state = data.get("server_state", {})
torrents = data.get("torrents", {})
if widget_kind == "totals":
by_state: dict[str, int] = {}
for t in torrents.values():
state = str(t.get("state", "unknown"))
by_state[state] = by_state.get(state, 0) + 1
return {"total": len(torrents), "by_state": by_state}
if widget_kind == "active":
active = [
{
"name": t.get("name"),
"state": t.get("state"),
"size": t.get("size"),
"progress": t.get("progress"),
"dl_speed": t.get("dlspeed"),
"up_speed": t.get("upspeed"),
}
for t in torrents.values()
if str(t.get("state", "")) in {"downloading", "uploading"}
]
return {"torrents": active}
if widget_kind == "speed":
dl = int(server_state.get("dl_info_speed", 0))
up = int(server_state.get("up_info_speed", 0))
ts = int(time.time())
store = QbittorrentSampleStore()
store.append(service.id, ts, dl, up)
samples = store.window(service.id)
series = [
{"label": "download", "points": [{"t": s["ts"] * 1000, "v": s["dl_speed"]} for s in samples]},
{"label": "upload", "points": [{"t": s["ts"] * 1000, "v": s["up_speed"]} for s in samples]},
]
return {"series": series}
return {"error": f"Unknown qBittorrent widget kind: {widget_kind}"}
except asyncio.TimeoutError:
return {"error": "qBittorrent data fetch timed out"}
except Exception as exc:
logger.exception("qbittorrent adapter failed")
return {"error": f"qBittorrent fetch failed: {exc}"}
# ---------------------------------------------------------------------------
# Registries
# ---------------------------------------------------------------------------
SERVICE_ADAPTERS: dict[str, WidgetSource] = {
"grafana": GrafanaWidgetSource(),
"prometheus": PrometheusWidgetSource(),
"qbittorrent": QbittorrentWidgetSource(),
"alertmanager": AlertmanagerWidgetSource(),
"jellyfin": JellyfinWidgetSource(),
"ssh_tasks": SshTaskWidgetSource(),
@@ -174,6 +174,7 @@ def run_build(final_index_path: str | Path, staging_index_path: str | Path, serv
fallback_prefix=settings.path_prefix,
progress_callback=lambda state: _progress_callback(final_index, pid, state),
should_cancel=lambda: _cancel_requested(final_index),
service_id=service_id,
)
# Swap the staging database into place atomically.
os.replace(staging_index.db_path, final_index.db_path)
+1 -43
View File
@@ -522,7 +522,7 @@ class TestResolveServiceRecord:
def test_service_id_type_mismatch_returns_none(self):
from media_library_viewer_api.services.service_resolution import resolve_service_record
row = {"id": "x1", "service_type": "grafana", "name": "G", "enabled": True, "config": {}, "secrets": {}}
row = {"id": "x1", "service_type": "prometheus", "name": "P", "enabled": True, "config": {}, "secrets": {}}
store = self._store([row])
assert resolve_service_record(store, "alertmanager", "x1") is None
@@ -741,48 +741,6 @@ class TestAlertmanagerWebhook:
assert "Received Alertmanager webhook with 1 alert(s)" in caplog.text
class TestGrafanaStatus:
def test_grafana_status_when_not_configured(self, test_client):
response = test_client.get("/api/monitoring/grafana-status")
assert response.status_code == 200
data = response.json()
assert data["up"] is False
assert data["error"] == "no_service_configured"
def test_grafana_status_when_unreachable(self, test_client):
service = ServiceRecord(
id="g1", service_type="grafana", name="Grafana", config={"base_url": "http://grafana:3000"}
)
with (
patch(f"{_MON}.resolve_service_record", return_value=service),
patch(f"{_MON}.requests.get", side_effect=Exception("refused")),
):
response = test_client.get("/api/monitoring/grafana-status")
assert response.status_code == 200
data = response.json()
assert data["up"] is False
assert data["error"] == "grafana_unreachable"
assert data["name"] == "Grafana"
def test_grafana_status_returns_version(self, test_client):
service = ServiceRecord(
id="g1", service_type="grafana", name="Grafana", config={"base_url": "http://grafana:3000"}
)
resp = MagicMock()
resp.json.return_value = {"version": "11.3.1", "database": "ok"}
resp.raise_for_status = MagicMock()
with (
patch(f"{_MON}.resolve_service_record", return_value=service),
patch(f"{_MON}.requests.get", return_value=resp),
):
response = test_client.get("/api/monitoring/grafana-status")
assert response.status_code == 200
data = response.json()
assert data["up"] is True
assert data["version"] == "11.3.1"
assert data["service_id"] == "g1"
class TestPrometheusStatus:
def test_prometheus_status_when_not_configured(self, test_client):
response = test_client.get("/api/monitoring/prometheus-status")
+62 -19
View File
@@ -228,19 +228,70 @@ class TestMediaIndexQuery:
class TestMediaIndexReplace:
def test_replace_clears_old(self, index):
index.replace_items([
{"id": "x", "title": "Old", "type": "Movie", "library_id": "l1", "library_name": "L1"},
])
index.replace_items(
[
{"id": "x", "title": "Old", "type": "Movie", "library_id": "l1", "library_name": "L1"},
]
)
status = index.status()
assert status.item_count == 1
index.replace_items([
{"id": "y", "title": "New1", "type": "Movie", "library_id": "l1", "library_name": "L1"},
{"id": "z", "title": "New2", "type": "Movie", "library_id": "l1", "library_name": "L1"},
])
index.replace_items(
[
{"id": "y", "title": "New1", "type": "Movie", "library_id": "l1", "library_name": "L1"},
{"id": "z", "title": "New2", "type": "Movie", "library_id": "l1", "library_name": "L1"},
]
)
status = index.status()
assert status.item_count == 2
def test_replace_scoped_by_service_id_preserves_other_services(self, index):
"""Regression test: building for one Jellyfin must not wipe another's rows.
Before the migration, ``replace_items`` did ``DELETE FROM media_items``
(global clear). This test locks in the fix: a scoped replace preserves
rows belonging to a different service_id.
"""
rows_a = [
{"id": "a1", "title": "Alpha Movie", "type": "Movie", "library_id": "l1", "library_name": "L1"},
{"id": "a2", "title": "Alpha Show", "type": "Episode", "library_id": "l2", "library_name": "L2"},
]
rows_b = [
{"id": "b1", "title": "Beta Movie", "type": "Movie", "library_id": "l1", "library_name": "L1"},
]
index.replace_items(rows_a, service_id="svc-a")
assert index.status().item_count == 2
# Replacing for svc-b must NOT wipe svc-a's rows.
index.replace_items(rows_b, service_id="svc-b")
assert index.status().item_count == 3 # 2 from svc-a + 1 from svc-b
# Querying svc-a returns only its rows.
rows_a_result, total_a = index.query(
library_ids=["l1", "l2"],
media_types=["Movie", "Episode"],
service_id="svc-a",
)
assert total_a == 2
assert {r["id"] for r in rows_a_result} == {"a1", "a2"}
# Querying svc-b returns only its rows.
rows_b_result, total_b = index.query(
library_ids=["l1"],
media_types=["Movie"],
service_id="svc-b",
)
assert total_b == 1
assert rows_b_result[0]["id"] == "b1"
# Querying with no service_id returns all rows (backward-compat).
_, total_all = index.query(
library_ids=["l1", "l2"],
media_types=["Movie", "Episode"],
)
assert total_all == 3
class TestMediaIndexMetadata:
def test_set_and_read_metadata(self, index):
@@ -288,15 +339,11 @@ class TestMediaIndexBuildPaths:
self.calls.append(kwargs.get("start_index", 0))
if kwargs.get("start_index", 0) == 0:
return {
"Items": [
{"Id": "m1", "Name": "Movie One", "Type": "Movie", "Path": "/media/a.mkv"}
],
"Items": [{"Id": "m1", "Name": "Movie One", "Type": "Movie", "Path": "/media/a.mkv"}],
"TotalRecordCount": 2,
}
return {
"Items": [
{"Id": "m2", "Name": "Movie Two", "Type": "Movie", "Path": "/media/b.mkv"}
],
"Items": [{"Id": "m2", "Name": "Movie Two", "Type": "Movie", "Path": "/media/b.mkv"}],
"TotalRecordCount": 2,
}
@@ -332,15 +379,11 @@ class TestMediaIndexBuildPaths:
self.calls.append(kwargs.get("start_index", 0))
if kwargs.get("start_index", 0) == 0:
return {
"Items": [
{"Id": "m1", "Name": "Movie One", "Type": "Movie", "Path": "/media/a.mkv"}
],
"Items": [{"Id": "m1", "Name": "Movie One", "Type": "Movie", "Path": "/media/a.mkv"}],
"TotalRecordCount": 2,
}
return {
"Items": [
{"Id": "m2", "Name": "Movie Two", "Type": "Movie", "Path": "/media/b.mkv"}
],
"Items": [{"Id": "m2", "Name": "Movie Two", "Type": "Movie", "Path": "/media/b.mkv"}],
"TotalRecordCount": 2,
}
+106
View File
@@ -0,0 +1,106 @@
"""Unit tests for the shared Prometheus range-query helpers (SC-101..SC-104)."""
from __future__ import annotations
import pytest
from media_library_viewer_api.widgets.prometheus_range import (
WINDOW_PRESETS,
normalize_prometheus_matrix,
step_for_window,
)
class TestStepForWindow:
"""SC-104: every preset must yield 100300 points."""
@pytest.mark.parametrize("preset", sorted(WINDOW_PRESETS))
def test_presets_yield_in_band_point_counts(self, preset: str) -> None:
window = WINDOW_PRESETS[preset]
step = step_for_window(window)
# Clamped minimum.
assert step >= 15
point_count = window // step
assert 100 <= point_count <= 300, f"{preset}: {point_count} points (step={step})"
def test_floor_of_fifteen_seconds(self) -> None:
# A tiny window that would otherwise produce a sub-15s step is clamped.
assert step_for_window(60) == 15
def test_custom_target_points(self) -> None:
# Targeting 100 points for 1h yields step 36 (3600/100).
assert step_for_window(3_600, target_points=100) == 36
class TestNormalizePrometheusMatrix:
"""SC-102: label rule + null handling + dedup."""
def test_empty_matrix(self) -> None:
assert normalize_prometheus_matrix([]) == []
def test_drops_dunder_labels_and_joins(self) -> None:
result = [
{
"metric": {"__name__": "node_cpu_seconds_total", "instance": "host:9100", "mode": "idle"},
"values": [[1_700_000_000, "12.5"], [1_700_000_030, "13.0"]],
}
]
out = normalize_prometheus_matrix(result)
assert len(out) == 1
assert out[0]["label"] == "instance=host:9100 mode=idle"
assert out[0]["points"] == [
{"t": 1_700_000_000, "v": 12.5},
{"t": 1_700_000_030, "v": 13.0},
]
def test_falls_back_to_value_when_no_labels(self) -> None:
result = [{"metric": {}, "values": [[100, "1"]]}]
out = normalize_prometheus_matrix(result)
assert out[0]["label"] == "value"
def test_dedup_collisions_with_suffix(self) -> None:
# Two series with identical visible labels get a "(1)" suffix on the 2nd.
result = [
{"metric": {"job": "x"}, "values": [[1, "1"]]},
{"metric": {"job": "x"}, "values": [[1, "2"]]},
]
out = normalize_prometheus_matrix(result)
labels = [s["label"] for s in out]
assert labels == ["job=x", "job=x (1)"]
def test_non_numeric_sentinels_become_none(self) -> None:
result = [
{
"metric": {"job": "x"},
"values": [
[1, "NaN"],
[2, "+Inf"],
[3, "-Inf"],
[4, "3.5"],
],
}
]
out = normalize_prometheus_matrix(result)
assert out[0]["points"] == [
{"t": 1, "v": None},
{"t": 2, "v": None},
{"t": 3, "v": None},
{"t": 4, "v": 3.5},
]
def test_malformed_values_are_ignored_not_raised(self) -> None:
result = [
{
"metric": {"job": "x"},
"values": [
[1, "3.5"],
["not-a-ts", "9"], # unusable timestamp → dropped
[3, "junk-value"], # unparseable value → v: None
],
}
]
out = normalize_prometheus_matrix(result)
assert out[0]["points"] == [
{"t": 1, "v": 3.5},
{"t": 3, "v": None},
]
+134
View File
@@ -0,0 +1,134 @@
"""Unit tests for the QbittorrentClient."""
from __future__ import annotations
import unittest
from unittest.mock import MagicMock, patch
import requests
from media_library_viewer_api.clients.qbittorrent import QbittorrentClient
class QbittorrentClientTests(unittest.TestCase):
def setUp(self) -> None:
self.client = QbittorrentClient("https://qb.example.com", "admin", "secret", timeout=5)
self.session = MagicMock()
self.client._session = self.session
def _login_response(self, text: str = "Ok.") -> MagicMock:
resp = MagicMock()
resp.text = text
resp.raise_for_status.return_value = None
resp.status_code = 200
return resp
def _get_response(self, json_data: dict, status_code: int = 200) -> MagicMock:
resp = MagicMock()
resp.json.return_value = json_data
resp.raise_for_status.return_value = None
resp.status_code = status_code
resp.text = ""
return resp
def test_base_url_appends_api_v2(self) -> None:
c = QbittorrentClient("https://qb.example.com", "u", "p")
self.assertEqual(c.base_url, "https://qb.example.com/api/v2")
def test_base_url_keeps_existing_api_v2(self) -> None:
c = QbittorrentClient("https://qb.example.com/api/v2", "u", "p")
self.assertEqual(c.base_url, "https://qb.example.com/api/v2")
def test_base_url_strips_trailing_slash(self) -> None:
c = QbittorrentClient("https://qb.example.com/", "u", "p")
self.assertEqual(c.base_url, "https://qb.example.com/api/v2")
def test_empty_base_url_raises(self) -> None:
with self.assertRaises(ValueError):
QbittorrentClient("", "u", "p")
def test_empty_username_raises(self) -> None:
with self.assertRaises(ValueError):
QbittorrentClient("https://qb.example.com", "", "p")
def test_login_posts_credentials(self) -> None:
self.session.post.return_value = self._login_response("Ok.")
self.client._login()
self.session.post.assert_called_once()
call_args = self.session.post.call_args
self.assertIn("/auth/login", call_args.args[0])
self.assertEqual(call_args.kwargs["data"], {"username": "admin", "password": "secret"})
self.assertTrue(self.client._logged_in)
def test_login_failure_raises_runtime_error(self) -> None:
self.session.post.return_value = self._login_response("Fails.")
with self.assertRaises(RuntimeError):
self.client._login()
def test_get_auto_logs_in_on_first_call(self) -> None:
"""First _get triggers login, then fetches data."""
self.session.post.return_value = self._login_response("Ok.")
self.session.get.return_value = self._get_response({"server_state": {}, "torrents": {}})
result = self.client._get("/sync/maindata")
self.session.post.assert_called_once() # login happened
self.assertEqual(result, {"server_state": {}, "torrents": {}})
def test_cookie_reuse_does_not_re_login(self) -> None:
"""After login, subsequent _get calls do NOT re-login."""
self.client._logged_in = True # simulate already logged in
self.session.get.return_value = self._get_response({"data": 1})
self.client._get("/some/path")
self.session.post.assert_not_called() # no re-login
def test_403_triggers_re_login(self) -> None:
"""A 403 response triggers re-login and retries the GET."""
self.client._logged_in = True # already logged in from a prior call
forbidden = MagicMock()
forbidden.status_code = 403
ok = self._get_response({"server_state": {}, "torrents": {}})
self.session.get.side_effect = [forbidden, ok]
self.session.post.return_value = self._login_response("Ok.")
result = self.client._get("/sync/maindata")
self.assertEqual(self.session.get.call_count, 2) # initial + retry
self.session.post.assert_called_once() # re-login happened
self.assertEqual(result, {"server_state": {}, "torrents": {}})
def test_maindata_returns_full_payload(self) -> None:
self.client._logged_in = True
payload = {
"server_state": {"dl_info_speed": 12345, "up_info_speed": 6789},
"torrents": {
"abc": {"name": "Movie.mkv", "state": "downloading", "progress": 0.5},
"def": {"name": "Show.mkv", "state": "uploading", "progress": 1.0},
},
}
self.session.get.return_value = self._get_response(payload)
result = self.client.maindata()
self.assertEqual(result["server_state"]["dl_info_speed"], 12345)
self.assertEqual(len(result["torrents"]), 2)
@patch("media_library_viewer_api.clients.qbittorrent.requests.Session")
def test_login_http_error_propagates(self, mock_session_cls: MagicMock) -> None:
"""A network error during login propagates as requests exception."""
mock_session = MagicMock()
mock_session_cls.return_value = mock_session
bad_resp = MagicMock()
bad_resp.raise_for_status.side_effect = requests.ConnectionError("refused")
bad_resp.text = ""
mock_session.post.return_value = bad_resp
client = QbittorrentClient("https://qb.example.com", "u", "p")
with self.assertRaises(requests.ConnectionError):
client._login()
if __name__ == "__main__":
unittest.main()
+148
View File
@@ -0,0 +1,148 @@
"""Tests for ServiceDataHarness lifecycle and QbittorrentSampleStore operations."""
from __future__ import annotations
import pytest
from media_library_viewer_api.services.qbittorrent_store import (
MAX_SAMPLES,
QBITTORRENT_CONCERN,
QbittorrentSampleStore,
)
from media_library_viewer_api.services.service_data import (
ServiceDataHarness,
StorageConcern,
)
# A throwaway concern used to test harness lifecycle in isolation.
_TEST_CONCERN = StorageConcern(
concern_key="test",
db_filename="test.db",
migrations=[
"""
CREATE TABLE IF NOT EXISTS test_items (
id INTEGER PRIMARY KEY,
service_id TEXT NOT NULL,
value TEXT
);
CREATE INDEX IF NOT EXISTS idx_test_service ON test_items(service_id);
"""
],
tables=["test_items"],
)
class TestServiceDataHarnessMigrations:
def test_run_migrations_creates_tables(self, tmp_path):
harness = ServiceDataHarness(tmp_path)
harness.register(_TEST_CONCERN)
harness.run_migrations()
with harness.connect("test") as conn:
tables = {row[0] for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall()}
assert "test_items" in tables
def test_migrations_are_idempotent(self, tmp_path):
"""Re-running migrations on an already-migrated DB must not crash."""
harness = ServiceDataHarness(tmp_path)
harness.register(_TEST_CONCERN)
harness.run_migrations()
harness.run_migrations() # should not raise
def test_alter_table_idempotency(self, tmp_path):
"""ALTER TABLE ADD COLUMN must be silently skipped on re-run."""
concern = StorageConcern(
concern_key="alter_test",
db_filename="alter.db",
migrations=[
"CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY)",
"ALTER TABLE items ADD COLUMN extra TEXT DEFAULT ''",
],
tables=["items"],
)
harness = ServiceDataHarness(tmp_path)
harness.register(concern)
harness.run_migrations()
harness.run_migrations() # second run: "duplicate column name" caught
with harness.connect("alter_test") as conn:
cols = {row[1] for row in conn.execute("PRAGMA table_info(items)").fetchall()}
assert "extra" in cols
class TestServiceDataHarnessCascadeDelete:
def test_cascade_delete_removes_only_matching_service(self, tmp_path):
harness = ServiceDataHarness(tmp_path)
harness.register(_TEST_CONCERN)
harness.run_migrations()
with harness.connect("test") as conn:
conn.execute("INSERT INTO test_items (id, service_id, value) VALUES (1, 'svc-a', 'a1')")
conn.execute("INSERT INTO test_items (id, service_id, value) VALUES (2, 'svc-a', 'a2')")
conn.execute("INSERT INTO test_items (id, service_id, value) VALUES (3, 'svc-b', 'b1')")
conn.commit()
harness.cascade_delete("svc-a")
with harness.connect("test") as conn:
remaining = conn.execute("SELECT service_id, value FROM test_items ORDER BY id").fetchall()
assert len(remaining) == 1
assert remaining[0][0] == "svc-b"
def test_cascade_delete_skips_missing_concern_db(self, tmp_path):
"""cascade_delete on a concern whose DB file doesn't exist should not crash."""
harness = ServiceDataHarness(tmp_path)
harness.register(_TEST_CONCERN)
# No run_migrations → DB file doesn't exist
harness.cascade_delete("svc-x") # should not raise
class TestQbittorrentSampleStore:
@pytest.fixture()
def store(self, tmp_path):
harness = ServiceDataHarness(tmp_path)
harness.register(QBITTORRENT_CONCERN)
harness.run_migrations()
return QbittorrentSampleStore(harness=harness)
def test_append_and_window(self, store):
store.append("svc-1", ts=100, dl_speed=500, up_speed=50)
store.append("svc-1", ts=200, dl_speed=600, up_speed=60)
store.append("svc-1", ts=300, dl_speed=700, up_speed=70)
samples = store.window("svc-1")
assert len(samples) == 3
assert samples[0]["ts"] == 100
assert samples[2]["ts"] == 300
assert samples[1]["dl_speed"] == 600
def test_window_with_since_ts(self, store):
store.append("svc-1", ts=100, dl_speed=500, up_speed=50)
store.append("svc-1", ts=200, dl_speed=600, up_speed=60)
store.append("svc-1", ts=300, dl_speed=700, up_speed=70)
samples = store.window("svc-1", since_ts=200)
assert len(samples) == 2
assert samples[0]["ts"] == 200
def test_prune_enforces_max_samples(self, store):
for i in range(MAX_SAMPLES + 10):
store.append("svc-1", ts=i, dl_speed=i, up_speed=i)
samples = store.window("svc-1")
assert len(samples) == MAX_SAMPLES
# The oldest 10 should have been pruned
assert samples[0]["ts"] == 10
assert samples[-1]["ts"] == MAX_SAMPLES + 9
def test_two_services_do_not_cross_contaminate(self, store):
store.append("svc-a", ts=100, dl_speed=500, up_speed=50)
store.append("svc-b", ts=200, dl_speed=600, up_speed=60)
a_samples = store.window("svc-a")
b_samples = store.window("svc-b")
assert len(a_samples) == 1
assert a_samples[0]["dl_speed"] == 500
assert len(b_samples) == 1
assert b_samples[0]["dl_speed"] == 600
+118 -50
View File
@@ -59,7 +59,6 @@ def client(tmp_path):
def test_registry_contains_eight_service_types():
assert set(SERVICE_DEFINITIONS) == {
"grafana",
"prometheus",
"alertmanager",
"jellyfin",
@@ -67,6 +66,7 @@ def test_registry_contains_eight_service_types():
"ssh_tasks",
"backups",
"authentik",
"qbittorrent",
}
@@ -99,8 +99,7 @@ def test_authentik_service_definition():
def test_definitions_declare_widget_kinds():
assert {wk.kind for wk in get_service_definition("grafana").widget_kinds} == {"link", "chart"}
assert {wk.kind for wk in get_service_definition("prometheus").widget_kinds} == {"metric"}
assert {wk.kind for wk in get_service_definition("prometheus").widget_kinds} == {"metric", "chart", "gauge", "mean"}
assert {wk.kind for wk in get_service_definition("alertmanager").widget_kinds} == {"active_alerts"}
assert {wk.kind for wk in get_service_definition("jellyfin").widget_kinds} == {"activity", "now_playing"}
assert get_service_definition("nextcloud").widget_kinds == []
@@ -110,13 +109,13 @@ def test_definitions_declare_widget_kinds():
def test_widget_kind_lookup():
assert get_widget_kind("grafana", "link") is not None
assert get_widget_kind("grafana", "missing") is None
assert get_widget_kind("unknown", "link") is None
assert get_widget_kind("prometheus", "metric") is not None
assert get_widget_kind("prometheus", "missing") is None
assert get_widget_kind("unknown", "metric") is None
def test_service_config_schema_is_json_schema():
schema = get_service_definition("grafana").config_schema
schema = get_service_definition("prometheus").config_schema
assert schema["type"] == "object"
assert "base_url" in schema["properties"]
@@ -172,19 +171,19 @@ def test_list_service_types(client):
"alertmanager",
"authentik",
"backups",
"grafana",
"jellyfin",
"nextcloud",
"prometheus",
"qbittorrent",
"ssh_tasks",
}
def test_service_type_includes_secret_and_widget_metadata(client):
response = client.get("/api/services/types")
grafana = next(item for item in response.json() if item["service_type"] == "grafana")
assert [sf["key"] for sf in grafana["secret_fields"]] == ["api_key"]
assert [wk["kind"] for wk in grafana["widget_kinds"]] == ["link", "chart"]
prom = next(item for item in response.json() if item["service_type"] == "prometheus")
assert [sf["key"] for sf in prom["secret_fields"]] == ["api_key"]
assert set(wk["kind"] for wk in prom["widget_kinds"]) == {"metric", "chart", "gauge", "mean"}
# ---------------------------------------------------------------------------
@@ -192,11 +191,11 @@ def test_service_type_includes_secret_and_widget_metadata(client):
# ---------------------------------------------------------------------------
def _grafana_payload(**overrides):
def _prometheus_payload(**overrides):
payload = {
"service_type": "grafana",
"name": "Production Grafana",
"config": {"base_url": "https://grafana.example.com"},
"service_type": "prometheus",
"name": "Production Prometheus",
"config": {"base_url": "https://prometheus.example.com"},
"secrets": {"api_key": "secret-token"},
"enabled": True,
}
@@ -205,11 +204,11 @@ def _grafana_payload(**overrides):
def test_create_and_list_service(client):
response = client.post("/api/services/instances", json=_grafana_payload())
response = client.post("/api/services/instances", json=_prometheus_payload())
assert response.status_code == 201
created = response.json()
assert created["service_type"] == "grafana"
assert created["config"]["base_url"] == "https://grafana.example.com"
assert created["service_type"] == "prometheus"
assert created["config"]["base_url"] == "https://prometheus.example.com"
# Plaintext secrets are never returned.
assert "secrets" not in created
assert created["secrets_set"] == {"api_key": True}
@@ -220,44 +219,44 @@ def test_create_and_list_service(client):
def test_list_instances_filters_by_type(client):
client.post("/api/services/instances", json=_grafana_payload())
client.post("/api/services/instances", json=_prometheus_payload())
client.post(
"/api/services/instances",
json={
"service_type": "prometheus",
"name": "Prom",
"config": {"base_url": "http://prometheus:9090"},
"service_type": "alertmanager",
"name": "AM",
"config": {"base_url": "http://am:9093"},
},
)
response = client.get("/api/services/instances?service_type=grafana")
response = client.get("/api/services/instances?service_type=prometheus")
assert response.status_code == 200
assert len(response.json()) == 1
assert response.json()[0]["service_type"] == "grafana"
assert response.json()[0]["service_type"] == "prometheus"
def test_update_service_preserves_unsent_secrets(client):
created = client.post("/api/services/instances", json=_grafana_payload()).json()
created = client.post("/api/services/instances", json=_prometheus_payload()).json()
# Update without sending secrets; the existing key should remain set.
updated = client.put(
f"/api/services/instances/{created['id']}",
json={
"service_type": "grafana",
"name": "Renamed Grafana",
"config": {"base_url": "https://grafana.example.com", "timeout_seconds": 10},
"service_type": "prometheus",
"name": "Renamed Prometheus",
"config": {"base_url": "https://prometheus.example.com", "timeout_seconds": 10},
},
).json()
assert updated["name"] == "Renamed Grafana"
assert updated["name"] == "Renamed Prometheus"
assert updated["secrets_set"] == {"api_key": True}
def test_update_service_can_clear_secret(client):
created = client.post("/api/services/instances", json=_grafana_payload()).json()
created = client.post("/api/services/instances", json=_prometheus_payload()).json()
updated = client.put(
f"/api/services/instances/{created['id']}",
json={
"service_type": "grafana",
"name": "Production Grafana",
"config": {"base_url": "https://grafana.example.com"},
"service_type": "prometheus",
"name": "Production Prometheus",
"config": {"base_url": "https://prometheus.example.com"},
"secrets": {"api_key": ""},
},
).json()
@@ -275,30 +274,28 @@ def test_unknown_service_type_rejected(client):
def test_invalid_config_rejected(client):
response = client.post(
"/api/services/instances",
json={"service_type": "grafana", "name": "x", "config": {"base_url": ""}},
json={"service_type": "prometheus", "name": "x", "config": {"base_url": ""}},
)
assert response.status_code == 422
# Force a real validation error via bad type.
response = client.post(
"/api/services/instances",
json={"service_type": "grafana", "name": "x", "config": {"timeout_seconds": "fast"}},
json={"service_type": "prometheus", "name": "x", "config": {"timeout_seconds": "fast"}},
)
assert response.status_code == 422
@pytest.mark.parametrize(
"bad_url", ["grafana.example.com", "localhost:3000", "//grafana.example.com", "ftp://grafana.example.com"]
"bad_url", ["prometheus.example.com", "localhost:3000", "//bad.example.com", "ftp://bad.example.com"]
)
def test_service_base_url_requires_http_schema(bad_url):
"""Every service base_url must include an http:// or https:// schema."""
model = get_service_definition("grafana").config_model
model = get_service_definition("prometheus").config_model
with pytest.raises(ValidationError):
model.model_validate({"base_url": bad_url, "timeout_seconds": 5})
@pytest.mark.parametrize(
"service_type", ["grafana", "prometheus", "alertmanager", "jellyfin", "authentik", "nextcloud"]
)
@pytest.mark.parametrize("service_type", ["prometheus", "alertmanager", "jellyfin", "authentik", "nextcloud"])
def test_service_base_url_accepts_absolute_urls(service_type):
model = get_service_definition(service_type).config_model
instance = model.model_validate({"base_url": "https://example.com"})
@@ -309,9 +306,9 @@ def test_unknown_secret_field_rejected(client):
response = client.post(
"/api/services/instances",
json={
"service_type": "grafana",
"service_type": "prometheus",
"name": "x",
"config": {"base_url": "https://grafana.example.com"},
"config": {"base_url": "https://prometheus.example.com"},
"secrets": {"password": "leak"},
},
)
@@ -322,9 +319,9 @@ def test_credential_key_in_config_rejected(client):
response = client.post(
"/api/services/instances",
json={
"service_type": "grafana",
"service_type": "prometheus",
"name": "x",
"config": {"base_url": "https://grafana.example.com", "api_key": "leak"},
"config": {"base_url": "https://prometheus.example.com", "api_key": "leak"},
},
)
assert response.status_code == 422
@@ -333,22 +330,22 @@ def test_credential_key_in_config_rejected(client):
def test_update_nonexistent_returns_404(client):
response = client.put(
"/api/services/instances/missing",
json=_grafana_payload(id="missing"),
json=_prometheus_payload(id="missing"),
)
assert response.status_code == 404
def test_update_id_mismatch_returns_400(client):
created = client.post("/api/services/instances", json=_grafana_payload()).json()
created = client.post("/api/services/instances", json=_prometheus_payload()).json()
response = client.put(
f"/api/services/instances/{created['id']}",
json=_grafana_payload(id="other-id"),
json=_prometheus_payload(id="other-id"),
)
assert response.status_code == 400
def test_delete_service(client):
created = client.post("/api/services/instances", json=_grafana_payload()).json()
created = client.post("/api/services/instances", json=_prometheus_payload()).json()
response = client.delete(f"/api/services/instances/{created['id']}")
assert response.status_code == 200
assert client.get("/api/services/instances").json() == []
@@ -372,7 +369,7 @@ def test_delete_service_cascades_to_widgets(client, tmp_path):
"""
store = app.dependency_overrides[get_settings_store]()
service = store.upsert_service(
{"service_type": "grafana", "name": "Grafana", "config": {"base_url": "u"}, "enabled": True}
{"service_type": "prometheus", "name": "Prometheus", "config": {"base_url": "u"}, "enabled": True}
)
# Ensure the service_id column exists and seed a referencing widget.
@@ -386,7 +383,7 @@ def test_delete_service_cascades_to_widgets(client, tmp_path):
enabled, sort_order, created_at, updated_at, service_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
("w1", "grafana", "grafana.link", "Link", "{}", 1, 0, 1, 1, service["id"]),
("w1", "prometheus", "prometheus.metric", "Link", "{}", 1, 0, 1, 1, service["id"]),
)
store.delete_service(service["id"])
@@ -399,6 +396,77 @@ def test_delete_service_cascades_to_widgets(client, tmp_path):
assert int(remaining[0]) == 0
# ---------------------------------------------------------------------------
# Harness cascade-delete (Slice 4)
# ---------------------------------------------------------------------------
def test_cascade_delete_removes_harness_data_across_concerns(tmp_path, monkeypatch):
"""Deleting a service cascades to both qbittorrent samples and media items.
Proves end-to-end cascade across both harness-managed concerns, and that
deleting one service preserves another service's data (multi-instance).
"""
monkeypatch.setenv("BACKEND_CACHE_DIR", str(tmp_path))
from media_library_viewer_api.services.media_index_impl import MediaIndex
from media_library_viewer_api.services.qbittorrent_store import QbittorrentSampleStore
from media_library_viewer_api.services.service_data import (
get_service_data_harness,
reset_service_data_harness,
)
reset_service_data_harness()
harness = get_service_data_harness()
store = SettingsStore(tmp_path / "settings.sqlite")
store.ensure_defaults()
# --- qBit: create service, add samples, delete, verify gone ---
qbit_a = store.upsert_service(
{"service_type": "qbittorrent", "name": "qA", "config": {"base_url": "http://a"}, "enabled": True}
)
qbit_b = store.upsert_service(
{"service_type": "qbittorrent", "name": "qB", "config": {"base_url": "http://b"}, "enabled": True}
)
sample_store = QbittorrentSampleStore(harness)
sample_store.append(qbit_a["id"], ts=1000, dl_speed=500, up_speed=100)
sample_store.append(qbit_b["id"], ts=1000, dl_speed=200, up_speed=50)
assert len(sample_store.window(qbit_a["id"])) == 1
assert len(sample_store.window(qbit_b["id"])) == 1
store.delete_service(qbit_a["id"])
assert sample_store.window(qbit_a["id"]) == []
assert len(sample_store.window(qbit_b["id"])) == 1 # B survives
# --- MediaIndex: create services, add items, delete, verify scoped ---
jelly_a = store.upsert_service(
{"service_type": "jellyfin", "name": "jA", "config": {"base_url": "http://ja"}, "enabled": True}
)
jelly_b = store.upsert_service(
{"service_type": "jellyfin", "name": "jB", "config": {"base_url": "http://jb"}, "enabled": True}
)
index = MediaIndex(harness.db_path("media_index"))
index.init_schema()
index.replace_items([{"id": "m1", "title": "A1"}], service_id=jelly_a["id"])
index.replace_items([{"id": "m2", "title": "B1"}], service_id=jelly_b["id"])
rows_a, total_a = index.query(service_id=jelly_a["id"])
rows_b, total_b = index.query(service_id=jelly_b["id"])
assert total_a == 1 and total_b == 1
store.delete_service(jelly_a["id"])
rows_a_after, total_a_after = index.query(service_id=jelly_a["id"])
rows_b_after, total_b_after = index.query(service_id=jelly_b["id"])
assert total_a_after == 0 # deleted
assert total_b_after == 1 # survives
reset_service_data_harness()
# ---------------------------------------------------------------------------
# Service task run history
# ---------------------------------------------------------------------------
+475 -224
View File
@@ -15,7 +15,6 @@ from media_library_viewer_api.services.settings_store import SettingsStore
from media_library_viewer_api.widgets.sources import (
AlertmanagerWidgetSource,
BackupsWidgetSource,
GrafanaWidgetSource,
JellyfinWidgetSource,
ServiceRecord,
StaticWidgetSource,
@@ -42,12 +41,12 @@ def client(tmp_path):
app.dependency_overrides.clear()
def _make_grafana_service(client, name="Production Grafana", **config_overrides):
config = {"base_url": "https://grafana.example.com"}
def _make_prometheus_service(client, name="Production Prometheus", **config_overrides):
config = {"base_url": "https://prometheus.example.com"}
config.update(config_overrides)
return client.post(
"/api/services/instances",
json={"service_type": "grafana", "name": name, "config": config, "enabled": True},
json={"service_type": "prometheus", "name": name, "config": config, "enabled": True},
).json()
@@ -93,7 +92,7 @@ def test_create_backups_widget(client):
def test_widget_filtering_by_service_id_and_scope(client):
"""Test ?service_id= and ?scope= query params on GET /api/widgets/instances."""
service = _make_grafana_service(client)
service = _make_prometheus_service(client)
# Create a dashboard-scoped (built-in) widget + a service-scoped widget.
client.post(
"/api/widgets/instances",
@@ -103,9 +102,9 @@ def test_widget_filtering_by_service_id_and_scope(client):
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"widget_kind": "metric",
"title": "Dash",
"config": {"dashboard_uid": "o"},
"config": {"promql": "up"},
},
)
@@ -151,29 +150,29 @@ def test_credential_key_in_config_rejected(client):
def test_create_service_bound_widget(client):
service = _make_grafana_service(client)
response = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"title": "Dashboard",
"config": {"dashboard_uid": "overview"},
},
)
assert response.status_code == 201
created = response.json()
assert created["service_id"] == service["id"]
assert created["widget_kind"] == "link"
def test_service_bound_widget_unknown_kind_rejected(client):
service = _make_grafana_service(client)
service = _make_prometheus_service(client)
response = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "metric",
"title": "Metrics",
"config": {"promql": "up"},
},
)
assert response.status_code == 201
created = response.json()
assert created["service_id"] == service["id"]
assert created["widget_kind"] == "metric"
def test_service_bound_widget_unknown_kind_rejected(client):
service = _make_prometheus_service(client)
response = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "nonexistent_kind",
"title": "x",
"config": {},
},
@@ -186,33 +185,23 @@ def test_service_bound_widget_service_not_found_rejected(client):
"/api/widgets/instances",
json={
"service_id": "missing",
"widget_kind": "link",
"widget_kind": "metric",
"title": "x",
"config": {"dashboard_uid": "u"},
"config": {"promql": "up"},
},
)
assert response.status_code == 422
def test_service_bound_widget_invalid_config_rejected(client):
service = _make_grafana_service(client)
service = _make_prometheus_service(client)
response = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"widget_kind": "metric",
"title": "x",
"config": {"dashboard_uid": ""}, # empty still validates; use bad type
},
)
# Empty string passes Pydantic; force a real failure with a bad type.
response = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"title": "x",
"config": {"dashboard_uid": 123},
"config": {"promql": 123}, # bad type: promql must be a string
},
)
assert response.status_code == 422
@@ -280,31 +269,15 @@ def test_fetch_backups_widget_data(client):
assert "total_jobs" in response.json()["data"]
def test_fetch_grafana_link_widget_data(client):
service = _make_grafana_service(client)
created = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"title": "Dashboard",
"config": {"dashboard_uid": "overview", "panel_id": 2},
},
).json()
response = client.get(f"/api/widgets/instances/{created['id']}/data")
assert response.status_code == 200
assert response.json()["data"]["url"] == "https://grafana.example.com/d/overview?viewPanel=2"
def test_fetch_widget_service_not_found(client):
service = _make_grafana_service(client)
service = _make_prometheus_service(client)
created = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"widget_kind": "metric",
"title": "x",
"config": {"dashboard_uid": "u"},
"config": {"promql": "up"},
},
).json()
# Deleting the service cascade-deletes its widgets, so the widget is gone.
@@ -314,22 +287,22 @@ def test_fetch_widget_service_not_found(client):
def test_fetch_widget_service_disabled(client):
service = _make_grafana_service(client)
service = _make_prometheus_service(client)
created = client.post(
"/api/widgets/instances",
json={
"service_id": service["id"],
"widget_kind": "link",
"widget_kind": "metric",
"title": "x",
"config": {"dashboard_uid": "u"},
"config": {"promql": "up"},
},
).json()
client.put(
f"/api/services/instances/{service['id']}",
json={
"service_type": "grafana",
"service_type": "prometheus",
"name": service["name"],
"config": {"base_url": "https://grafana.example.com"},
"config": {"base_url": "https://prometheus.example.com"},
"enabled": False,
},
)
@@ -347,23 +320,6 @@ def test_fetch_widget_not_found(client):
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_grafana_adapter_builds_url():
adapter = GrafanaWidgetSource()
service = ServiceRecord(id="s", service_type="grafana", name="g", config={"base_url": "http://g:3000"})
result = await adapter.fetch(service, "link", {"dashboard_uid": "ov"})
assert result["url"] == "http://g:3000/d/ov"
result = await adapter.fetch(service, "link", {"dashboard_uid": "ov", "panel_id": 4})
assert result["url"] == "http://g:3000/d/ov?viewPanel=4"
@pytest.mark.asyncio
async def test_grafana_adapter_missing_service():
adapter = GrafanaWidgetSource()
result = await adapter.fetch(None, "link", {"dashboard_uid": "ov"})
assert "error" in result
@pytest.mark.asyncio
async def test_alertmanager_adapter_summarizes_alerts():
adapter = AlertmanagerWidgetSource()
@@ -498,7 +454,7 @@ async def test_ssh_task_adapter_records_history_on_run(client):
# ---------------------------------------------------------------------------
# New widget kind tests (jellyfin now_playing + grafana panel)
# New widget kind tests (jellyfin now_playing)
# ---------------------------------------------------------------------------
@@ -511,158 +467,69 @@ def test_jellyfin_definition_has_now_playing_widget():
assert "activity" in kinds
def test_grafana_definition_has_chart_widget():
from media_library_viewer_api.integrations.registry import get_service_definition
definition = get_service_definition("grafana")
kinds = {wk.kind for wk in definition.widget_kinds}
assert "chart" in kinds
assert "link" in kinds
@pytest.mark.asyncio
async def test_grafana_adapter_chart_queries_datasource():
"""Chart widget should POST to /api/ds/query and normalize the response."""
from unittest.mock import MagicMock, patch
async def test_prometheus_chart_adapter_runs_range_query():
"""SC-101: chart kind hits /api/v1/query_range and returns {series}."""
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = GrafanaWidgetSource()
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="grafana",
name="g",
config={"base_url": "http://g:3000", "timeout_seconds": 5},
secrets={"api_key": "tok"},
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090", "timeout_seconds": 5},
)
mock_resp = MagicMock()
mock_resp.json.return_value = {
"results": {
"A": {
"frames": [
payload = SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"data": {
"result": [
{
"data": {"values": [[1000, 2000], [0.5, 0.8]]},
"schema": {"fields": [{"name": "Time"}, {"name": "cpu_usage"}]},
"metric": {"__name__": "up", "instance": "h:9100"},
"values": [[100, "1"], [130, "1"]],
}
]
}
}
}
mock_resp.raise_for_status = MagicMock()
with patch("media_library_viewer_api.widgets.sources.requests.post", return_value=mock_resp):
result = await adapter.fetch(
service,
"chart",
{"query": "rate(cpu[5m])", "datasource_uid": "prometheus"},
)
},
)
with patch("media_library_viewer_api.widgets.sources.requests.get", return_value=payload) as mock_get:
result = await adapter.fetch(service, "chart", {"promql": "up", "window": "1h"})
# query_range endpoint + window-derived start/end/step params.
call = mock_get.call_args
assert call.args[0].endswith("/api/v1/query_range")
params = call.kwargs["params"]
assert params["query"] == "up"
assert {"start", "end", "step"}.issubset(params)
# {series} shape with the shared normalization (label drops __name__).
assert "series" in result
assert len(result["series"]) == 1
assert result["series"][0]["label"] == "cpu_usage"
assert result["series"][0]["points"] == [
{"t": 1000, "v": 0.5},
{"t": 2000, "v": 0.8},
]
assert result["series"][0]["label"] == "instance=h:9100"
assert result["series"][0]["points"] == [{"t": 100, "v": 1.0}, {"t": 130, "v": 1.0}]
@pytest.mark.asyncio
async def test_grafana_adapter_chart_extracts_prometheus_labels():
"""Multiple Prometheus series should get unique labels from frame metadata."""
from unittest.mock import MagicMock, patch
async def test_prometheus_chart_adapter_requires_promql():
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = GrafanaWidgetSource()
service = ServiceRecord(
id="s",
service_type="grafana",
name="g",
config={"base_url": "http://g:3000", "timeout_seconds": 5},
secrets={"api_key": "tok"},
)
mock_resp = MagicMock()
mock_resp.json.return_value = {
"results": {
"A": {
"frames": [
{
"data": {"values": [[1000], [0.5]]},
"schema": {
"fields": [
{"name": "Time"},
{
"name": "Value",
"labels": {
"instance": "server1:9100",
"mode": "iowait",
},
},
]
},
},
{
"data": {"values": [[1000], [0.3]]},
"schema": {
"fields": [
{"name": "Time"},
{
"name": "Value",
"labels": {
"instance": "server2:9100",
"mode": "iowait",
},
},
]
},
},
]
}
}
}
mock_resp.raise_for_status = MagicMock()
with patch("media_library_viewer_api.widgets.sources.requests.post", return_value=mock_resp):
result = await adapter.fetch(service, "chart", {"query": "rate(cpu[5m])"})
assert len(result["series"]) == 2
assert result["series"][0]["label"] == "instance=server1:9100 mode=iowait"
assert result["series"][1]["label"] == "instance=server2:9100 mode=iowait"
adapter = PrometheusWidgetSource()
service = ServiceRecord(id="s", service_type="prometheus", name="p", config={"base_url": "http://p:9090"})
result = await adapter.fetch(service, "chart", {"promql": ""})
assert result == {"error": "promql is required"}
@pytest.mark.asyncio
async def test_grafana_adapter_chart_requires_api_key():
adapter = GrafanaWidgetSource()
service = ServiceRecord(
id="s",
service_type="grafana",
name="g",
config={"base_url": "http://g:3000"},
)
result = await adapter.fetch(service, "chart", {"query": "up"})
assert "error" in result
assert "api_key" in result["error"]
@pytest.mark.asyncio
async def test_grafana_adapter_chart_handles_http_failure():
from unittest.mock import patch
async def test_prometheus_chart_adapter_degrades_on_http_error():
"""SC-103: a connection error returns {error} rather than raising."""
import requests as req_mod
adapter = GrafanaWidgetSource()
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="grafana",
name="g",
config={"base_url": "http://g:3000", "timeout_seconds": 2},
secrets={"api_key": "tok"},
id="s", service_type="prometheus", name="p", config={"base_url": "http://p:9090", "timeout_seconds": 2}
)
with patch(
"media_library_viewer_api.widgets.sources.requests.post",
side_effect=req_mod.ConnectionError("refused"),
):
result = await adapter.fetch(service, "chart", {"query": "up"})
with patch("media_library_viewer_api.widgets.sources.requests.get", side_effect=req_mod.ConnectionError("refused")):
result = await adapter.fetch(service, "chart", {"promql": "up", "window": "1h"})
assert "error" in result
assert "failed" in result["error"].lower()
@@ -760,23 +627,23 @@ def test_widget_reference_lifecycle(widget_ref_client):
"""Create a widget, reference it on 'main', verify it appears, delete reference."""
client, store = widget_ref_client
# Create a service-bound widget (simulating one on a Grafana Overview).
# Create a service-bound widget (simulating one on a Prometheus service).
store.upsert_service(
{
"service_type": "grafana",
"name": "Grafana",
"config": {"base_url": "https://grafana.example.com"},
"service_type": "prometheus",
"name": "Prometheus",
"config": {"base_url": "https://prometheus.example.com"},
"secrets": {"api_key": "tok"},
"enabled": True,
},
)
service = store.list_services("grafana")[0]
service = store.list_services("prometheus")[0]
widget = store.upsert_widget(
{
"service_id": service["id"],
"widget_kind": "chart",
"title": "CPU IOWait",
"config": {"query": "rate(cpu[5m])", "datasource_uid": "prometheus"},
"config": {"promql": "rate(cpu[5m])", "window": "1h"},
"enabled": True,
"sort_order": 0,
}
@@ -821,20 +688,20 @@ def test_widget_reference_detach(widget_ref_client):
store.upsert_service(
{
"service_type": "grafana",
"name": "Grafana",
"config": {"base_url": "https://grafana.example.com"},
"service_type": "prometheus",
"name": "Prometheus",
"config": {"base_url": "https://prometheus.example.com"},
"secrets": {"api_key": "tok"},
"enabled": True,
},
)
service = store.list_services("grafana")[0]
service = store.list_services("prometheus")[0]
widget = store.upsert_widget(
{
"service_id": service["id"],
"widget_kind": "chart",
"title": "Memory",
"config": {"query": "mem", "datasource_uid": "prometheus"},
"config": {"promql": "mem", "window": "1h"},
"enabled": True,
"sort_order": 0,
}
@@ -857,7 +724,7 @@ def test_widget_reference_detach(widget_ref_client):
assert cloned["title"] == "Memory"
assert cloned["widget_kind"] == "chart"
assert cloned["service_id"] == service["id"] # Fix 2: preserves service binding
assert cloned["config"]["query"] == "mem"
assert cloned["config"]["promql"] == "mem"
assert cloned["id"] != widget["id"] # new independent widget
# Reference is gone.
@@ -924,3 +791,387 @@ def test_widget_reference_update_sort_order(widget_ref_client):
# Widget instances themselves are unchanged.
assert store.get_widget(widget_a["id"])["sort_order"] == 0
assert store.get_widget(widget_b["id"])["sort_order"] == 1
# ---------------------------------------------------------------------------
# Prometheus gauge + mean adapter tests (SC-109..SC-114)
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_prometheus_gauge_adapter_returns_scalar():
"""SC-109: gauge kind hits /api/v1/query and returns {value, thresholds}."""
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090", "timeout_seconds": 5},
)
payload = SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"data": {
"result": [
{"metric": {"__name__": "cpu"}, "value": [100, "0.75"]},
]
}
},
)
with patch("media_library_viewer_api.widgets.sources.requests.get", return_value=payload) as mock_get:
result = await adapter.fetch(
service,
"gauge",
{
"promql": "cpu_usage",
"warn_at": 0.8,
"crit_at": 0.95,
"unit": "%",
},
)
call = mock_get.call_args
assert call.args[0].endswith("/api/v1/query")
assert call.kwargs["params"]["query"] == "cpu_usage"
assert result["value"] == 0.75
assert result["warn_at"] == 0.8
assert result["crit_at"] == 0.95
assert result["unit"] == "%"
@pytest.mark.asyncio
async def test_prometheus_gauge_adapter_rejects_multi_series():
"""SC-111: gauge must be scalar-only; multi-series returns error."""
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090"},
)
payload = SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"data": {
"result": [
{"metric": {"instance": "a"}, "value": [100, "1"]},
{"metric": {"instance": "b"}, "value": [100, "2"]},
]
}
},
)
with patch("media_library_viewer_api.widgets.sources.requests.get", return_value=payload):
result = await adapter.fetch(service, "gauge", {"promql": "up"})
assert "error" in result
assert "single-series" in result["error"].lower()
@pytest.mark.asyncio
async def test_prometheus_gauge_adapter_requires_promql():
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090"},
)
result = await adapter.fetch(service, "gauge", {"promql": ""})
assert result == {"error": "promql is required"}
@pytest.mark.asyncio
async def test_prometheus_mean_adapter_computes_average():
"""SC-112: mean kind averages non-null values over the window."""
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090", "timeout_seconds": 5},
)
payload = SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"data": {
"result": [
{
"metric": {"__name__": "cpu"},
"values": [[100, "1.0"], [130, "2.0"], [160, "3.0"]],
}
]
}
},
)
with patch("media_library_viewer_api.widgets.sources.requests.get", return_value=payload):
result = await adapter.fetch(service, "mean", {"promql": "cpu", "window": "1h"})
assert result["value"] == 2.0
assert result["unit"] is None
@pytest.mark.asyncio
async def test_prometheus_mean_adapter_rejects_multi_series():
"""SC-114: mean must be scalar-only; multi-series returns error."""
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090"},
)
payload = SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"data": {
"result": [
{"metric": {"instance": "a"}, "values": [[100, "1"]]},
{"metric": {"instance": "b"}, "values": [[100, "2"]]},
]
}
},
)
with patch("media_library_viewer_api.widgets.sources.requests.get", return_value=payload):
result = await adapter.fetch(service, "mean", {"promql": "up", "window": "1h"})
assert "error" in result
assert "single-series" in result["error"].lower()
@pytest.mark.asyncio
async def test_prometheus_mean_adapter_skips_nan_values():
"""SC-112: NaN / Inf values are excluded from the mean computation."""
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090"},
)
payload = SimpleNamespace(
raise_for_status=lambda: None,
json=lambda: {
"data": {
"result": [
{
"metric": {},
"values": [[100, "2.0"], [130, "NaN"], [160, "4.0"]],
}
]
}
},
)
with patch("media_library_viewer_api.widgets.sources.requests.get", return_value=payload):
result = await adapter.fetch(service, "mean", {"promql": "up", "window": "1h"})
# (2.0 + 4.0) / 2 = 3.0 (NaN excluded)
assert result["value"] == 3.0
@pytest.mark.asyncio
async def test_prometheus_mean_adapter_requires_promql():
from media_library_viewer_api.widgets.sources import PrometheusWidgetSource
adapter = PrometheusWidgetSource()
service = ServiceRecord(
id="s",
service_type="prometheus",
name="p",
config={"base_url": "http://p:9090"},
)
result = await adapter.fetch(service, "mean", {"promql": ""})
assert result == {"error": "promql is required"}
# ---------------------------------------------------------------------------
# qBittorrent widget source adapter
# ---------------------------------------------------------------------------
def _fake_qbit_maindata():
"""Return a mock maindata response (server_state + torrents dict)."""
return {
"server_state": {"dl_info_speed": 500000, "up_info_speed": 100000},
"torrents": {
"h1": {
"name": "Movie.mkv",
"state": "downloading",
"size": 1000,
"progress": 0.5,
"dlspeed": 500,
"upspeed": 10,
},
"h2": {
"name": "Show.mkv",
"state": "uploading",
"size": 2000,
"progress": 1.0,
"dlspeed": 0,
"upspeed": 100,
},
"h3": {
"name": "Queued",
"state": "queuedDL",
"size": 3000,
"progress": 0.0,
"dlspeed": 0,
"upspeed": 0,
},
"h4": {
"name": "Paused",
"state": "pausedDL",
"size": 4000,
"progress": 0.3,
"dlspeed": 0,
"upspeed": 0,
},
},
}
@pytest.mark.asyncio
async def test_qbittorrent_totals_counts_all_torrents():
"""Totals kind returns count of all listed items + by_state breakdown."""
from media_library_viewer_api.widgets.sources import QbittorrentWidgetSource
adapter = QbittorrentWidgetSource()
service = ServiceRecord(
id="svc-1",
service_type="qbittorrent",
name="qbit",
config={"base_url": "http://qbit:8080", "timeout_seconds": 5},
secrets={"username": "admin", "password": "pass"},
)
with patch("media_library_viewer_api.widgets.sources.QbittorrentClient") as mock_client:
mock_client.return_value.maindata.return_value = _fake_qbit_maindata()
result = await adapter.fetch(service, "totals", {})
assert result["total"] == 4
assert result["by_state"]["downloading"] == 1
assert result["by_state"]["uploading"] == 1
assert result["by_state"]["queuedDL"] == 1
assert result["by_state"]["pausedDL"] == 1
@pytest.mark.asyncio
async def test_qbittorrent_active_filters_dl_ul_only():
"""Active kind returns only downloading/uploading torrents (Q3)."""
from media_library_viewer_api.widgets.sources import QbittorrentWidgetSource
adapter = QbittorrentWidgetSource()
service = ServiceRecord(
id="svc-1",
service_type="qbittorrent",
name="qbit",
config={"base_url": "http://qbit:8080", "timeout_seconds": 5},
secrets={"username": "admin", "password": "pass"},
)
with patch("media_library_viewer_api.widgets.sources.QbittorrentClient") as mock_client:
mock_client.return_value.maindata.return_value = _fake_qbit_maindata()
result = await adapter.fetch(service, "active", {})
active = result["torrents"]
assert len(active) == 2
names = [t["name"] for t in active]
assert "Movie.mkv" in names
assert "Show.mkv" in names
# Queued and paused are excluded
assert "Queued" not in names
assert "Paused" not in names
@pytest.mark.asyncio
async def test_qbittorrent_speed_appends_and_returns_series(tmp_path):
"""Speed kind appends a sample and returns {series} with two labeled series."""
from media_library_viewer_api.services.qbittorrent_store import QBITTORRENT_CONCERN, QbittorrentSampleStore
from media_library_viewer_api.services.service_data import ServiceDataHarness
from media_library_viewer_api.widgets.sources import QbittorrentWidgetSource
# Isolated harness so we don't pollute the real DB
harness = ServiceDataHarness(base_dir=str(tmp_path))
harness.register(QBITTORRENT_CONCERN)
harness.run_migrations()
adapter = QbittorrentWidgetSource()
service = ServiceRecord(
id="svc-speed",
service_type="qbittorrent",
name="qbit",
config={"base_url": "http://qbit:8080", "timeout_seconds": 5},
secrets={"username": "admin", "password": "pass"},
)
with (
patch("media_library_viewer_api.widgets.sources.QbittorrentClient") as mock_client,
patch("media_library_viewer_api.widgets.sources.QbittorrentSampleStore") as mock_store_cls,
):
mock_client.return_value.maindata.return_value = _fake_qbit_maindata()
# Wire the mock store to a real isolated store
real_store = QbittorrentSampleStore(harness)
mock_store_cls.return_value = real_store
result = await adapter.fetch(service, "speed", {})
assert "series" in result
labels = [s["label"] for s in result["series"]]
assert labels == ["download", "upload"]
# The sample just appended should be present
dl_points = result["series"][0]["points"]
assert len(dl_points) >= 1
# timestamps multiplied by 1000 for JS epoch
assert dl_points[-1]["v"] == 500000
@pytest.mark.asyncio
async def test_qbittorrent_adapter_missing_service():
from media_library_viewer_api.widgets.sources import QbittorrentWidgetSource
adapter = QbittorrentWidgetSource()
result = await adapter.fetch(None, "totals", {})
assert "error" in result
@pytest.mark.asyncio
async def test_qbittorrent_adapter_missing_credentials():
from media_library_viewer_api.widgets.sources import QbittorrentWidgetSource
adapter = QbittorrentWidgetSource()
service = ServiceRecord(
id="s",
service_type="qbittorrent",
name="qbit",
config={"base_url": "http://qbit:8080"},
secrets={"username": "", "password": ""},
)
result = await adapter.fetch(service, "totals", {})
assert "error" in result
@pytest.mark.asyncio
async def test_qbittorrent_adapter_timeout():
"""A timeout returns {error} rather than raising."""
from media_library_viewer_api.widgets.sources import QbittorrentWidgetSource
adapter = QbittorrentWidgetSource()
service = ServiceRecord(
id="s",
service_type="qbittorrent",
name="qbit",
config={"base_url": "http://qbit:8080", "timeout_seconds": 1},
secrets={"username": "admin", "password": "pass"},
)
with patch("media_library_viewer_api.widgets.sources.QbittorrentClient") as mock_client:
import asyncio as _asyncio
async def _slow(*a, **kw):
await _asyncio.sleep(10)
# Make to_thread hang so wait_for times out
mock_client.return_value.maindata.side_effect = lambda: (_ for _ in ()).throw(TimeoutError())
result = await adapter.fetch(service, "totals", {})
assert "error" in result
-4
View File
@@ -32,7 +32,6 @@ import type {
DashboardShortcutInput,
AlertmanagerAlertSummary,
AlertmanagerStatus,
GrafanaStatus,
PrometheusStatus,
PrometheusTarget,
} from "../types";
@@ -299,9 +298,6 @@ export const fetchAlertmanagerAlerts = () =>
export const fetchAlertmanagerStatus = () =>
get<AlertmanagerStatus>("/api/monitoring/alertmanager-status");
export const fetchGrafanaStatus = () =>
get<GrafanaStatus>("/api/monitoring/grafana-status");
export const fetchPrometheusStatus = () =>
get<PrometheusStatus>("/api/monitoring/prometheus-status");
@@ -0,0 +1,95 @@
import {
LineChart,
Line,
XAxis,
YAxis,
CartesianGrid,
Tooltip,
ResponsiveContainer,
} from "recharts";
export interface SeriesPoint {
t: number;
v: number | null;
}
export interface ChartSeries {
label: string;
points: SeriesPoint[];
}
/** Merge multiple time-series into a single recharts-friendly array. */
function mergeSeries(series: ChartSeries[]): Record<string, unknown>[] {
const map = new Map<number, Record<string, unknown>>();
for (const s of series) {
for (const p of s.points) {
const existing = map.get(p.t) ?? { time: p.t };
existing[s.label] = p.v;
map.set(p.t, existing);
}
}
return [...map.values()].sort(
(a, b) => (a.time as number) - (b.time as number),
);
}
function formatTime(ms: number): string {
return new Date(ms).toLocaleTimeString([], {
hour: "2-digit",
minute: "2-digit",
});
}
const CHART_COLORS = [
"var(--color-chart-1)",
"var(--color-chart-2)",
"var(--color-chart-3)",
"var(--color-chart-4)",
"var(--color-chart-5)",
];
interface LineSeriesChartProps {
series: ChartSeries[];
height?: number;
}
/** Shared recharts line-chart renderer used by PrometheusChart + qBit speed widgets. */
export function LineSeriesChart({
series,
height = 300,
}: LineSeriesChartProps) {
return (
<ResponsiveContainer width="100%" height={height}>
<LineChart data={mergeSeries(series)}>
<CartesianGrid strokeDasharray="3 3" className="stroke-muted" />
<XAxis
dataKey="time"
tickFormatter={formatTime}
tick={{ fontSize: 11 }}
className="fill-muted-foreground"
/>
<YAxis tick={{ fontSize: 11 }} className="fill-muted-foreground" />
<Tooltip
labelFormatter={(label) => formatTime(Number(label))}
contentStyle={{
backgroundColor: "hsl(var(--popover))",
border: "1px solid hsl(var(--border))",
borderRadius: "0.5rem",
color: "hsl(var(--popover-foreground))",
}}
/>
{series.map((s, i) => (
<Line
key={s.label}
type="monotone"
dataKey={s.label}
stroke={CHART_COLORS[i % CHART_COLORS.length]}
dot={false}
strokeWidth={2}
connectNulls
/>
))}
</LineChart>
</ResponsiveContainer>
);
}
@@ -0,0 +1,34 @@
import { describe, it, expect } from "vitest";
import { render } from "@testing-library/react";
import { LineSeriesChart } from "../LineSeriesChart";
import type { ChartSeries } from "../LineSeriesChart";
describe("LineSeriesChart", () => {
it("renders without crashing with series data", () => {
const series: ChartSeries[] = [
{
label: "cpu",
points: [
{ t: 1000, v: 0.5 },
{ t: 2000, v: 0.8 },
],
},
];
const { container } = render(<LineSeriesChart series={series} />);
// ResponsiveContainer renders a wrapper div even in jsdom
expect(container.firstChild).not.toBeNull();
});
it("renders without crashing with empty series", () => {
const { container } = render(<LineSeriesChart series={[]} />);
expect(container.firstChild).not.toBeNull();
});
it("renders with custom height", () => {
const series: ChartSeries[] = [{ label: "dl", points: [{ t: 1, v: 1 }] }];
const { container } = render(
<LineSeriesChart series={series} height={200} />,
);
expect(container.firstChild).not.toBeNull();
});
});
-11
View File
@@ -2,7 +2,6 @@ import { useQuery } from "@tanstack/react-query";
import {
fetchAlertmanagerAlerts,
fetchAlertmanagerStatus,
fetchGrafanaStatus,
fetchPrometheusStatus,
fetchPrometheusTargets,
fetchMonitoringMachines,
@@ -28,16 +27,6 @@ export function useAlertmanagerStatus() {
});
}
export function useGrafanaStatus() {
return useQuery({
queryKey: ["observability", "grafana-status"],
queryFn: fetchGrafanaStatus,
retry: 2,
staleTime: 10_000,
refetchInterval: 30_000,
});
}
export function usePrometheusStatus() {
return useQuery({
queryKey: ["observability", "prometheus-status"],
+2 -2
View File
@@ -23,7 +23,7 @@
--color-border: #e2e8f0;
--color-input: #e2e8f0;
--color-ring: #4f8cff;
/* Status / Grafana-link semantic cues single source of truth for Badges.
/* Status semantic cues single source of truth for Badges.
chart-1=info/brand, chart-2=success, chart-3=warning,
chart-4=destructive, chart-5=neutral-accent. */
--color-chart-1: #4f8cff;
@@ -62,7 +62,7 @@
--color-border: #334155;
--color-input: #334155;
--color-ring: #4f8cff;
/* Status / Grafana-link semantic cues single source of truth for Badges.
/* Status semantic cues single source of truth for Badges.
chart-1=info/brand, chart-2=success, chart-3=warning,
chart-4=destructive, chart-5=neutral-accent. */
--color-chart-1: #4f8cff;
@@ -21,13 +21,9 @@ describe("navEntries", () => {
it("returns all observability entries", () => {
const entries = configuredNavEntries(
new Set(["alertmanager", "grafana", "prometheus"]),
new Set(["alertmanager", "prometheus"]),
);
expect(entries.map((e) => e.label)).toEqual([
"Alertmanager",
"Grafana",
"Prometheus",
]);
expect(entries.map((e) => e.label)).toEqual(["Alertmanager", "Prometheus"]);
});
it("returns Backups + Authentik when configured", () => {
-7
View File
@@ -10,7 +10,6 @@ import {
Activity,
DatabaseBackup,
GanttChartSquare,
Link2,
Monitor,
Server,
Users,
@@ -49,12 +48,6 @@ export const SERVICE_TYPE_NAV_ENTRIES: NavEntry[] = [
icon: Activity,
path: "/services/alertmanager",
},
{
serviceType: "grafana",
label: "Grafana",
icon: Link2,
path: "/services/grafana",
},
{
serviceType: "prometheus",
label: "Prometheus",
+12 -10
View File
@@ -12,18 +12,20 @@ describe("service registry", () => {
it("registers the backend service types", () => {
expect(Object.keys(SERVICE_REGISTRY).sort()).toEqual([
"alertmanager",
"grafana",
"jellyfin",
"nextcloud",
"prometheus",
"qbittorrent",
"ssh_tasks",
]);
});
it("binds widget kinds per service", () => {
expect(SERVICE_REGISTRY.grafana.widgets.map((w) => w.kind)).toEqual([
"link",
expect(SERVICE_REGISTRY.prometheus.widgets.map((w) => w.kind)).toEqual([
"metric",
"chart",
"gauge",
"mean",
]);
expect(SERVICE_REGISTRY.alertmanager.widgets.map((w) => w.kind)).toEqual([
"active_alerts",
@@ -38,12 +40,12 @@ describe("service registry", () => {
expect(Object.keys(BUILTIN_WIDGETS).sort()).toEqual(["backups", "static"]);
});
it("resolves a service-bound widget via the services list", () => {
it("resolves a prometheus metric widget via the services list", () => {
const widget: WidgetInstance = {
id: "w1",
service_id: "s1",
widget_kind: "link",
title: "Dashboard",
widget_kind: "metric",
title: "Metric",
config: {},
enabled: true,
sort_order: 0,
@@ -53,9 +55,9 @@ describe("service registry", () => {
const services: ServiceInstance[] = [
{
id: "s1",
service_type: "grafana",
name: "Grafana",
config: { base_url: "https://grafana.example.com" },
service_type: "prometheus",
name: "Prometheus",
config: { base_url: "https://prometheus.example.com" },
secrets_set: { api_key: true },
enabled: true,
created_at: 0,
@@ -64,7 +66,7 @@ describe("service registry", () => {
];
const resolved = resolveWidget(widget, services);
expect(resolved).toBeDefined();
expect(resolved?.refreshIntervalMs).toBe(0);
expect(resolved?.refreshIntervalMs).toBe(30_000);
});
it("resolves an alertmanager active_alerts widget", () => {
+109 -58
View File
@@ -1,11 +1,15 @@
import type { ComponentType } from "react";
import { AlertmanagerAlertsWidget } from "../widgets/AlertmanagerAlertsWidget";
import { BackupsWidget } from "../widgets/BackupsWidget";
import { GrafanaLinkWidget } from "../widgets/GrafanaLinkWidget";
import { GrafanaChartWidget } from "../widgets/GrafanaChartWidget";
import { PrometheusChartWidget } from "../widgets/PrometheusChartWidget";
import { PrometheusGaugeWidget } from "../widgets/PrometheusGaugeWidget";
import { PrometheusMeanWidget } from "../widgets/PrometheusMeanWidget";
import { JellyfinWidget } from "../widgets/JellyfinWidget";
import { JellyfinNowPlayingWidget } from "../widgets/JellyfinNowPlayingWidget";
import { PrometheusMetricWidget } from "../widgets/PrometheusMetricWidget";
import { QbittorrentActiveTorrentsWidget } from "../widgets/QbittorrentActiveTorrentsWidget";
import { QbittorrentSpeedWidget } from "../widgets/QbittorrentSpeedWidget";
import { QbittorrentTotalsWidget } from "../widgets/QbittorrentTotalsWidget";
import { SshTaskWidget } from "../widgets/SshTaskWidget";
import { StaticWidget } from "../widgets/StaticWidget";
import type {
@@ -67,62 +71,6 @@ export const SERVICE_REGISTRY: Record<string, ServiceBinding> = {
},
],
},
grafana: {
serviceType: "grafana",
name: "Grafana",
description: "Dashboards, metrics, and logs.",
widgets: [
{
kind: "link",
name: "Dashboard link",
description: "Deep-link to a Grafana dashboard or panel.",
refreshIntervalMs: 0,
defaultConfig: { dashboard_uid: "" },
configSchema: {
type: "object",
properties: {
dashboard_uid: { type: "string" },
panel_id: { type: "integer" },
},
required: ["dashboard_uid"],
},
component: GrafanaLinkWidget,
},
{
kind: "chart",
name: "Chart",
description: "Live time-series chart from a Grafana datasource query.",
refreshIntervalMs: 60_000,
defaultConfig: {
datasource_uid: "prometheus",
query: "",
from_ts: "now-1h",
to_ts: "now",
interval_ms: 30_000,
max_data_points: 100,
},
configSchema: {
type: "object",
properties: {
datasource_uid: {
type: "string",
description: "Grafana datasource UID (e.g. 'prometheus')",
},
query: {
type: "string",
description: "Query expression (e.g. PromQL)",
},
from_ts: { type: "string" },
to_ts: { type: "string" },
interval_ms: { type: "integer" },
max_data_points: { type: "integer" },
},
required: ["query"],
},
component: GrafanaChartWidget,
},
],
},
prometheus: {
serviceType: "prometheus",
name: "Prometheus",
@@ -141,6 +89,109 @@ export const SERVICE_REGISTRY: Record<string, ServiceBinding> = {
},
component: PrometheusMetricWidget,
},
{
kind: "chart",
name: "Chart",
description: "Multi-series line chart from a PromQL range query.",
refreshIntervalMs: 60_000,
defaultConfig: { promql: "", window: "1h" },
configSchema: {
type: "object",
properties: {
promql: {
type: "string",
description: "PromQL range query expression",
},
window: {
type: "string",
description: "Time window preset (1h, 6h, 24h, 7d)",
},
},
required: ["promql"],
},
component: PrometheusChartWidget,
},
{
kind: "gauge",
name: "Gauge",
description:
"Instant query rendered as a gauge with optional threshold bands.",
refreshIntervalMs: 30_000,
defaultConfig: { promql: "" },
configSchema: {
type: "object",
properties: {
promql: {
type: "string",
description: "PromQL instant query (must return a single scalar)",
},
warn_at: { type: "number", description: "Warning threshold" },
crit_at: { type: "number", description: "Critical threshold" },
min: { type: "number" },
max: { type: "number" },
unit: { type: "string" },
},
required: ["promql"],
},
component: PrometheusGaugeWidget,
},
{
kind: "mean",
name: "Mean",
description: "Average value of a PromQL query over a time window.",
refreshIntervalMs: 60_000,
defaultConfig: { promql: "", window: "1h" },
configSchema: {
type: "object",
properties: {
promql: {
type: "string",
description: "PromQL range query (must return a single series)",
},
window: {
type: "string",
description: "Time window preset (1h, 6h, 24h, 7d)",
},
unit: { type: "string" },
},
required: ["promql"],
},
component: PrometheusMeanWidget,
},
],
},
qbittorrent: {
serviceType: "qbittorrent",
name: "qBittorrent",
description: "Torrent client activity, speeds, and item counts.",
widgets: [
{
kind: "totals",
name: "Totals",
description: "Count of all listed torrents, broken down by state.",
refreshIntervalMs: 30_000,
defaultConfig: {},
configSchema: { type: "object", properties: {}, required: [] },
component: QbittorrentTotalsWidget,
},
{
kind: "active",
name: "Active torrents",
description: "Torrents currently downloading or uploading.",
refreshIntervalMs: 15_000,
defaultConfig: {},
configSchema: { type: "object", properties: {}, required: [] },
component: QbittorrentActiveTorrentsWidget,
},
{
kind: "speed",
name: "Speed chart",
description: "Live download/upload speed over a short window.",
refreshIntervalMs: 5_000,
defaultConfig: {},
configSchema: { type: "object", properties: {}, required: [] },
component: QbittorrentSpeedWidget,
},
],
},
jellyfin: {
+1 -1
View File
@@ -65,7 +65,7 @@ const SECTION_META: Record<
custom: { label: "Custom", icon: LayoutDashboard },
};
const OBSERVABILITY_TYPES = new Set(["alertmanager", "prometheus", "grafana"]);
const OBSERVABILITY_TYPES = new Set(["alertmanager", "prometheus"]);
function widgetSection(
widget: WidgetInstance,
+2 -2
View File
@@ -548,8 +548,8 @@ export function ServicesPage() {
{services.length === 0 ? (
<Alert>
<AlertDescription>
No services yet. Add a Grafana, Prometheus, Jellyfin, Nextcloud,
or SSH task runner.
No services yet. Add a Prometheus, Jellyfin, Nextcloud, or SSH
task runner.
</AlertDescription>
</Alert>
) : (
@@ -1,194 +0,0 @@
/**
* Grafana Links tab (spec R2.4, R8.2).
*
* Lifts the Grafana deep-link content from the old cross-service
* ObservabilityPage into an instance-scoped tab. Shows service health + the
* configured Grafana deep-links (node-exporter dashboard, Loki logs per
* machine).
*
* The hooks (useGrafanaStatus, useMonitoringMachines) are global /
* first-configured for now. Wiring `instance.id` into the status hook is a
* follow-up. The machine links use the configured Grafana base_url from the
* instance's config.
*/
import { useMemo, useState } from "react";
import { Link } from "react-router-dom";
import { Activity, ExternalLink, Gauge, ServerOff } from "lucide-react";
import {
useGrafanaStatus,
useMonitoringMachines,
} from "../../hooks/useObservability";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert";
import { Button } from "@/components/ui/button";
import { Skeleton } from "@/components/ui/skeleton";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import type { ServiceInstance } from "../../types";
function GrafanaLinkCard({
title,
description,
href,
}: {
title: string;
description: string;
href: string;
}) {
return (
<div className="rounded-md border p-4">
<div className="flex flex-col gap-3 sm:flex-row sm:items-center sm:justify-between">
<div>
<div className="font-medium">{title}</div>
<div className="text-sm text-muted-foreground">{description}</div>
</div>
<Button variant="outline" size="sm" asChild>
<a
href={href}
target="_blank"
rel="noopener noreferrer"
className="gap-1"
>
Open in Grafana
<ExternalLink className="h-3 w-3" />
</a>
</Button>
</div>
</div>
);
}
export function LinksTab({ instance }: { instance: ServiceInstance }) {
const { data: status, isLoading, error } = useGrafanaStatus();
const { data: machines = [], isLoading: machinesLoading } =
useMonitoringMachines();
const [selectedMachineId, setSelectedMachineId] = useState("");
const grafanaBaseUrl =
(instance.config?.base_url as string | undefined) ?? "";
const selectedMachine = useMemo(
() =>
machines.find((m) => m.id === selectedMachineId) ?? machines[0] ?? null,
[machines, selectedMachineId],
);
const nodeExporterDashboardUrl = useMemo(() => {
if (!selectedMachine || !grafanaBaseUrl) return "";
const inst = `${selectedMachine.host || "localhost"}:9100`;
return `${grafanaBaseUrl}/d/node-exporter-overview/node-exporter-overview?kiosk&var-instance=${encodeURIComponent(inst)}`;
}, [selectedMachine, grafanaBaseUrl]);
const logsUrl = useMemo(() => {
if (!selectedMachine || !grafanaBaseUrl) return "";
const container =
selectedMachine.mode === "local" ? "backend" : selectedMachine.name;
return `${grafanaBaseUrl}/explore?orgId=1&left=${encodeURIComponent(
JSON.stringify({
datasource: "Loki",
queries: [{ refId: "A", expr: `{container="${container}"}` }],
range: { from: "now-1h", to: "now" },
}),
)}`;
}, [selectedMachine, grafanaBaseUrl]);
const statusDetail = status?.up
? status.version
? `version ${status.version}`
: "reachable"
: isLoading
? "checking…"
: error
? "unreachable"
: "not configured";
return (
<div className="space-y-4">
<div className="flex items-center gap-2 text-sm text-muted-foreground">
<Gauge className="h-4 w-4" />
Grafana {statusDetail}
</div>
{error && (
<Alert variant="destructive">
<AlertTitle>Failed to reach Grafana</AlertTitle>
<AlertDescription>{error.message}</AlertDescription>
</Alert>
)}
<Card>
<CardHeader className="flex flex-col gap-4 sm:flex-row sm:items-center sm:justify-between">
<CardTitle className="flex items-center gap-2">
<Activity className="h-4 w-4" />
Machine Dashboard
</CardTitle>
{machines.length > 0 ? (
<Select
value={selectedMachine?.id ?? ""}
onValueChange={setSelectedMachineId}
disabled={machinesLoading}
>
<SelectTrigger className="w-full sm:w-[240px]">
<SelectValue placeholder="Select machine" />
</SelectTrigger>
<SelectContent>
{machines.map((machine) => (
<SelectItem key={machine.id} value={machine.id}>
{machine.name}
</SelectItem>
))}
</SelectContent>
</Select>
) : null}
</CardHeader>
<CardContent className="space-y-4">
{isLoading ? (
<Skeleton className="h-24 w-full" />
) : selectedMachine && grafanaBaseUrl ? (
<>
<GrafanaLinkCard
title={`${selectedMachine.name} metrics`}
description="Open the Node Exporter overview dashboard for this machine in Grafana."
href={nodeExporterDashboardUrl}
/>
<GrafanaLinkCard
title={`${selectedMachine.name} logs`}
description="Explore Loki logs for this machine in Grafana."
href={logsUrl}
/>
</>
) : !grafanaBaseUrl ? (
<div className="flex h-full min-h-[160px] flex-col items-center justify-center gap-2 rounded-md border p-6 text-center">
<Gauge className="h-8 w-8 text-muted-foreground" />
<div className="font-medium">No Grafana base URL configured</div>
<div className="max-w-md text-sm text-muted-foreground">
Add a Grafana service instance to enable deep-links to
dashboards and logs.
</div>
<Button variant="outline" size="sm" asChild>
<Link to="/services">Open Services</Link>
</Button>
</div>
) : (
<div className="flex h-full min-h-[160px] flex-col items-center justify-center gap-2 rounded-md border p-6 text-center">
<ServerOff className="h-8 w-8 text-muted-foreground" />
<div className="font-medium">No machine selected</div>
<div className="max-w-md text-sm text-muted-foreground">
Add monitoring machines in Settings to see Grafana drill-down
links.
</div>
<Button variant="outline" size="sm" asChild>
<Link to="/settings">Open Settings</Link>
</Button>
</div>
)}
</CardContent>
</Card>
</div>
);
}
@@ -1,58 +0,0 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { LinksTab } from "../LinksTab";
import type { ServiceInstance } from "../../../types";
const instance: ServiceInstance = {
id: "graf-1",
service_type: "grafana",
name: "Main Grafana",
config: { base_url: "https://grafana.example.com", timeout_seconds: 5 },
secrets_set: {},
enabled: true,
created_at: 1_700_000_000,
updated_at: 1_700_000_000,
};
vi.mock("../../../hooks/useObservability", () => ({
useGrafanaStatus: () => ({
data: {
up: true,
version: "11.0.0",
service_id: "graf-1",
name: "Main Grafana",
},
isLoading: false,
error: null,
}),
useMonitoringMachines: () => ({
data: [
{
id: "m1",
name: "storage",
mode: "ssh",
host: "10.0.0.5",
enabled: true,
services: [],
port: 22,
username: "admin",
},
],
isLoading: false,
}),
}));
describe("LinksTab", () => {
it("renders the Grafana version and machine dashboard links", () => {
render(<LinksTab instance={instance} />);
expect(screen.getByText(/version 11\.0\.0/)).toBeInTheDocument();
expect(screen.getByText(/storage metrics/)).toBeInTheDocument();
expect(screen.getByText(/storage logs/)).toBeInTheDocument();
});
it("renders open-in-grafana link buttons", () => {
render(<LinksTab instance={instance} />);
const links = screen.getAllByText("Open in Grafana");
expect(links).toHaveLength(2);
});
});
-3
View File
@@ -8,7 +8,6 @@ import type { ComponentType } from "react";
import type { ServiceInstance } from "../../types";
import { OverviewTab } from "./OverviewTab";
import { AlertsTab } from "./AlertsTab";
import { LinksTab } from "./LinksTab";
import { MetricsTab } from "./MetricsTab";
import { MediaTab } from "./MediaTab";
import { RequestsTab } from "./RequestsTab";
@@ -56,8 +55,6 @@ export function serviceContentTabs(serviceType: string): ContentTab[] {
];
case "alertmanager":
return [{ label: "Alerts", Component: AlertsTab }];
case "grafana":
return [{ label: "Links", Component: LinksTab }];
case "prometheus":
return [{ label: "Metrics", Component: MetricsTab }];
default:
-8
View File
@@ -386,14 +386,6 @@ export interface AlertmanagerStatus {
error?: string | null;
}
export interface GrafanaStatus {
up: boolean;
version: string;
service_id: string;
name: string;
error?: string | null;
}
export interface PrometheusStatus {
up: boolean;
version: string;
-120
View File
@@ -1,120 +0,0 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Skeleton } from "@/components/ui/skeleton";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
import {
LineChart,
Line,
XAxis,
YAxis,
CartesianGrid,
Tooltip,
ResponsiveContainer,
} from "recharts";
interface Props {
widget: WidgetInstance;
refreshIntervalMs: number;
description?: string;
}
interface SeriesPoint {
t: number;
v: number | null;
}
interface ChartSeries {
label: string;
points: SeriesPoint[];
}
/** Merge multiple time-series into a single recharts-friendly array. */
function mergeSeries(series: ChartSeries[]): Record<string, unknown>[] {
const map = new Map<number, Record<string, unknown>>();
for (const s of series) {
for (const p of s.points) {
const existing = map.get(p.t) ?? { time: p.t };
existing[s.label] = p.v;
map.set(p.t, existing);
}
}
return [...map.values()].sort(
(a, b) => (a.time as number) - (b.time as number),
);
}
function formatTime(ms: number): string {
return new Date(ms).toLocaleTimeString([], {
hour: "2-digit",
minute: "2-digit",
});
}
const CHART_COLORS = [
"var(--color-chart-1)",
"var(--color-chart-2)",
"var(--color-chart-3)",
"var(--color-chart-4)",
"var(--color-chart-5)",
];
export function GrafanaChartWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const series = data?.data?.series as ChartSeries[] | undefined;
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<Skeleton className="h-[300px] w-full" />
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : series && series.length > 0 ? (
<ResponsiveContainer width="100%" height={300}>
<LineChart data={mergeSeries(series)}>
<CartesianGrid strokeDasharray="3 3" className="stroke-muted" />
<XAxis
dataKey="time"
tickFormatter={formatTime}
tick={{ fontSize: 11 }}
className="fill-muted-foreground"
/>
<YAxis tick={{ fontSize: 11 }} className="fill-muted-foreground" />
<Tooltip
labelFormatter={(label) => formatTime(Number(label))}
contentStyle={{
backgroundColor: "hsl(var(--popover))",
border: "1px solid hsl(var(--border))",
borderRadius: "0.5rem",
color: "hsl(var(--popover-foreground))",
}}
/>
{series.map((s, i) => (
<Line
key={s.label}
type="monotone"
dataKey={s.label}
stroke={CHART_COLORS[i % CHART_COLORS.length]}
dot={false}
strokeWidth={2}
connectNulls
/>
))}
</LineChart>
</ResponsiveContainer>
) : (
<Alert>
<AlertDescription>
No data. Check your query and datasource_uid in the widget config.
</AlertDescription>
</Alert>
)}
</SectionCard>
);
}
@@ -0,0 +1,42 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Skeleton } from "@/components/ui/skeleton";
import { LineSeriesChart } from "../components/LineSeriesChart";
import type { ChartSeries } from "../components/LineSeriesChart";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
interface Props {
widget: WidgetInstance;
refreshIntervalMs: number;
description?: string;
}
export function PrometheusChartWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const series = data?.data?.series as ChartSeries[] | undefined;
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<Skeleton className="h-[300px] w-full" />
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : series && series.length > 0 ? (
<LineSeriesChart series={series} />
) : (
<Alert>
<AlertDescription>
No data. Check your PromQL query and window in the widget config.
</AlertDescription>
</Alert>
)}
</SectionCard>
);
}
@@ -0,0 +1,137 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Skeleton } from "@/components/ui/skeleton";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
import {
RadialBarChart,
RadialBar,
ResponsiveContainer,
PolarAngleAxis,
} from "recharts";
interface Props {
widget: WidgetInstance;
refreshIntervalMs: number;
description?: string;
}
interface GaugeData {
value: number;
warn_at?: number | null;
crit_at?: number | null;
min?: number | null;
max?: number | null;
unit?: string | null;
}
function formatValue(value: number, unit?: string | null): string {
let formatted: string;
if (Math.abs(value) >= 100) {
formatted = value.toFixed(0);
} else if (Math.abs(value) >= 1) {
formatted = value.toFixed(2).replace(/\.?0+$/, "");
} else {
formatted = value.toFixed(4).replace(/\.?0+$/, "");
}
return unit ? `${formatted} ${unit}` : formatted;
}
export function PrometheusGaugeWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const gauge = data?.data as GaugeData | undefined;
// Compute gauge domain and threshold bands.
const value = gauge?.value ?? 0;
const min = gauge?.min ?? 0;
const max =
gauge?.max ?? Math.max(value, gauge?.warn_at ?? 0, gauge?.crit_at ?? 0, 1);
const warnAt = gauge?.warn_at;
const critAt = gauge?.crit_at;
const hasBands = warnAt != null && critAt != null;
// recharts RadialBarChart uses a 0100 domain for the angle axis.
// Map our [min, max] domain to [0, 100].
const range = max - min || 1;
const toPercent = (v: number) => Math.round(((v - min) / range) * 100);
const valuePct = Math.max(0, Math.min(100, toPercent(value)));
// Build track cells: green / amber / red when bands are set, else neutral.
const trackCells = hasBands
? [
{ pct: toPercent(warnAt!), fill: "hsl(var(--chart-1))" }, // green
{ pct: toPercent(critAt!), fill: "hsl(var(--chart-4))" }, // amber
{ pct: 100, fill: "hsl(var(--destructive))" }, // red
]
: [{ pct: 100, fill: "hsl(var(--muted))" }];
const valueColor = hasBands
? value >= critAt!
? "hsl(var(--destructive))"
: value >= warnAt!
? "hsl(var(--chart-4))"
: "hsl(var(--chart-1))"
: "hsl(var(--primary))";
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<Skeleton className="h-[220px] w-full" />
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : gauge ? (
<div className="flex flex-col items-center">
<ResponsiveContainer width="100%" height={200}>
<RadialBarChart
innerRadius="65%"
outerRadius="100%"
data={[
...trackCells.map((c) => ({
name: "track",
pct: c.pct,
fill: c.fill,
})),
{ name: "value", pct: valuePct, fill: valueColor },
]}
startAngle={90}
endAngle={-270}
>
<PolarAngleAxis
type="number"
domain={[0, 100]}
angleAxisId={0}
tick={false}
/>
<RadialBar
background={{ fill: "hsl(var(--muted))" }}
dataKey="pct"
cornerRadius={6}
/>
</RadialBarChart>
</ResponsiveContainer>
<div className="-mt-12 flex flex-col items-center">
<span className="text-2xl font-bold" style={{ color: valueColor }}>
{formatValue(value, gauge.unit)}
</span>
{hasBands ? (
<span className="text-xs text-muted-foreground">
warn {formatValue(warnAt!, gauge.unit)} · crit{" "}
{formatValue(critAt!, gauge.unit)}
</span>
) : null}
</div>
</div>
) : (
<Alert>
<AlertDescription>No data. Check your PromQL query.</AlertDescription>
</Alert>
)}
</SectionCard>
);
}
@@ -0,0 +1,59 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Skeleton } from "@/components/ui/skeleton";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
interface Props {
widget: WidgetInstance;
refreshIntervalMs: number;
description?: string;
}
interface MeanData {
value: number;
unit?: string | null;
}
function formatMean(value: number, unit?: string | null): string {
let formatted: string;
if (Math.abs(value) >= 1000) {
formatted = value.toFixed(0);
} else if (Math.abs(value) >= 1) {
formatted = value.toFixed(2).replace(/\.?0+$/, "");
} else {
formatted = value.toFixed(4).replace(/\.?0+$/, "");
}
return unit ? `${formatted} ${unit}` : formatted;
}
export function PrometheusMeanWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const mean = data?.data as MeanData | undefined;
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<Skeleton className="h-16 w-32" />
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : mean ? (
<div className="flex flex-col items-center justify-center py-4">
<span className="text-3xl font-bold">
{formatMean(mean.value, mean.unit)}
</span>
</div>
) : (
<Alert>
<AlertDescription>No data. Check your PromQL query.</AlertDescription>
</Alert>
)}
</SectionCard>
);
}
@@ -0,0 +1,76 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
interface Props {
widget: WidgetInstance;
refreshIntervalMs: number;
description?: string;
}
interface ActiveTorrent {
name: string | null;
state: string | null;
size: number | null;
progress: number | null;
dl_speed: number | null;
up_speed: number | null;
}
function formatSpeed(bytesPerSec: number | null): string {
if (bytesPerSec === null || bytesPerSec <= 0) return "—";
const mb = bytesPerSec / 1_000_000;
if (mb >= 1) return `${mb.toFixed(1)} MB/s`;
return `${(bytesPerSec / 1000).toFixed(0)} KB/s`;
}
export function QbittorrentActiveTorrentsWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const payload = data?.data as { torrents?: ActiveTorrent[] } | undefined;
const torrents = payload?.torrents ?? [];
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<div className="space-y-2">
<Skeleton className="h-8 w-full" />
<Skeleton className="h-8 w-full" />
</div>
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : torrents.length > 0 ? (
<ul className="max-h-80 space-y-1.5 overflow-y-auto">
{torrents.map((t, i) => (
<li
key={`${t.name}-${i}`}
className="flex items-center justify-between gap-2 rounded-md border px-2 py-1 text-sm"
>
<span className="truncate">{t.name ?? "Unknown"}</span>
<div className="flex shrink-0 items-center gap-2">
<span className="text-xs text-muted-foreground">
{formatSpeed(t.dl_speed)} {formatSpeed(t.up_speed)}
</span>
<Badge
variant={t.state === "downloading" ? "default" : "secondary"}
>
{t.state ?? "?"}
</Badge>
</div>
</li>
))}
</ul>
) : (
<div className="text-xs text-muted-foreground">No active torrents</div>
)}
</SectionCard>
);
}
@@ -1,7 +1,7 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Button } from "@/components/ui/button";
import { Skeleton } from "@/components/ui/skeleton";
import { ExternalLink } from "lucide-react";
import { LineSeriesChart } from "../components/LineSeriesChart";
import type { ChartSeries } from "../components/LineSeriesChart";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
@@ -12,32 +12,27 @@ interface Props {
description?: string;
}
export function GrafanaLinkWidget({
export function QbittorrentSpeedWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const url = data?.data?.url as string | undefined;
const series = data?.data?.series as ChartSeries[] | undefined;
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<Skeleton className="h-10 w-48" />
<Skeleton className="h-[220px] w-full" />
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : url ? (
<Button asChild>
<a href={url} target="_blank" rel="noopener noreferrer">
Open Grafana
<ExternalLink className="ml-2 h-4 w-4" />
</a>
</Button>
) : series && series.length > 0 ? (
<LineSeriesChart series={series} height={220} />
) : (
<Alert>
<AlertDescription>No Grafana URL configured.</AlertDescription>
<AlertDescription>No speed data yet</AlertDescription>
</Alert>
)}
</SectionCard>
@@ -0,0 +1,51 @@
import { Alert, AlertDescription } from "@/components/ui/alert";
import { Badge } from "@/components/ui/badge";
import { Skeleton } from "@/components/ui/skeleton";
import { SectionCard } from "../components/SectionCard";
import { useWidgetData } from "../hooks/useWidgets";
import type { WidgetInstance } from "../types";
interface Props {
widget: WidgetInstance;
refreshIntervalMs: number;
description?: string;
}
export function QbittorrentTotalsWidget({
widget,
refreshIntervalMs,
description,
}: Props) {
const { data, isLoading } = useWidgetData(widget.id, refreshIntervalMs);
const payload = data?.data as
| { total?: number; by_state?: Record<string, number> }
| undefined;
return (
<SectionCard title={widget.title} description={description}>
{isLoading && !data ? (
<Skeleton className="h-20 w-full" />
) : data?.error ? (
<Alert variant="destructive">
<AlertDescription>{data.error}</AlertDescription>
</Alert>
) : payload ? (
<div className="space-y-3">
<div>
<div className="text-3xl font-semibold">{payload.total ?? 0}</div>
<div className="text-xs text-muted-foreground">Total torrents</div>
</div>
{payload.by_state && Object.keys(payload.by_state).length > 0 ? (
<div className="flex flex-wrap gap-1.5">
{Object.entries(payload.by_state).map(([state, count]) => (
<Badge key={state} variant="secondary">
{state}: {count}
</Badge>
))}
</div>
) : null}
</div>
) : null}
</SectionCard>
);
}
@@ -1,6 +1,6 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { GrafanaChartWidget } from "../GrafanaChartWidget";
import { PrometheusChartWidget } from "../PrometheusChartWidget";
import type { WidgetInstance } from "../../types";
import * as useWidgets from "../../hooks/useWidgets";
@@ -29,7 +29,18 @@ function mockData(data: unknown, error?: string) {
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
}
describe("GrafanaChartWidget", () => {
describe("PrometheusChartWidget", () => {
it("renders skeleton while loading", () => {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: undefined,
isLoading: true,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
render(<PrometheusChartWidget widget={widget} refreshIntervalMs={60000} />);
expect(
document.querySelector('[data-slot="skeleton"]'),
).toBeInTheDocument();
});
it("renders a chart with series data", () => {
mockData({
series: [
@@ -42,20 +53,20 @@ describe("GrafanaChartWidget", () => {
},
],
});
render(<GrafanaChartWidget widget={widget} refreshIntervalMs={60000} />);
render(<PrometheusChartWidget widget={widget} refreshIntervalMs={60000} />);
// recharts renders an SVG; the title from SectionCard should be present.
expect(screen.getByText("CPU Usage")).toBeInTheDocument();
});
it("shows error Alert on error", () => {
mockData(null, "Grafana api_key is required for chart queries");
render(<GrafanaChartWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText(/api_key is required/i)).toBeInTheDocument();
mockData(null, "promql is required");
render(<PrometheusChartWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText(/promql is required/i)).toBeInTheDocument();
});
it("shows empty state when no series", () => {
mockData({ series: [] });
render(<GrafanaChartWidget widget={widget} refreshIntervalMs={60000} />);
render(<PrometheusChartWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText(/No data/i)).toBeInTheDocument();
});
});
@@ -0,0 +1,78 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { PrometheusGaugeWidget } from "../PrometheusGaugeWidget";
import type { WidgetInstance } from "../../types";
import * as useWidgets from "../../hooks/useWidgets";
vi.mock("../../hooks/useWidgets", () => ({
useWidgetData: vi.fn(),
}));
const widget: WidgetInstance = {
id: "wg1",
service_id: "s1",
widget_kind: "gauge",
title: "CPU Gauge",
config: {},
enabled: true,
sort_order: 0,
created_at: 0,
updated_at: 0,
};
function mockData(data: unknown, error?: string) {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: error
? { widget_id: "wg1", error, fetched_at: 0 }
: { widget_id: "wg1", data, fetched_at: 0 },
isLoading: false,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
}
describe("PrometheusGaugeWidget", () => {
it("renders skeleton while loading", () => {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: undefined,
isLoading: true,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
render(<PrometheusGaugeWidget widget={widget} refreshIntervalMs={30000} />);
expect(
document.querySelector('[data-slot="skeleton"]'),
).toBeInTheDocument();
});
it("renders a gauge with value and threshold bands", () => {
mockData({
value: 0.75,
warn_at: 0.8,
crit_at: 0.95,
unit: "%",
});
render(<PrometheusGaugeWidget widget={widget} refreshIntervalMs={30000} />);
expect(screen.getByText("CPU Gauge")).toBeInTheDocument();
expect(screen.getByText(/0.75 %/)).toBeInTheDocument();
// Threshold labels present when bands are set.
expect(screen.getByText(/warn/i)).toBeInTheDocument();
expect(screen.getByText(/crit/i)).toBeInTheDocument();
});
it("renders a gauge without threshold bands (single color)", () => {
mockData({ value: 42, unit: "req/s" });
render(<PrometheusGaugeWidget widget={widget} refreshIntervalMs={30000} />);
expect(screen.getByText(/42 req\/s/)).toBeInTheDocument();
// No threshold labels when bands are absent.
expect(screen.queryByText(/warn/i)).not.toBeInTheDocument();
});
it("shows error Alert on error", () => {
mockData(null, "Gauge requires a single-series query");
render(<PrometheusGaugeWidget widget={widget} refreshIntervalMs={30000} />);
expect(screen.getByText(/single-series/i)).toBeInTheDocument();
});
it("shows empty state when no data", () => {
mockData(null);
render(<PrometheusGaugeWidget widget={widget} refreshIntervalMs={30000} />);
expect(screen.getByText(/No data/i)).toBeInTheDocument();
});
});
@@ -0,0 +1,68 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { PrometheusMeanWidget } from "../PrometheusMeanWidget";
import type { WidgetInstance } from "../../types";
import * as useWidgets from "../../hooks/useWidgets";
vi.mock("../../hooks/useWidgets", () => ({
useWidgetData: vi.fn(),
}));
const widget: WidgetInstance = {
id: "wm1",
service_id: "s1",
widget_kind: "mean",
title: "Avg CPU",
config: {},
enabled: true,
sort_order: 0,
created_at: 0,
updated_at: 0,
};
function mockData(data: unknown, error?: string) {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: error
? { widget_id: "wm1", error, fetched_at: 0 }
: { widget_id: "wm1", data, fetched_at: 0 },
isLoading: false,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
}
describe("PrometheusMeanWidget", () => {
it("renders skeleton while loading", () => {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: undefined,
isLoading: true,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
render(<PrometheusMeanWidget widget={widget} refreshIntervalMs={60000} />);
expect(
document.querySelector('[data-slot="skeleton"]'),
).toBeInTheDocument();
});
it("renders the mean value with unit", () => {
mockData({ value: 23.5, unit: "%" });
render(<PrometheusMeanWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText("Avg CPU")).toBeInTheDocument();
expect(screen.getByText(/23.5 %/)).toBeInTheDocument();
});
it("renders the mean value without unit", () => {
mockData({ value: 1500, unit: null });
render(<PrometheusMeanWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText(/1500/)).toBeInTheDocument();
});
it("shows error Alert on error", () => {
mockData(null, "Mean requires a single-series query");
render(<PrometheusMeanWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText(/single-series/i)).toBeInTheDocument();
});
it("shows empty state when no data", () => {
mockData(null);
render(<PrometheusMeanWidget widget={widget} refreshIntervalMs={60000} />);
expect(screen.getByText(/No data/i)).toBeInTheDocument();
});
});
@@ -0,0 +1,103 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { QbittorrentActiveTorrentsWidget } from "../QbittorrentActiveTorrentsWidget";
import type { WidgetInstance } from "../../types";
import * as useWidgets from "../../hooks/useWidgets";
vi.mock("../../hooks/useWidgets", () => ({
useWidgetData: vi.fn(),
}));
const widget: WidgetInstance = {
id: "w1",
service_id: "s1",
widget_kind: "active",
title: "Active Torrents",
config: {},
enabled: true,
sort_order: 0,
created_at: 0,
updated_at: 0,
};
function mockData(data: unknown, error?: string) {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: error
? { widget_id: "w1", error, fetched_at: 0 }
: { widget_id: "w1", data, fetched_at: 0 },
isLoading: false,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
}
describe("QbittorrentActiveTorrentsWidget", () => {
it("renders skeleton while loading", () => {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: undefined,
isLoading: true,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
render(
<QbittorrentActiveTorrentsWidget
widget={widget}
refreshIntervalMs={15000}
/>,
);
expect(
document.querySelector('[data-slot="skeleton"]'),
).toBeInTheDocument();
});
it("renders active torrent rows", () => {
mockData({
torrents: [
{
name: "Movie.mkv",
state: "downloading",
size: 1000,
progress: 0.5,
dl_speed: 500000,
up_speed: 1000,
},
{
name: "Show.mkv",
state: "uploading",
size: 2000,
progress: 1.0,
dl_speed: 0,
up_speed: 50000,
},
],
});
render(
<QbittorrentActiveTorrentsWidget
widget={widget}
refreshIntervalMs={15000}
/>,
);
expect(screen.getByText("Movie.mkv")).toBeInTheDocument();
expect(screen.getByText("Show.mkv")).toBeInTheDocument();
expect(screen.getByText("downloading")).toBeInTheDocument();
expect(screen.getByText("uploading")).toBeInTheDocument();
});
it("shows empty state when no active torrents", () => {
mockData({ torrents: [] });
render(
<QbittorrentActiveTorrentsWidget
widget={widget}
refreshIntervalMs={15000}
/>,
);
expect(screen.getByText(/No active torrents/i)).toBeInTheDocument();
});
it("shows error alert on error", () => {
mockData(null, "qBittorrent fetch failed");
render(
<QbittorrentActiveTorrentsWidget
widget={widget}
refreshIntervalMs={15000}
/>,
);
expect(screen.getByText(/qBittorrent fetch failed/i)).toBeInTheDocument();
});
});
@@ -0,0 +1,69 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { QbittorrentSpeedWidget } from "../QbittorrentSpeedWidget";
import type { WidgetInstance } from "../../types";
import * as useWidgets from "../../hooks/useWidgets";
vi.mock("../../hooks/useWidgets", () => ({
useWidgetData: vi.fn(),
}));
const widget: WidgetInstance = {
id: "w1",
service_id: "s1",
widget_kind: "speed",
title: "Speed Chart",
config: {},
enabled: true,
sort_order: 0,
created_at: 0,
updated_at: 0,
};
function mockData(data: unknown, error?: string) {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: error
? { widget_id: "w1", error, fetched_at: 0 }
: { widget_id: "w1", data, fetched_at: 0 },
isLoading: false,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
}
describe("QbittorrentSpeedWidget", () => {
it("renders skeleton while loading", () => {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: undefined,
isLoading: true,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
render(<QbittorrentSpeedWidget widget={widget} refreshIntervalMs={5000} />);
expect(
document.querySelector('[data-slot="skeleton"]'),
).toBeInTheDocument();
});
it("renders chart with series data", () => {
mockData({
series: [
{ label: "download", points: [{ t: 1000, v: 500000 }] },
{ label: "upload", points: [{ t: 1000, v: 100000 }] },
],
});
const { container } = render(
<QbittorrentSpeedWidget widget={widget} refreshIntervalMs={5000} />,
);
expect(screen.getByText("Speed Chart")).toBeInTheDocument();
expect(container.firstChild).not.toBeNull();
});
it("shows empty state when no series", () => {
mockData({ series: [] });
render(<QbittorrentSpeedWidget widget={widget} refreshIntervalMs={5000} />);
expect(screen.getByText(/No speed data yet/i)).toBeInTheDocument();
});
it("shows error alert on error", () => {
mockData(null, "qBittorrent fetch failed");
render(<QbittorrentSpeedWidget widget={widget} refreshIntervalMs={5000} />);
expect(screen.getByText(/qBittorrent fetch failed/i)).toBeInTheDocument();
});
});
@@ -0,0 +1,66 @@
import { describe, it, expect, vi } from "vitest";
import { render, screen } from "@testing-library/react";
import { QbittorrentTotalsWidget } from "../QbittorrentTotalsWidget";
import type { WidgetInstance } from "../../types";
import * as useWidgets from "../../hooks/useWidgets";
vi.mock("../../hooks/useWidgets", () => ({
useWidgetData: vi.fn(),
}));
const widget: WidgetInstance = {
id: "w1",
service_id: "s1",
widget_kind: "totals",
title: "Torrent Totals",
config: {},
enabled: true,
sort_order: 0,
created_at: 0,
updated_at: 0,
};
function mockData(data: unknown, error?: string) {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: error
? { widget_id: "w1", error, fetched_at: 0 }
: { widget_id: "w1", data, fetched_at: 0 },
isLoading: false,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
}
describe("QbittorrentTotalsWidget", () => {
it("renders skeleton while loading", () => {
vi.mocked(useWidgets.useWidgetData).mockReturnValue({
data: undefined,
isLoading: true,
} as unknown as ReturnType<typeof useWidgets.useWidgetData>);
render(
<QbittorrentTotalsWidget widget={widget} refreshIntervalMs={30000} />,
);
expect(
document.querySelector('[data-slot="skeleton"]'),
).toBeInTheDocument();
});
it("renders total count and state badges", () => {
mockData({
total: 4,
by_state: { downloading: 1, uploading: 1, pausedDL: 2 },
});
render(
<QbittorrentTotalsWidget widget={widget} refreshIntervalMs={30000} />,
);
expect(screen.getByText("4")).toBeInTheDocument();
expect(screen.getByText("downloading: 1")).toBeInTheDocument();
expect(screen.getByText("pausedDL: 2")).toBeInTheDocument();
});
it("shows error alert on error", () => {
mockData(null, "qBittorrent fetch failed");
render(
<QbittorrentTotalsWidget widget={widget} refreshIntervalMs={30000} />,
);
expect(screen.getByText(/qBittorrent fetch failed/i)).toBeInTheDocument();
});
});
+7 -2
View File
@@ -1,8 +1,13 @@
export { AlertmanagerAlertsWidget } from "./AlertmanagerAlertsWidget";
export { BackupsWidget } from "./BackupsWidget";
export { GrafanaLinkWidget } from "./GrafanaLinkWidget";
export { GrafanaChartWidget } from "./GrafanaChartWidget";
export { PrometheusChartWidget } from "./PrometheusChartWidget";
export { PrometheusGaugeWidget } from "./PrometheusGaugeWidget";
export { PrometheusMeanWidget } from "./PrometheusMeanWidget";
export { JellyfinWidget } from "./JellyfinWidget";
export { JellyfinNowPlayingWidget } from "./JellyfinNowPlayingWidget";
export { PrometheusMetricWidget } from "./PrometheusMetricWidget";
export { QbittorrentActiveTorrentsWidget } from "./QbittorrentActiveTorrentsWidget";
export { QbittorrentSpeedWidget } from "./QbittorrentSpeedWidget";
export { QbittorrentTotalsWidget } from "./QbittorrentTotalsWidget";
export { SshTaskWidget } from "./SshTaskWidget";
export { StaticWidget } from "./StaticWidget";
@@ -0,0 +1,54 @@
# Apply Progress: Prometheus Direct Charting
**Change:** `prometheus-direct-charting`
**Phase:** apply-progress
**Date:** 2026-07-08
**Status:** complete — all 39 tasks done, all gates green, verified (see `verify-report.md`)
## Slices delivered
Three slices, each its own commit, each leaving `pytest` / `npm run build` / `npm run lint` / `ruff` green.
### Slice 1 — Prometheus range query + chart rebrand (commit `5dad982`, amended)
- Created `backend/src/media_library_viewer_api/widgets/prometheus_range.py` with `step_for_window(s)` (`max(15, round(s/200))` → ~200 pts/window) and `normalize_prometheus_matrix(result)` (shared label rule extracted from the to-be-removed Grafana path; drops `__name__`/`__*`, joins sorted `k=v`, falls back to `"value"`, dedups with `(n)`).
- Extended `PrometheusWidgetSource.fetch` to dispatch `widget_kind == "chart"``_fetch_chart` hitting `/api/v1/query_range` directly, returning `{series:[...]}` (SC-101..104). Grafana path left intact at this slice.
- Declared `chart` widget kind in `integrations/prometheus.py` (config: `promql`, `window``{1h,6h,24h,7d}`).
- `git mv GrafanaChartWidget.tsx → PrometheusChartWidget.tsx` (recharts body preserved verbatim; empty-state copy updated); `git mv` of its test. Rebound `chart` grafana→prometheus in both registries (SC-105..108).
- Backend tests: new `test_prometheus_range.py` (step + normalization); chart-adapter test in `test_widgets.py`. Frontend registry test updated.
### Slice 2 — Gauge + mean widgets (commit `58be6e0`, amended)
- Extracted shared `_instant_query` helper; added `_fetch_gauge` (instant → scalar, multi-series → `{error}`) and `_fetch_mean` (range query over preset → client-side arithmetic mean of non-null values, scalar-only).
- Declared `gauge` (`promql`, `warn_at`/`crit_at`/`min`/`max`/`unit`) and `mean` (`promql`, `window`, `unit`) kinds in `integrations/prometheus.py`.
- Created `PrometheusGaugeWidget.tsx` (recharts `RadialBarChart`; green/amber/red threshold bands when `warn_at`+`crit_at` set; neutral single track otherwise) and `PrometheusMeanWidget.tsx` (MetricCard-style single value). Wired both into the frontend `prometheus` binding + barrel.
- Tests: backend adapter tests (scalar-only enforcement, mean aggregation incl. NaN-skip, error cases); frontend component tests (error + rendered, gauge with/without bands, mean with/without unit).
### Slice 3 — Grafana removal + config + changelog (commit `ba94317`, amended)
- Deleted `integrations/grafana.py`, `GrafanaLinkWidget.tsx`, `service-tabs/LinksTab.tsx` (+ test). Removed `GrafanaWidgetSource` + adapter registration; `grafana` from `SERVICE_DEFINITIONS`/`SERVICE_ADAPTERS` (BE) and `SERVICE_REGISTRY`/`BUILTIN_WIDGETS` (FE); `get_grafana_status` endpoint; `useGrafanaStatus`/`fetchGrafanaStatus`/`GrafanaStatus`; nav entry; `service-tabs/index.ts` grafana case; `Dashboard.tsx` `OBSERVABILITY_TYPES` grafana member; `ServicesPage.tsx` empty-state copy; grafana tests.
- Rewrote `openspec/config.yaml`: removed stale "Do NOT re-implement charting in-app" + "No recharts/d3" claims; states Manage renders Prometheus-backed metrics directly via recharts and that Grafana is no longer integrated.
- Added `CHANGELOG.md` `[Unreleased]` entry: **BREAKING** — Grafana service type removed; migrate by deleting grafana instances and recreating as Prometheus; `grafana/chart` widgets → `prometheus/chart`.
- Net: **920 lines** across 27 files.
### Coverage close — SC-125 loading-state tests
- Added one `it("renders skeleton while loading")` case to each of the three Prometheus widget test files, asserting the `Skeleton` (`data-slot="skeleton"`) renders under `{ data: undefined, isLoading: true }`. Closes the PARTIAL finding on SC-125.
## Deviations from tasks.md
- None functional. The only textual drift is SC-118: `ObservabilityPage.tsx` had already been refactored into `service-tabs/` (the project map was stale). Removal targets adjusted to the real files (`LinksTab.tsx`, `service-tabs/index.ts`, `navEntries.ts`, `Dashboard.tsx`, `ServicesPage.tsx`); the spec was patched (SC-116/SC-118) to reflect this before apply. SC-118's *intent* (no Grafana UI surface) is fully satisfied.
## Final gate results (re-run after coverage close)
| Gate | Result |
|---|---|
| `backend && PYTHONPATH=src python3 -m pytest -q` | **293 passed**, 2 warnings (pre-existing pythonjsonlogger DeprecationWarning) |
| `backend && PYTHONPATH=src python3 -m ruff check src tests` | **All checks passed** |
| `frontend && npm run build` (`tsc -b` + `vite build`) | **exit 0** (pre-existing chunk-size warning) |
| `frontend && npm run lint` | **0 errors**, 1 pre-existing warning (`WidgetConfigDialog.tsx:370`, untouched) |
| `frontend && npx vitest run` (3 Prom widget tests) | **14 passed** (11 original + 3 new loading) |
## Verification
See `verify-report.md` — adversarial fresh-context review: **26/27 fully PASS, 1 PARTIAL→PASS** (SC-125 closed here). No blocking findings remain.
@@ -0,0 +1,204 @@
# Archive Report — `prometheus-direct-charting`
> Phase: **archive** · Change: `prometheus-direct-charting` · Repo: `/home/user/manage`
> Mode: file-backed (`artifactStore: openspec`). No source-code edits; only OpenSpec artifacts
> were touched. **Not committed** — the parent/orchestrator owns the archive commit. No push, no `gh`.
**Status: ARCHIVED.** All eight lifecycle phases are complete (proposal → spec → design → tasks →
apply → verify → sync → **archive**). Every archive precondition is verified PASS (see §2). The
canonical `openspec/specs/prometheus-charting/spec.md` (created by `sdd-sync`) remains in place as
the durable end-state spec and is **not** moved (archive never moves canonical specs). The change
folder was moved to `openspec/changes/archive/2026-07-08-prometheus-direct-charting/` via `git mv`
to preserve history.
---
## 0. Archive disposition
- **Disposition: `archived`.** The folder move was performed inline as instructed (unlike a
`documented-pending-manual` outcome): the parent explicitly requested the `git mv` and owns the
commit, so the move is executed here and left staged for the parent's explicit-path commit.
- **Archive convention:** OpenSpec SDD archive contract for `openspec` mode — completed file-backed
sync → write the in-folder archive report → move the change folder to
`openspec/changes/archive/YYYY-MM-DD-{change}/`. No standalone manifest/index exists under
`openspec/` (only `config.yaml`, `changes/`, `specs/`), so the folder move **is** the archive
mechanism. No `rules.archive` override exists in `openspec/config.yaml`.
- **Target archived path:** `openspec/changes/archive/2026-07-08-prometheus-direct-charting/`
- **Archive date:** `2026-07-08` (ISO).
- **Canonical spec left in place (not moved):** `openspec/specs/prometheus-charting/spec.md`
27 requirements (SC-101 … SC-127). Verified present and untouched after the move.
- **`openspec/specs/web-ui/spec.md` also left untouched** (the other canonical domain).
- **Audit-trail integrity:** the change folder was moved as a whole, including the legacy flat
`spec.md` and the per-domain delta `specs/prometheus-charting/spec.md`, which travel with the
record. Nothing was silently deleted or rewritten. The flat `spec.md` is retained as the
authoritative planning artifact the work was built against.
## 1. Native `sdd-status` read & discrepancy statement
The native `gentle-pi.sdd-status` engine supplied by the parent reports **non-actionable state for
this archive** because it was resolved without a change context: `changeName: null`,
`artifacts: all missing`, `applyState: blocked`, `dependencies.archive: blocked`,
`blockedReasons: ["Change selection is ambiguous: mobile-responsive-parity,
prometheus-direct-charting, service-storage-harness, services-as-hub-ia."]`,
`isNonAuthoritative: false`. This is a **parent-resolution artifact**: the engine auto-detected four
active changes and could not pick one. The ambiguity does **not** reflect the state of
`prometheus-direct-charting`, which this archive task was **explicitly assigned**.
**Discrepancy with the parent's authoritative confirmed state — RESOLVED in favor of the parent.**
The parent physically verified (and this executor re-confirmed directly against the filesystem in
§2) that all 39 tasks are ticked, `apply-progress.md` exists and records the work, and the verify
report confirms functional completion with no blocking findings. Per the archive contract's
non-authoritative-store carve-out guidance and the parent's explicit instruction, the stale
`archive: blocked` / "ambiguous" labels are **disregarded** and the archive **proceeds**.
Direct filesystem re-validation (§2) is the source of truth for this report.
## 2. Archive preconditions (validated directly against the filesystem)
| Precondition | Evidence | Result |
|---|---|---|
| Verify report present | `verify-report.md` | ✓ verdict **PASS** |
| Verify clearly passing — no unresolved `FAIL`/`BLOCKED`/`CRITICAL` | verify-report §9: the sole CRITICAL was an archive-only checkbox/apply-progress gap (now reconciled); SC-125 PARTIAL→PASS closed | ✓ |
| Sync report present & successful | `sync-report.md`**Status: SYNCED** | ✓ |
| Canonical spec exists (sync target) | `openspec/specs/prometheus-charting/spec.md` (27 requirements) | ✓ |
| Change-side domain delta exists | `specs/prometheus-charting/spec.md` | ✓ |
| Delta op-class = pure `## ADDED` (non-destructive) | ADDED=1, MODIFIED=0, REMOVED=0, RENAMED=0 (new domain) | ✓ |
| Requirement-ID parity (flat ↔ delta ↔ canonical) | 27 == 27 == 27, identical IDs SC-101…SC-127 | ✓ |
| proposal / design / tasks artifacts present | all populated | ✓ |
| **Final Task Completion Gate — zero unchecked `- [ ]`** | `grep -nE '^\s*- \[ \]' tasks.md`**NONE**; `grep -cE '^\s*- \[x\]'`**39** | ✓ |
| `apply-progress.md` present & records the work | present; status "complete — all 39 tasks done", 3 slices + coverage close documented | ✓ |
| No active same-domain (`prometheus-charting`) collision | only this change carries a `prometheus-charting` delta | ✓ |
**Stale-checkbox reconciliation note.** At verify time, 19 implementation/verification checkboxes
(Slice 3 §3.13.14 and Integration §4.14.5) were unchecked and `apply-progress.md` did not exist.
That condition was reconciled **before** archive: the boxes are now all ticked and
`apply-progress.md` was authored documenting the three landed slices and the SC-125 coverage close.
`apply-progress.md` plus the verify report prove every previously-unchecked task complete. No
archive-time mechanical checkbox repair was needed — the gate now passes on the persisted
`tasks.md` as-is. No partial-archive approval applies.
## 3. Artifacts read (archive preflight)
- `openspec/changes/prometheus-direct-charting/proposal.md`
- `openspec/changes/prometheus-direct-charting/spec.md` (flat, authoritative planning artifact — 27 requirements)
- `openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md` (change-side domain delta)
- `openspec/changes/prometheus-direct-charting/design.md`
- `openspec/changes/prometheus-direct-charting/tasks.md`
- `openspec/changes/prometheus-direct-charting/apply-progress.md`
- `openspec/changes/prometheus-direct-charting/verify-report.md`
- `openspec/changes/prometheus-direct-charting/sync-report.md`
- `openspec/specs/prometheus-charting/spec.md` (canonical, sync target — verified present and untouched)
- `openspec/config.yaml` (rules: proposal/tasks; no `rules.archive` override)
> The legacy flat `spec.md` is **not** the only spec artifact: a per-domain delta
> (`specs/prometheus-charting/spec.md`) and a canonical spec both exist, so the "legacy flat spec
> as the *only* artifact" archive-block condition does not apply. The flat spec travels with the
> archived folder as part of the audit trail.
## 4. Domains synced & requirement delta
| Domain | Change-side delta | Canonical | Action |
|---|---|---|---|
| `prometheus-charting` | `specs/prometheus-charting/spec.md` | `openspec/specs/prometheus-charting/spec.md` | **NEW domain** — pure ADDED (27 requirements) |
- **ADDED (27)** — all to the new `prometheus-charting` domain (canonical did not exist pre-change).
IDs and text preserved verbatim from the verified flat `spec.md`. Grouped logically:
- *Direct Prometheus range query path* — SC-101, SC-102, SC-103, SC-104
- *Prometheus chart widget (rebrand + rebind)* — SC-105, SC-106, SC-107, SC-108
- *Prometheus gauge widget* — SC-109, SC-110, SC-111
- *Prometheus mean widget* — SC-112, SC-113, SC-114
- *Grafana removal* — SC-115, SC-116, SC-117, SC-118, SC-119, SC-120
- *Configuration documentation accuracy* — SC-121, SC-122
- *Test and build greenness* — SC-123, SC-124, SC-125
- *Migration guidance* — SC-126, SC-127
- **MODIFIED (0)** · **REMOVED (0)** · **RENAMED (0)** — new domain; nothing destructive.
> No destructive-merge guard or parent approval was triggered (zero REMOVED / zero MODIFIED). The
> new `prometheus-charting` domain is distinct from the existing `web-ui` canonical domain.
## 5. Final lifecycle status (all 8 phases done)
| Phase | Status | Evidence |
|---|---|---|
| Proposal | ✅ done | `proposal.md` |
| Spec | ✅ done | flat `spec.md` (27) + domain delta `specs/prometheus-charting/spec.md` (27 ADDED) |
| Design | ✅ done | `design.md` |
| Tasks | ✅ done | `tasks.md`**39/39** checked, zero `- [ ]` |
| Apply | ✅ done | 3 slices delivered (commits `5dad982`, `65bae95`/`58be6e0`, `67ca0fc`/`ba94317`) |
| Verify | ✅ PASS | `verify-report.md` — 26/27 PASS, SC-125 PARTIAL→PASS; gates green |
| Sync | ✅ done | `sync-report.md` — SYNCED; canonical `prometheus-charting` domain created |
| Archive | ✅ done | this report + folder move performed |
## 6. Gate results (run at head `67ca0fc`; per verify-report)
| Gate | Command | Result |
|---|---|---|
| Backend tests | `cd backend && PYTHONPATH=src python3 -m pytest -q` | **PASS** — 293 passed (2 pre-existing warnings) |
| Backend lint | `cd backend && PYTHONPATH=src python3 -m ruff check src tests` | **PASS** — All checks passed |
| Frontend build | `cd frontend && npm run build` | **PASS** — exit 0 (pre-existing chunk-size warning) |
| Frontend lint | `cd frontend && npm run lint` | **PASS** — 0 errors (1 pre-existing warning) |
## 7. Carry-over follow-ups (non-blocking; recorded per task)
1. **[INFO] Stale generated `.pi-map.md` still references Grafana** — generated project-map artifacts
(`.pi-map.md` / `.pi-map.index.md`) still mention Grafana / `ObservabilityPage.tsx` /
`GrafanaLinkWidget`. These are **generated artifacts, not deliverable source**, and are explicitly
ignored by SC-115/SC-116 (which scope only `backend/src`/`frontend/src` source). **Out of scope
for this change** — regenerate via `project_map_patch` / `project_map_validate` in a separate
housekeeping pass; the project-map protocol already flags these `dirty`.
2. **[INFO] SC-125 was PARTIAL→closed.** Verify found SC-125 (new widget kinds have tests) PARTIAL
because no explicit `isLoading:true` loading-state test existed in any of the three Prometheus
widget test files — a coverage gap, not a functional defect. The gap was **closed** before
archive: `apply-progress.md` documents adding one `renders skeleton while loading` case to each
of `PrometheusChartWidget` / `PrometheusGaugeWidget` / `PrometheusMeanWidget`, asserting the
`Skeleton` renders under `{ data: undefined, isLoading: true }`. SC-125 is now PASS.
3. **[INFO] `config.yaml` context block** still names `frontend/src/components/ObservabilityPage.tsx`
(refactored away into `service-tabs/`). Not an SC-121 criterion (which targets the
charting/grafana claims — those are fixed); minor doc staleness. Out of scope.
4. **[INFO] Slice-2 review-budget variance** — slice 2 (~707 insertions) exceeded the ~310400
forecast, but is additive feature code + tests (gauge + mean); the boundary is the feature, not
scope creep. Non-blocking; recorded for the record.
5. **[INFO] Unrelated dirty working-tree items** predating/orthogonal to this change (an uncommitted
cosmetic reformat of `frontend/src/pages/service-tabs/MediaTab.tsx`, untracked `.pi-tmp/*`, and
the separate `service-storage-harness` proposal folder) were **not touched** by this archive.
SC-127 (independence from `service-storage-harness`) holds.
## 8. Residual risks & destructive-merge statement
- **Destructive sync / merge:** **not applicable.** Zero REMOVED and zero MODIFIED requirements
(new `prometheus-charting` domain; pure ADDED). No destructive-merge guard or parent approval was
triggered.
- **Backend / data-contract impact:** none — Grafana was fully excised from live code paths
(service type, adapters, widgets, hook, API client, type, status endpoint, nav entry, service
tab); Prometheus is the direct chart source via `/api/v1/query_range`. The `metric` widget data
shape is preserved. Archive touched only OpenSpec docs + the folder move.
- **No critical verification issues** remain (CRITICAL issues are non-overridable; the one verify
CRITICAL was the reconcilable checkbox/apply-progress gap, now resolved).
- **No browser/visual smoke** was performed (out of scope); the recharts `RadialBarChart` gauge and
`LineChart` rendering are structurally tested only.
- **Memory observation IDs:** none — `artifactStore: openspec`; traceability lives in the
filesystem archive + canonical spec.
## 9. Move performed
```
git mv openspec/changes/prometheus-direct-charting openspec/changes/archive/2026-07-08-prometheus-direct-charting
```
- **All 9 artifacts confirmed present at the archived path:** `proposal.md`, `spec.md`,
`specs/prometheus-charting/spec.md` (delta), `design.md`, `tasks.md`, `apply-progress.md`,
`verify-report.md`, `sync-report.md`, `archive-report.md` (this file).
- **Canonical `openspec/specs/prometheus-charting/spec.md` remains in place** (verified untouched
after the move). `openspec/specs/web-ui/spec.md` also untouched.
---
### Appendix — Files written/moved by this archive (OpenSpec only; no source code)
- **Written:** `openspec/changes/prometheus-direct-charting/archive-report.md` (this file) — at the
active path before the move; travels with the move into the archive.
- **Moved (via `git mv`):** the entire
`openspec/changes/prometheus-direct-charting/` directory →
`openspec/changes/archive/2026-07-08-prometheus-direct-charting/`.
- **Left in place (durable canonical):** `openspec/specs/prometheus-charting/spec.md`.
- **Not committed / not pushed** — the parent owns the commit with explicit paths.
@@ -0,0 +1,462 @@
# SDD Design: Prometheus Direct Charting (drop Grafana middleman)
**Change:** `prometheus-direct-charting`
**Phase:** design
**Date:** 2026-07-08
## 0. Source findings (read before anything else)
The proposal and spec were written against a **stale project map**. Reading actual source surfaced deviations the design must account for. Trust source, not the map.
| Spec claim | Actual source reality | Design impact |
|---|---|---|
| SC-118: "remove the Grafana branch from `ObservabilityPage`" | **`ObservabilityPage.tsx` no longer exists.** It was refactored into a per-service-type `service-tabs/` architecture (confirmed by `LinksTab.tsx` docstring: "Lifts the Grafana deep-link content from the old cross-service ObservabilityPage into an instance-scoped tab"). The stale reference survives only in `.pi-map.md` files. | Removal targets are `service-tabs/LinksTab.tsx` + its test, the `grafana` case in `service-tabs/index.ts`, the `navEntries.ts` grafana entry, `Dashboard.tsx`'s `OBSERVABILITY_TYPES` set, `ServicesPage.tsx` empty-state text, and the `useGrafanaStatus` hook. See §5. |
| "No recharts/d3 in use" (config rule to repeal) | `recharts ^3.9.2` is **declared, installed, and imported** by `GrafanaChartWidget.tsx`. Confirmed. | Repeal is a doc fix matching reality; recharts is the sanctioned renderer. |
| Proposal §5.1: "extend `PrometheusWidgetSource` ... to handle `chart`" | `PrometheusWidgetSource.fetch` currently only does instant `/api/v1/query` and returns `{"result": data}`. The frontend `PrometheusMetricWidget` consumes `data.result`. | The chart/mean paths return a **different** shape (`{series}` / `{value}`); dispatch inside `.fetch()` by `widget_kind`. See §2.2. |
| Map said registry has 7 service types | Source registry has **8**: `alertmanager, authentik, backups, grafana, jellyfin, nextcloud, prometheus, ssh_tasks`. (`authentik` and `backups` were added after the map froze.) | No design impact beyond acknowledging grafana is one of eight, not one of seven. |
| Frontend `SERVICE_REGISTRY` (registry.ts) | Has a `grafana` binding with `link` + `chart` kinds, and a `prometheus` binding with only `metric`. The `chart` kind must move grafana→prometheus. | Confirmed; §3.2 details the rebinding. |
No proposal/spec scope change is required — the *intent* (remove Grafana, direct Prom charting) still holds. Only the **removal targets** differ from what SC-118 literally names. This is flagged explicitly so the tasks phase and reviewer aren't surprised.
---
## 1. Architecture overview
This change cuts Grafana out of the chart data path and makes Prometheus the direct source. The existing recharts renderer is reused unchanged for `chart`; two new widget kinds (`gauge`, `mean`) are added to the `prometheus` service. Grafana is removed entirely.
```
BEFORE AFTER
─────── ─────
WidgetData fetch WidgetData fetch
└► GrafanaWidgetSource └► PrometheusWidgetSource
└► POST {grafana}/api/ds/query ├► kind=metric → /api/v1/query {result} [unchanged]
(datasource.type=prometheus) ├► kind=chart → /api/v1/query_range {series} [NEW]
└► normalize frames → {series} ├► kind=gauge → /api/v1/query {value,...} [NEW]
└► kind=mean → /api/v1/query_range {value} [NEW]
Frontend GrafanaChartWidget (recharts) Frontend PrometheusChartWidget (recharts) [renamed, reused]
+ PrometheusGaugeWidget (recharts RadialBarChart) [NEW]
+ PrometheusMeanWidget (MetricCard-style) [NEW]
grafana service type / LinkWidget / [REMOVED entirely]
LinksTab / navEntry / status endpoint
```
**Key constraints carried from the spec:**
- Reuse the recharts chart renderer unchanged (SC-106) — the rename is structural.
- Shared series normalization (SC-102) — one helper, no duplication.
- Step derived from window presets (SC-104), users never set `step`.
- `gauge`/`mean` are scalar-only (SC-111, SC-114); `chart` stays multi-series (SC-107).
- Window presets: `1h`, `6h`, `24h`, `7d` (SC-108, SC-112).
- All adapter errors return `{"error": str}`, never raise (SC-103).
- No Grafana references remain (SC-115, SC-116, SC-117, SC-120).
---
## 2. Backend design
### 2.1 New module: `widgets/prometheus_range.py` (shared helpers)
Holds the two pieces the spec's downstream-notes asked for: the step-derivation function (SC-104) and the shared normalization helper (SC-102). Putting them in their own module (rather than inside `sources.py`) makes them unit-testable in isolation and reusable by the future `service-storage-harness` change's in-service data path, without `sources.py` growing unbounded.
**Window presets and step derivation (SC-104):**
```python
WINDOW_PRESETS: dict[str, int] = {
"1h": 3_600,
"6h": 21_600,
"24h": 86_400,
"7d": 604_800,
}
# Target ~200 points per window. Step is clamped to >= 15s so Prometheus
# doesn't reject sub-15s resolutions on high-cardinality queries.
def step_for_window(window_seconds: int, target_points: int = 200) -> int:
return max(15, round(window_seconds / target_points))
```
Resulting table (verified, all within the 100300 target band):
| Preset | Window (s) | Derived step (s) | Points |
|--------|-----------|------------------|--------|
| `1h` | 3,600 | `max(15, round(3600/200))` = 18 | 200 |
| `6h` | 21,600 | 108 | 200 |
| `24h` | 86,400 | 432 | 200 |
| `7d` | 604,800 | 3,024 | 200 |
(At apply time the implementer may round steps to "nicer" values like 15/60/300/1800 for cache-friendliness; the spec only requires the 100300 band, which the formula satisfies. The formula is the source of truth; the table is illustrative.)
**Shared normalization (SC-102) — `normalize_prometheus_matrix`:**
```python
def normalize_prometheus_matrix(
result: list[dict[str, Any]],
) -> list[dict[str, Any]]:
"""Turn a Prometheus /api/v1/query_range `data.result` matrix into
`{label, points:[{t:int, v:float|None}]}` series — the exact shape the
frontend chart renderer consumes.
Label derivation reuses the rule from the removed Grafana path:
1. Drop __name__ from metric labels.
2. If labels remain, join as `k=v k=v`.
3. Else fall back to "value".
4. Dedup collisions with a ` (n)` suffix.
"""
series: list[dict[str, Any]] = []
seen: dict[str, int] = {}
for entry in result:
metric = entry.get("metric") or {}
values = entry.get("values") or []
parts = [f"{k}={v}" for k, v in sorted(metric.items()) if not k.startswith("__")]
label = " ".join(parts) if parts else "value"
if label in seen:
seen[label] += 1
label = f"{label} ({seen[label]})"
else:
seen[label] = 0
points = []
for ts, raw in values:
v = float(raw) if raw not in (None, "NaN", "+Inf", "-Inf") else None
points.append({"t": int(ts), "v": v})
series.append({"label": label, "points": points})
return series
```
This is a **direct extraction** of the label/dedup logic currently inside `GrafanaWidgetSource._fetch_chart`, retargeted at the Prometheus matrix shape (`{metric, values:[[ts,"str"],...]}`) instead of Grafana frames. The dedup rule is identical so users moving a `grafana/chart` widget to `prometheus/chart` see the same labels.
### 2.2 `PrometheusWidgetSource` — extend `.fetch()` by `widget_kind`
Today `.fetch()` does only the instant-query → `{"result": ...}` path. Extend it to dispatch by `widget_kind` while preserving the existing `metric` behavior byte-for-byte:
```python
class PrometheusWidgetSource:
async def fetch(self, service, widget_kind, config) -> dict[str, Any]:
if service is None:
return {"error": "Prometheus widget is missing its service"}
base_url = str(service.config.get("base_url") or "").rstrip("/")
timeout = int(service.config.get("timeout_seconds") or 10)
if widget_kind == "metric":
return await self._fetch_instant(base_url, timeout, config) # unchanged
if widget_kind == "chart":
return await self._fetch_chart(base_url, timeout, config) # NEW
if widget_kind == "gauge":
return await self._fetch_gauge(base_url, timeout, config) # NEW
if widget_kind == "mean":
return await self._fetch_mean(base_url, timeout, config) # NEW
return {"error": f"Unknown widget kind: {widget_kind}"}
```
**`_fetch_chart` (SC-101, SC-103, SC-104):**
```python
async def _fetch_chart(self, base_url, timeout, config) -> dict[str, Any]:
promql = config.get("promql")
if not promql:
return {"error": "promql is required"}
window = WINDOW_PRESETS.get(config.get("window", "1h"), WINDOW_PRESETS["1h"])
step = step_for_window(window)
end = int(time.time())
start = end - window
try:
resp = await asyncio.wait_for(asyncio.to_thread(
requests.get, f"{base_url}/api/v1/query_range",
params={"query": promql, "start": start, "end": end, "step": step},
timeout=timeout,
), timeout=timeout)
resp.raise_for_status()
payload = resp.json()
except asyncio.TimeoutError:
return {"error": "Prometheus query timed out"}
except requests.RequestException as exc:
return {"error": f"Prometheus query failed: {exc}"}
result = payload.get("data", {}).get("result", [])
return {"series": normalize_prometheus_matrix(result)}
```
**`_fetch_gauge` (SC-109, SC-110, SC-111) — instant query, scalar-only:**
```python
async def _fetch_gauge(self, base_url, timeout, config) -> dict[str, Any]:
promql = config.get("promql")
if not promql:
return {"error": "promql is required"}
raw = await self._instant_query(base_url, timeout, promql) # shared helper
if "error" in raw:
return raw
result = raw["result"]
if len(result) != 1:
return {"error": "Gauge requires a single-series query; refine your PromQL"}
# vector entry: {metric, value:[ts, "str"]}
try:
value = float(result[0]["value"][1])
except (KeyError, IndexError, ValueError, TypeError):
return {"error": "Gauge query returned no scalar value"}
return {
"value": value,
"warn_at": config.get("warn_at"),
"crit_at": config.get("crit_at"),
"min": config.get("min"),
"max": config.get("max"),
"unit": config.get("unit"),
}
```
`_instant_query` is extracted from the current `metric` path so `metric`/`gauge` share it; it returns `{"result": [...]}` or `{"error": ...}`.
**`_fetch_mean` (SC-112, SC-113, SC-114) — range query, client-side mean, scalar-only:**
```python
async def _fetch_mean(self, base_url, timeout, config) -> dict[str, Any]:
promql = config.get("promql")
if not promql:
return {"error": "promql is required"}
window = WINDOW_PRESETS.get(config.get("window", "1h"), WINDOW_PRESETS["1h"])
step = step_for_window(window)
end = int(time.time()); start = end - window
# ... run query_range, handle errors identically to _fetch_chart ...
result = payload.get("data", {}).get("result", [])
if len(result) != 1:
return {"error": "Mean requires a single-series query; refine your PromQL"}
points = result[0].get("values") or []
nums = [float(v) for _, v in points if v not in (None, "NaN", "+Inf", "-Inf")]
if not nums:
return {"error": "Mean query returned no numeric samples in the window"}
mean = sum(nums) / len(nums)
return {"value": mean, "unit": config.get("unit")}
```
The range-query HTTP call + error handling is shared between `_fetch_chart` and `_fetch_mean` via a private `_range_query(base_url, timeout, promql, window) -> dict` returning `{"matrix": result}` or `{"error": ...}`. This keeps the chart and mean paths DRY without inventing a generic adapter registry (non-goal enforced).
### 2.3 `integrations/prometheus.py` — declare new widget kinds
Add two Pydantic widget-config models and two `widget_kind(...)` entries; leave `PrometheusMetricWidgetConfig` and the existing `metric` kind untouched (non-goal: `prometheus_metric` stays as-is):
```python
class PrometheusChartWidgetConfig(WidgetConfigBase):
promql: str
window: str = "1h" # one of 1h/6h/24h/7d
class PrometheusGaugeWidgetConfig(WidgetConfigBase):
promql: str
warn_at: float | None = None
crit_at: float | None = None
min: float | None = None
max: float | None = None
unit: str | None = None
class PrometheusMeanWidgetConfig(WidgetConfigBase):
promql: str
window: str = "1h"
unit: str | None = None
```
`DEFINITION.widget_kinds` gains `chart`, `gauge`, `mean` (refresh 60s for chart/mean, 30s for gauge). Validation that `window ∈ {1h,6h,24h,7d}` can be a `field_validator` on the two models that use it, or simply enforced by `WINDOW_PRESETS.get(..., default)` server-side — prefer the latter (lenient) so a future preset addition doesn't require a model change.
### 2.4 `integrations/grafana.py` — delete; registry drops the entry
- Delete the file.
- In `integrations/registry.py`, drop the `from ...grafana import DEFINITION as GRAFANA` import and the `GRAFANA.service_type: GRAFANA,` line from `SERVICE_DEFINITIONS`. No other registry change.
### 2.5 `widgets/sources.py` — drop Grafana; wire nothing new
- Delete `GrafanaWidgetSource` and its `_fetch_chart`.
- Remove `"grafana": GrafanaWidgetSource(),` from `SERVICE_ADAPTERS`. (The label/dedup logic has already been *extracted* into `prometheus_range.normalize_prometheus_matrix` in §2.1; it is not lost when the Grafana class is deleted.)
- `PrometheusWidgetSource` gains the three new methods from §2.2.
### 2.6 `routers/monitoring.py` — remove `get_grafana_status`
Delete the `@router.get("/grafana-status")` endpoint (lines ~175195). No other monitoring change; `get_prometheus_status` / `get_alertmanager_status` stay.
### 2.7 No SettingsStore or DB schema change
Widget *instances* are stored generically (`service_id`, `widget_kind`, `config_json`). A `prometheus` `chart` instance is just a row with `service_id=<prom instance>` and `widget_kind="chart"`. Existing `grafana/chart` rows become orphans resolved by the existing "unknown widget" path (SC-119) — no migration code, no schema change (SC-126).
---
## 3. Frontend design
### 3.1 Rename + reuse: `GrafanaChartWidget``PrometheusChartWidget`
- Rename `frontend/src/widgets/GrafanaChartWidget.tsx``PrometheusChartWidget.tsx`; rename the exported function/component.
- The recharts body (`LineChart`, `Line`, `XAxis`, `YAxis`, `CartesianGrid`, `Tooltip`, `ResponsiveContainer`, `mergeSeries`, `CHART_COLORS`, `formatTime`) is **preserved unchanged** (SC-106).
- One tiny text fix: the empty-state Alert copy "Check your query and datasource_uid" becomes "Check your PromQL query and window." (`datasource_uid` no longer exists.)
- Rename test file `__tests__/GrafanaChartWidget.test.tsx``PrometheusChartWidget.test.tsx`; update import + the one error-message assertion (`"Grafana api_key is required"` → a Prom error string).
### 3.2 `integrations/registry.ts` — rebind + add
- Delete the entire `grafana` key from `SERVICE_REGISTRY`.
- In the `prometheus` binding's `widgets` array, add `chart`, `gauge`, `mean` alongside the existing `metric`. Each has a `configSchema` mirroring §2.3 (`promql`, `window` for chart/mean; `promql`, `warn_at`, `crit_at`, `min`, `max`, `unit` for gauge). `chart`/`mean` refresh 60s; `gauge` 30s.
- Import the renamed `PrometheusChartWidget` and the two new components.
### 3.3 New: `PrometheusGaugeWidget.tsx` (SC-109, SC-110, SC-111)
**Renderer choice: recharts `RadialBarChart`.** Justification: recharts is already a dependency (no new dep), `RadialBarChart` renders a single-value gauge with domain bands natively, and it shares tooltip/styling conventions with the chart widget — keeping the two visualizations consistent. The alternative (a ~50-line bespoke SVG gauge) was rejected because it would introduce a second rendering dialect for no benefit; the proposal's "fall back to SVG if recharts proves heavy" fallback remains documented but is not the default.
Threshold-band rendering: render three stacked `RadialBar` cells (green `0→warn`, amber `warn→crit`, red `crit→max`) as the track, and a fourth cell (the actual value) as the needle/bar. When `warn_at`/`crit_at` are absent, render a single neutral-color track. `min`/`max` default to `0`/`max(value, 1)` when omitted so the gauge has a sane domain. The component reuses `SectionCard` + `Alert`/`Skeleton` for loading/error states, matching every other widget.
Config fields surfaced to the user (matching §2.3): `promql`, `warn_at`, `crit_at`, `min`, `max`, `unit`.
### 3.4 New: `PrometheusMeanWidget.tsx` (SC-112, SC-113, SC-114)
A single-value display reusing the existing `MetricCard` pattern (already used by `BackupDashboardWidget` / `PrometheusMetricWidget`-adjacent tiles): big number, optional `unit` suffix, optional subtext showing the window ("mean over last 1h"). Loading/error/empty states via `Skeleton`/`Alert` as usual. No charting library involvement — it's a number, deliberately.
### 3.5 Grafana removal on the frontend
Per the source findings (§0), the removal targets are **not** an `ObservabilityPage` section. They are:
| File | Change |
|------|--------|
| `widgets/GrafanaLinkWidget.tsx` | **delete** |
| `widgets/GrafanaChartWidget.tsx` | **rename** to PrometheusChartWidget (§3.1) — not a pure delete |
| `widgets/index.ts` | drop `GrafanaLinkWidget` export; rename the chart export |
| `integrations/registry.ts` | drop `grafana` binding (§3.2) |
| `integrations/navEntries.ts` | drop the `grafana` entry from `SERVICE_TYPE_NAV_ENTRIES`; drop the now-unused `Link2` icon import |
| `hooks/useObservability.ts` | drop `useGrafanaStatus` + its `fetchGrafanaStatus` import |
| `api/client.ts` | drop `fetchGrafanaStatus` |
| `types/index.ts` | drop `GrafanaStatus` interface |
| `pages/service-tabs/LinksTab.tsx` | **delete** (it is the grafana-specific tab; its `GrafanaLinkCard` + machine deep-links are Grafana-only) |
| `pages/service-tabs/__tests__/LinksTab.test.tsx` | **delete** |
| `pages/service-tabs/index.ts` | drop the `LinksTab` import and the `case "grafana":` from `serviceContentTabs` |
| `pages/Dashboard.tsx` | drop `"grafana"` from `OBSERVABILITY_TYPES` set (line ~68) |
| `pages/ServicesPage.tsx` | update empty-state copy "Add a Grafana, Prometheus, …" → "Add a Prometheus, …" |
| `pages/__tests__/Dashboard.test.tsx` | the one "Grafana" label literal there is for a *shortcut* (a website link), unrelated to the Grafana service — **leave it** (grep-clean criterion SC-116 still passes; it's not a grafana service reference, just user-typed shortcut text in a test fixture). Flag for reviewer. |
**Spec-text note (not a design change):** SC-118 literally names "the ObservabilityPage Grafana section," which no longer exists. The *intent* of SC-118 ("Grafana status checks are removed") is satisfied by dropping `get_grafana_status` (§2.6) + `useGrafanaStatus`. The tasks phase should note this textual drift so the reviewer doesn't treat it as a missed requirement. If the parent prefers, SC-118 can be reworded in `spec.md` to name `LinksTab`/`useGrafanaStatus` instead; this design does not require that edit to proceed.
### 3.6 Types
`types/index.ts`: remove `GrafanaStatus`. No new widget-payload types — `chart` uses `{series}` (existing), `gauge` uses `{value, warn_at, crit_at, min, max, unit}` (all optional beyond `value`), `mean` uses `{value, unit?}`. These are read off `data?.data` untyped-as-before; no `WidgetDataResponse` generic change is needed (it's already `data: dict | None`).
---
## 4. Data flow
1. **Chart**`useWidgetData(widgetId, 60_000)``GET /api/widgets/instances/{id}/data``PrometheusWidgetSource.fetch(kind="chart")``_fetch_chart``GET {prom}/api/v1/query_range?query=...&start=...&end=...&step=...``normalize_prometheus_matrix``{"series":[...]}``PrometheusChartWidget` renders recharts (unchanged).
2. **Gauge**`useWidgetData(widgetId, 30_000)``fetch(kind="gauge")``_fetch_gauge` → instant query → scalar-only check → `{"value":..., "warn_at":..., ...}``PrometheusGaugeWidget` renders RadialBarChart.
3. **Mean**`useWidgetData(widgetId, 60_000)``fetch(kind="mean")``_fetch_mean` → range query → client-side mean → `{"value":..., "unit":...}``PrometheusMeanWidget` renders a MetricCard.
4. **Metric (unchanged)** — existing path preserved byte-for-byte.
5. **Orphaned grafana widget**`resolveWidget` finds no `grafana` binding → returns `undefined``WidgetInstanceCard` renders its existing "unknown widget" Alert (SC-119). No crash, no migration.
Errors at any step return `{"error": str}` (SC-103); the per-widget `Alert variant="destructive"` renders it and siblings keep polling.
---
## 5. Testing approach
### Backend
Extend `backend/tests/test_widgets.py` (and/or a focused `test_prometheus_range.py`):
- `normalize_prometheus_matrix`: feed a sample Prom `/api/v1/query_range` `data.result` (two entries, one with `__name__`, one colliding label) → assert `{series}` shape, label dedup, null handling for `"NaN"`.
- `step_for_window`: assert the 1h/6h/24h/7d → step mapping stays within 100300 points.
- `PrometheusWidgetSource` chart path: mock `requests.get` → assert `query_range` URL + params (`start`/`end`/`step` present, no `from_ts`/`to_ts`) and `{"series": ...}` return.
- Gauge scalar-only: mock instant query returning 2 series → assert `{"error": ...}`.
- Mean scalar-only + client-side mean: mock range query returning 1 series with known values → assert the arithmetic mean; mock 2 series → assert error.
- Error paths: timeout / `RequestException``{"error": ...}` (no raise).
### Frontend
- `PrometheusChartWidget.test.tsx` (renamed from GrafanaChartWidget test): loading, error, rendered-data cases (SC-125).
- `PrometheusGaugeWidget.test.tsx`: loading, error, rendered-with-bands, rendered-without-bands.
- `PrometheusMeanWidget.test.tsx`: loading, error, rendered value.
- `registry.test.ts`: assert no `grafana` binding; assert `prometheus` binding has `metric`, `chart`, `gauge`, `mean`.
### Build/lint gates
Each slice: `PYTHONPATH=src pytest` (from `backend/`), `npm run build`, `npm run lint` must be green.
---
## 6. File-level plan
### Create
| File | Rationale |
|------|-----------|
| `backend/src/media_library_viewer_api/widgets/prometheus_range.py` | `WINDOW_PRESETS`, `step_for_window`, `normalize_prometheus_matrix` (SC-102, SC-104). |
| `backend/tests/test_prometheus_range.py` | Unit tests for the helpers. |
| `frontend/src/widgets/PrometheusGaugeWidget.tsx` | recharts RadialBarChart gauge (SC-109/110/111). |
| `frontend/src/widgets/PrometheusMeanWidget.tsx` | MetricCard-style mean (SC-112/113/114). |
| `frontend/src/widgets/__tests__/PrometheusGaugeWidget.test.tsx` | SC-125. |
| `frontend/src/widgets/__tests__/PrometheusMeanWidget.test.tsx` | SC-125. |
### Rename (git mv)
| From → To | Rationale |
|-----------|-----------|
| `widgets/GrafanaChartWidget.tsx``widgets/PrometheusChartWidget.tsx` | SC-105/106. Body preserved. |
| `widgets/__tests__/GrafanaChartWidget.test.tsx``widgets/__tests__/PrometheusChartWidget.test.tsx` | match rename. |
### Modify
| File | Rationale |
|------|-----------|
| `backend/.../widgets/sources.py` | Drop `GrafanaWidgetSource`; add chart/gauge/mean to `PrometheusWidgetSource`. |
| `backend/.../integrations/prometheus.py` | Add 3 widget-config models + 3 `widget_kind` entries. |
| `backend/.../integrations/registry.py` | Drop grafana import + entry. |
| `backend/.../routers/monitoring.py` | Drop `get_grafana_status`. |
| `backend/tests/test_widgets.py` | Drop grafana adapter tests; add prom chart/gauge/mean tests. |
| `frontend/src/integrations/registry.ts` | Drop grafana binding; add chart/gauge/mean to prometheus. |
| `frontend/src/integrations/navEntries.ts` | Drop grafana entry. |
| `frontend/src/hooks/useObservability.ts` | Drop `useGrafanaStatus`. |
| `frontend/src/api/client.ts` | Drop `fetchGrafanaStatus`. |
| `frontend/src/types/index.ts` | Drop `GrafanaStatus`. |
| `frontend/src/widgets/index.ts` | Drop GrafanaLinkWidget export; rename chart export. |
| `frontend/src/pages/service-tabs/index.ts` | Drop LinksTab import + grafana case. |
| `frontend/src/pages/Dashboard.tsx` | Drop "grafana" from `OBSERVABILITY_TYPES`. |
| `frontend/src/pages/ServicesPage.tsx` | Update empty-state copy. |
| `openspec/config.yaml` | Repeal stale thin-dashboard/no-recharts wording (SC-121). |
| `CHANGELOG.md` | Migration note (SC-122). |
### Delete
| File | Rationale |
|------|-----------|
| `backend/.../integrations/grafana.py` | SC-117. |
| `frontend/src/widgets/GrafanaLinkWidget.tsx` | SC-117. |
| `frontend/src/pages/service-tabs/LinksTab.tsx` | Grafana-only tab; no Prom equivalent needed (chart widget covers viz). |
| `frontend/src/pages/service-tabs/__tests__/LinksTab.test.tsx` | matches deletion. |
---
## 7. Slice boundaries (≤400 changed lines each)
Each slice leaves `pytest` + `npm run build` + `npm run lint` green and the app in a working state.
### Slice 1 — Prom range path + chart rebrand + shared helper (foundational)
**Files:** create `prometheus_range.py` + test; rename `GrafanaChartWidget``PrometheusChartWidget` (+ test); modify `sources.py` (`_fetch_chart`, drop nothing yet), `integrations/prometheus.py` (add `chart` kind only), `registry.ts` (move `chart` grafana→prometheus), `widgets/index.ts`, `registry.test.ts`.
**State after S1:** a `prometheus/chart` widget renders from direct Prom `query_range`; the `grafana/chart` binding is gone but `GrafanaWidgetSource`/`GrafanaLinkWidget`/grafana service type still exist (removal is S3). Grafana link widgets still work. ~300380 lines.
### Slice 2 — Gauge + mean widgets (additive)
**Files:** create `PrometheusGaugeWidget.tsx` (+test), `PrometheusMeanWidget.tsx` (+test); modify `sources.py` (`_fetch_gauge`, `_fetch_mean`, shared `_instant_query`/`_range_query`), `integrations/prometheus.py` (add `gauge`/`mean` kinds), `registry.ts` (add gauge/mean bindings), `test_widgets.py` (add adapter tests).
**State after S2:** gauge + mean widgets selectable and rendering; no grafana change. ~300380 lines.
### Slice 3 — Grafana removal + config + changelog (cleanup)
**Files:** delete `integrations/grafana.py`, `GrafanaLinkWidget.tsx`, `LinksTab.tsx` (+test); modify `sources.py` (drop `GrafanaWidgetSource` + `SERVICE_ADAPTERS` entry), `registry.py`, `monitoring.py`, `navEntries.ts`, `useObservability.ts`, `client.ts`, `types/index.ts`, `widgets/index.ts`, `service-tabs/index.ts`, `Dashboard.tsx`, `ServicesPage.tsx`, `test_widgets.py` (drop grafana tests); rewrite `config.yaml`; add `CHANGELOG.md` entry.
**State after S3:** grep-clean (SC-115/116), config accurate (SC-121), migration documented (SC-122). ~250350 lines (mostly deletions).
**Order:** S1 → S2 → S3. S1 and S2 are independently shippable; S3 must follow S1 (it removes the grafana chart binding S1 replaces).
---
## 8. Decisions log (answers to spec downstream-notes)
1. **Step derivation (SC-104):** `step_for_window(window_seconds, target_points=200) = max(15, round(window_seconds/200))`, in `widgets/prometheus_range.py`. Table in §2.1.
2. **Shared normalization (SC-102):** `normalize_prometheus_matrix(result)` in `widgets/prometheus_range.py`; extracts the dedup rule from the to-be-deleted Grafana path; retargeted at Prom matrix shape. Signature in §2.1.
3. **Gauge renderer (SC-110):** recharts `RadialBarChart` (no new dep, consistent styling). Threshold bands via stacked track cells; neutral single-color when `warn_at`/`crit_at` absent. Config fields: `promql, warn_at, crit_at, min, max, unit`. §3.3.
4. **Mean adapter (SC-112/113):** `_fetch_mean` runs `query_range` over the window preset, averages non-null samples of the single series client-side, returns `{value, unit?}`. Scalar-only enforced via `len(result) != 1 → error`. §2.2.
5. **Slice plan:** 3 slices (S1 range+rebrand+helper, S2 gauge+mean, S3 grafana removal+config+changelog), each ≤400 lines, order S1→S2→S3. §7.
---
## 9. Open items for the tasks/apply phases
- Implementer should `git mv` the chart widget/test so history is preserved (not delete+create).
- Implementer should verify the recharts `RadialBarChart` gauge renders acceptably; if it proves fiddly, the proposal's SVG fallback (~50 lines) is sanctioned — but try recharts first.
- SC-118 text names a file that no longer exists; tasks phase should record this so verify doesn't flag it as a miss. Intent is satisfied by removing `get_grafana_status` + `useGrafanaStatus` + `LinksTab`.
- `Dashboard.test.tsx` contains the literal "Grafana" in a shortcut test fixture (not a grafana service reference) — grep for SC-116 should be scoped to service/widget references, or that line whitelisted. Flag for reviewer.
@@ -0,0 +1,131 @@
# SDD Proposal: Prometheus Direct Charting (drop Grafana middleman)
**Change:** `prometheus-direct-charting`
**Phase:** proposal
**Date:** 2026-07-08
## 1. Problem / Why Now
Manage already does in-app charting — but indirectly. Today the only chart path is:
```
GrafanaChartWidget (recharts) ← GrafanaWidgetSource._fetch_chart
← POST {grafana}/api/ds/query (datasource.type = "prometheus", hardcoded)
← Grafana proxies to Prometheus
```
Two problems with this:
1. **Grafana is a pure middleman.** The backend already hardcodes `datasource.type: "prometheus"` in every chart query. Grafana adds a hop, an API key, a different response shape (`/api/ds/query` frames), and a normalization layer — all to reach a Prometheus instance Manage could query directly via `/api/v1/query_range`. The existing `PrometheusWidgetSource` already proves Manage can talk to Prom straight; it just only does *instant* queries today.
2. **The project's documented rules are stale.** `openspec/config.yaml` says "Do NOT re-implement charting in-app" and "No recharts/d3 is in use." Both are already false in code: `recharts ^3.9.2` is declared and imported by `GrafanaChartWidget`. The config documents a reality the code left behind.
Meanwhile the operator wants two more metric visualizations Manage cannot currently render: a **gauge** and a **single value / mean-over-time** widget. Both belong naturally on the `prometheus` service, which today only exposes an instant-query numeric widget.
The clean answer is to stop routing charts through Grafana: query Prometheus directly, reuse the rendering infrastructure already written, add the two new modes, and remove the now-redundant Grafana surface.
## 2. Target Users and Situations
- **Primary users:** Homelab operators who want metric visualizations on the Manage dashboard without bouncing to Grafana for a quick glance.
- **Workflow moments:**
- Glance at the dashboard: see a trend line, a gauge, or a mean value for a key PromQL query.
- Build a custom at-a-glance panel from any PromQL expression Manage can already evaluate.
- Decommission the Grafana hop for charts (one fewer external dependency in the chart path, one fewer API key to rotate).
- **Urgency:** Medium. Charts already work via Grafana today; this is a simplification plus two new widget modes, not an outage fix.
## 3. Product Outcome
After this change, an authenticated user can:
- Place a **Prometheus Chart** widget (line chart, multi-series) backed by a direct `/api/v1/query_range` call — same look as today's Grafana chart, no Grafana required.
- Place a **Prometheus Gauge** widget rendering an instant PromQL scalar as a gauge.
- Place a **Prometheus Mean** widget rendering a single value aggregated over a time window (e.g. `avg_over_time(...)`, or a query_range aggregate).
- Manage the dashboard without any Grafana service configured: the Grafana service type, link widget, chart widget, and status checks are removed.
## 4. Scope Boundaries and Non-Goals
### In scope
- **Direct Prometheus range query path** — backend hits `/api/v1/query_range` and returns the existing `{series:[{label,points}]}` shape so the frontend renderer is reused unchanged.
- **Shared series normalization** — extract the metric-label → readable-label logic currently inside `GrafanaWidgetSource._fetch_chart` into a reusable helper used by both the chart path and (where relevant) the new modes.
- **Rebrand + rebind**`GrafanaChartWidget``PrometheusChartWidget`, moved from the `grafana` service to the `prometheus` service as the `chart` widget kind.
- **New `prometheus` widget kinds:** `gauge` and `mean` (semantics confirmed in the question round).
- **Grafana removal** — delete `integrations/grafana.py`, `GrafanaWidgetSource`, `GrafanaLinkWidget`, `get_grafana_status`, the ObservabilityPage Grafana section, both registries' `grafana` entries, nav entries, and their tests.
- **`config.yaml` rewrite** — replace the stale thin-dashboard / no-recharts wording with the reality: in-app charting via `recharts` is the sanctioned approach for Prometheus-backed series; Grafana is no longer referenced.
- **Migration note** — CHANGELOG entry: existing Grafana service instances must be deleted and recreated as Prometheus services (true data migration is impossible; different URLs).
### Non-goals (explicitly out of scope)
- **A general chart-widget framework with pluggable data-source middlewares.** Chart rendering is reused; data sources are not abstracted into a swappable adapter registry. Two concrete sources (Prometheus range query, and later the service-storage harness) are wired directly where needed.
- **Grafana datasource proxy for non-Prometheus sources** (Loki, InfluxDB, Postgres). If a real non-Prom need appears later, it is a separate change.
- **Embedding Grafana panels as images/iframes.** Grafana is removed, not embedded.
- **Editing PromQL in a rich editor** (autocomplete, metric explorer). Plain text input only, matching today's `prometheus_metric` widget.
- **qBittorrent widgets and the service-storage harness.** Those are a separate change (`service-storage-harness`); only the qBit speed widget *depends on* this change's chart capability.
- **Touching `prometheus_metric` (instant numeric widget).** It stays as-is; `mean` and `chart` are sibling kinds, not modes bolted onto it.
- **Re-indexing or migrating existing widget instance rows automatically.** Existing `grafana` `chart` widgets are orphaned by the removal and must be recreated as `prometheus` `chart` widgets by the operator (documented in CHANGELOG).
## 5. High-Level Approach
### 5.1 Backend
1. **Prometheus range query** — extend `PrometheusWidgetSource` (or add a sibling code path) to handle `widget_kind == "chart"`:
- `GET {base_url}/api/v1/query_range?query=...&start=...&end=...&step=...`
- Parse Prom `{value:[ts, val]}` matrix into the existing `{series:[{label, points}]}` shape.
2. **Shared normalization** — move the "metric labels → readable label" logic out of `GrafanaWidgetSource._fetch_chart` into `widgets/series.py` (or similar), so the Prom path and any future consumer reuse it.
3. **New kinds wiring**`gauge` and `mean` resolve in `PrometheusWidgetSource.fetch`:
- `gauge`: instant query (`/api/v1/query`), return `{value, threshold?, ...}` for a gauge renderer.
- `mean`: range query aggregated to a single value (either PromQL `avg_over_time` via instant query, or client-side mean over a query_range window). Semantics decided in the question round.
4. **Integration update**`integrations/prometheus.py` declares the three widget kinds (`metric`, `chart`, `gauge`, `mean`) with config schemas (`promql`, plus range params for `chart`/`mean`).
5. **Grafana removal** — delete `integrations/grafana.py`, drop `grafana` from `SERVICE_ADAPTERS`, `SERVICE_DEFINITIONS`, remove `_fetch_chart` and the Grafana link logic. Remove `get_grafana_status` from `routers/monitoring.py` and the Grafana branch from `ObservabilityPage`.
6. **`config.yaml` rewrite** — replace the stale charting rules with accurate wording.
### 5.2 Frontend
1. **Rebrand** — rename `GrafanaChartWidget.tsx``PrometheusChartWidget.tsx`; the recharts rendering (`LineChart`/`Line`/`XAxis`/`YAxis`/`Tooltip`/`mergeSeries`/`CHART_COLORS`) stays essentially unchanged.
2. **Registry** — in `integrations/registry.ts`, move `chart` to the `prometheus` binding and add `gauge` + `mean` bindings; delete the entire `grafana` binding.
3. **New components:**
- `PrometheusGaugeWidget.tsx` — recharts `<RadialBarChart>` or a small SVG gauge; instant value.
- `PrometheusMeanWidget.tsx` — single-value display (reuses `MetricCard`-style rendering) of the windowed mean.
4. **Nav + ObservabilityPage** — remove Grafana nav entries and the Grafana status card.
5. **Types**`frontend/src/types/index.ts` drops Grafana status types; no new endpoint types (data still flows through `useWidgetData`).
### 5.3 Type contracts
- Backend: update `integrations/prometheus.py` widget-kind config models; remove Grafana models.
- Frontend: remove `GrafanaStatus` type; widget payloads stay `{series}` / `{value}` shaped.
## 6. Success Criteria / Acceptance Criteria
1. A user can configure a Prometheus service and place `chart`, `gauge`, `mean`, and `metric` widgets without any Grafana service present.
2. The `chart` widget renders multi-series line charts from `/api/v1/query_range` with the same look as the prior Grafana-backed chart.
3. The `gauge` widget renders an instant PromQL scalar as a gauge.
4. The `mean` widget renders a single value aggregated over the configured window.
5. No `grafana` references remain in `backend/src` or `frontend/src` (grep clean).
6. `openspec/config.yaml` no longer claims "no recharts" or "do not chart in-app"; its wording matches the implementation.
7. Existing `pytest`, `npm run build`, and `npm run lint` stay green; Grafana tests are removed, Prom chart/gauge/mean tests are added.
8. CHANGELOG documents the migration (delete Grafana services, recreate as Prometheus).
## 7. Risks and Mitigations
| Risk | Mitigation |
|------|------------|
| **Response-shape regression.** Prom `/api/v1/query_range` matrix differs from Grafana `/api/ds/query` frames. | Extract normalization into a shared helper; add a backend test that feeds a sample Prom range response and asserts the `{series}` shape the frontend already consumes. |
| **Orphaned configured widgets.** Existing `grafana/chart` widget rows break at render. | Acceptable + documented in CHANGELOG; the widget resolves to "unknown widget" gracefully (existing `WidgetInstance` error path). No silent data corruption. |
| **Gauge rendering complexity.** recharts gauges can be fiddly. | Constrain gauge to a single scalar + optional thresholds; if recharts gauge proves heavy, fall back to a ~50-line SVG gauge (contained, no new dep). |
| **Scope creep into a generic chart framework.** Tempting to abstract data sources. | Non-goal enforced: two direct wirings, no adapter registry. |
| **Stale docs/tests lingering.** | "No grafana references" acceptance criterion (grep) catches leftovers. |
| **Review budget (>400 lines).** | Slice into chained PRs (e.g. Slice 1: Prom range query + rebrand chart; Slice 2: gauge + mean; Slice 3: Grafana removal + config rewrite). Each slice leaves build/lint/test green. |
| **DataGrid migration (config rule callout).** | Not applicable — no DataGrid work here. The key technical risk is the response-shape regression above. |
## 8. Open Questions (for the proposal question round)
- **Q1 — `mean` semantics.** Default: client-side mean over a `query_range` window (e.g. last 1h, step 30s → average all returned values). Alternative: require the user to write `avg_over_time(...)` in the PromQL and just run an instant query. Which UX do you want?
- **Q2 — `gauge` thresholds.** Default: green/amber/red bands at user-configured thresholds (e.g. 70%/90%). Alternative: single color, no bands (simplest).
- **Q3 — Chart time window config.** Default: expose `from`/`to`/`step` (or a simpler "window" preset like 1h/6h/24h). Confirm the granularity users can configure.
- **Q4 — Multi-series on `mean`/`gauge`.** Default: single series only (one scalar). Confirm we do not try to render multi-series gauges.
## 9. Future Phases
1. **Service-storage harness integration** — the qBittorrent speed widget (separate `service-storage-harness` change) reuses this change's chart rendering with an in-service data source wired directly.
2. **Rich PromQL editing** — metric explorer / autocomplete.
3. **Non-Prom datasources** — only if a concrete need (Loki logs in-app) appears.
4. **Threshold-based alerting from chart widgets** — surface bands/lines from Alertmanager rules.
@@ -0,0 +1,152 @@
# SDD Spec: Prometheus Direct Charting (drop Grafana middleman)
**Change:** `prometheus-direct-charting`
**Phase:** spec
**Date:** 2026-07-08
This spec defines the acceptance requirements for the change. Each requirement is testable. Requirements derived from `proposal.md` §6 (success criteria) and the resolved §8 question round.
## Requirement categories
1. Direct Prometheus range query path
2. Prometheus chart widget (rebrand + rebind)
3. Prometheus gauge widget
4. Prometheus mean widget
5. Grafana removal
6. Configuration documentation accuracy
7. Test and build greenness
8. Migration guidance
---
## 1. Direct Prometheus range query path
### SC-101 — Prometheus range query returns the existing series shape
When a `prometheus` widget of kind `chart` is fetched, the backend MUST query `{base_url}/api/v1/query_range` with `query`, `start`, `end`, and `step` derived from the widget config, and return a payload of shape `{ "series": [{ "label": str, "points": [{ "t": int, "v": float|null }] }] }` — the exact shape the frontend chart renderer already consumes.
### SC-102 — Series label normalization is shared and Prometheus-native
The metric-label → readable-label normalization MUST live in a single shared helper (not duplicated in a Grafana path) and MUST produce meaningful labels for Prometheus matrix results, including deduplicating repeated labels via a `label (n)` suffix.
### SC-103 — Range query errors degrade gracefully
A Prometheus timeout, connection error, or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional.
### SC-104 — Step is derived from the window preset
Given a window preset (1h / 6h / 24h / 7d), the backend MUST derive a `step` that yields a reasonable number of points (target ~100300 points). Users do not configure `step` directly.
## 2. Prometheus chart widget (rebrand + rebind)
### SC-105 — Chart widget moves from grafana to prometheus
A widget kind named `chart` MUST be bound to the `prometheus` service type in both the backend registry and the frontend `SERVICE_REGISTRY`. The `grafana` service type MUST NOT offer a `chart` kind.
### SC-106 — Chart renderer is reused unchanged
The recharts rendering (line chart, multi-series, axes, tooltip, `mergeSeries`, color tokens) MUST be preserved in the rebranded `PrometheusChartWidget`. The rename is structural; the rendering code is not rewritten.
### SC-107 — Chart supports multiple series
The `chart` widget MUST render all series returned by the range query, each as its own line with a distinct color. There is no single-series restriction on `chart`.
### SC-108 — Chart window is a preset
The `chart` widget config MUST expose the time window as a preset selector (`1h`, `6h`, `24h`, `7d`), not raw `from`/`to`/`step` fields. The preset is stored in widget config and resolved to `start`/`end` server-side.
## 3. Prometheus gauge widget
### SC-109 — Gauge renders an instant scalar
A widget kind named `gauge` MUST be bound to the `prometheus` service. Its data fetch MUST run an instant PromQL query (`/api/v1/query`) and return the scalar result for rendering as a gauge.
### SC-110 — Gauge supports configurable threshold bands
The `gauge` widget config MUST accept optional threshold values (e.g. `warn_at`, `crit_at`) and the renderer MUST display green / amber / red bands accordingly. When thresholds are omitted, the gauge renders with a single neutral color and no bands.
### SC-111 — Gauge is scalar-only
The `gauge` widget MUST render exactly one scalar value. If the instant query returns multiple series, the adapter MUST return `{ "error": str }` (not silently pick one), directing the user to refine the PromQL.
## 4. Prometheus mean widget
### SC-112 — Mean computes client-side over a window
A widget kind named `mean` MUST be bound to the `prometheus` service. Its data fetch MUST run a range query over the configured window preset and return the arithmetic mean of all non-null point values as a single scalar.
### SC-113 — Mean uses plain PromQL + window preset
The `mean` widget config MUST accept a plain PromQL expression (no requirement to wrap in `avg_over_time`) plus a window preset. Users do not write range-vector functions.
### SC-114 — Mean is scalar-only
The `mean` widget MUST render exactly one scalar value. If the range query returns multiple series, the adapter MUST return `{ "error": str }` (not silently aggregate across series).
## 5. Grafana removal
### SC-115 — No grafana references in backend source
After the change, `grep -ri grafana backend/src --include='*.py'` MUST return no matches (excluding comments/changelog that are explicitly about the removal, if any are retained — but ideally zero).
### SC-116 — No grafana references in frontend source
After the change, `grep -ri grafana frontend/src` MUST return no matches, **excluding** (a) test fixtures where "Grafana" appears as a user-authored dashboard *shortcut label* unrelated to the grafana service type (e.g. `Dashboard.test.tsx`), and (b) `LinksTab.tsx` / `service-tabs/index.ts` lines that are themselves being deleted as part of SC-118. (Source finding: `ObservabilityPage.tsx` was refactored into `service-tabs/`.)
### SC-117 — Grafana service type is gone from registries
Neither the backend `SERVICE_DEFINITIONS` / `SERVICE_ADAPTERS` nor the frontend `SERVICE_REGISTRY` / `BUILTIN_WIDGETS` MUST contain a `grafana` entry. The `integrations/grafana.py` file MUST be deleted.
### SC-118 — Grafana status checks and UI sections are removed
The `get_grafana_status` endpoint and its frontend hook (`useGrafanaStatus`) MUST be removed. The UI surface previously in `ObservabilityPage.tsx` has been refactored into a per-service-type `service-tabs/` architecture; the Grafana removal targets are therefore `service-tabs/LinksTab.tsx` + its test, the `grafana` case in `service-tabs/index.ts`, the grafana entry in `integrations/navEntries.ts`, the `grafana` member of `Dashboard.tsx`'s `OBSERVABILITY_TYPES` set, and any Grafana empty-state copy in `ServicesPage.tsx`. The literal `ObservabilityPage.tsx` no longer exists; SC-118's *intent* (no Grafana UI surface) is what is verified.
### SC-119 — Grafana widget instances degrade gracefully
An existing persisted widget row referencing a `grafana` service MUST NOT crash the dashboard. It resolves to the existing "unknown widget" error state and surfaces a clear message; the operator can then delete it.
### SC-120 — Grafana tests are removed
All Grafana-specific tests (backend and frontend) MUST be deleted; no test references grafana.
## 6. Configuration documentation accuracy
### SC-121 — config.yaml matches implementation
`openspec/config.yaml` MUST NOT contain the stale claims "Do NOT re-implement charting in-app" or "No recharts/d3 is in use." It MUST reflect that in-app charting via `recharts` is the sanctioned approach for Prometheus-backed series, and MUST NOT reference Grafana as a chart path.
### SC-122 — CHANGELOG documents the migration
`CHANGELOG.md` MUST include an entry instructing operators to delete existing Grafana service instances and recreate them as Prometheus services, noting that configured `grafana/chart` widgets must be recreated as `prometheus/chart` widgets.
## 7. Test and build greenness
### SC-123 — Backend tests pass
`pytest` run from `backend/` MUST pass, including new tests covering: Prom range query → `{series}` normalization, gauge scalar-only enforcement, mean client-side aggregation, and the shared label helper.
### SC-124 — Frontend typechecks, builds, and lints
`npm run build` (which runs `tsc -b` + `vite build`) and `npm run lint` from `frontend/` MUST pass.
### SC-125 — New widget kinds have tests
`PrometheusChartWidget`, `PrometheusGaugeWidget`, and `PrometheusMeanWidget` MUST each have a frontend test covering at least: loading state, error state, and a rendered data case.
## 8. Migration guidance
### SC-126 — No silent data migration
The change MUST NOT attempt to auto-migrate existing `grafana` service rows into `prometheus` rows (URLs differ; true migration is impossible). Migration is operator-driven per the CHANGELOG note.
### SC-127 — Non-blocking on the service-storage-harness change
This change MUST NOT depend on the `service-storage-harness` change. It is independently buildable, testable, and deployable. (The reverse dependency holds: the qBit speed widget depends on this change's chart capability.)
---
## Notes for downstream phases
- **Design (next phase)** should specify: the exact `step`-derivation function for window presets (SC-104), the shared normalization helper's location and signature (SC-102), and whether the gauge renderer uses recharts `RadialBarChart` or a contained SVG (SC-110).
- **Tasks** should slice into chained PRs ≤400 lines per `config.yaml` rules: e.g. (1) Prom range path + chart rebrand, (2) gauge + mean, (3) Grafana removal + config rewrite + CHANGELOG.
- The **review-budget guard** applies: if total changed lines exceed ~400, the chained-PR strategy from the `tasks` phase is mandatory.
@@ -0,0 +1,135 @@
# Prometheus Charting — Delta (`prometheus-direct-charting`)
> Change: `prometheus-direct-charting` · Domain: `prometheus-charting` · Phase: **spec** (reconciled during `sdd-sync`).
> Distilled verbatim from the verified flat `spec.md` (27 requirements, SC-101 … SC-127) of change
> `prometheus-direct-charting`, cross-referenced against `design.md` and `verify-report.md`. Captures
> the **durable, post-change end-state contracts** for direct Prometheus-backed metric visualization
> and the Grafana removal that established the new model.
## ADDED Requirements
> The canonical `openspec/specs/prometheus-charting/spec.md` did not exist before this change. All
> requirements below are therefore **ADDED** to a new `prometheus-charting` domain; `sdd-sync` copies
> them into the canonical spec (native helper rule: when the canonical spec does not exist, the
> change spec becomes the new canonical spec).
>
> Requirement IDs (SC-101 … SC-127) and body text are preserved **exactly** from the verified flat
> `spec.md`, including the patched SC-116 / SC-118 wording. Requirements are grouped logically and
> listed in the following group order:
>
> - **Direct Prometheus range query path** — SC-101 … SC-104
> - **Prometheus chart widget (rebrand + rebind)** — SC-105 … SC-108
> - **Prometheus gauge widget** — SC-109 … SC-111
> - **Prometheus mean widget** — SC-112 … SC-114
> - **Grafana removal** — SC-115 … SC-120
> - **Configuration documentation accuracy** — SC-121 … SC-122
> - **Test and build greenness** — SC-123 … SC-125
> - **Migration guidance** — SC-126 … SC-127
### Requirement: SC-101 — Prometheus range query returns the existing series shape
When a `prometheus` widget of kind `chart` is fetched, the backend MUST query `{base_url}/api/v1/query_range` with `query`, `start`, `end`, and `step` derived from the widget config, and return a payload of shape `{ "series": [{ "label": str, "points": [{ "t": int, "v": float|null }] }] }` — the exact shape the frontend chart renderer already consumes.
### Requirement: SC-102 — Series label normalization is shared and Prometheus-native
The metric-label → readable-label normalization MUST live in a single shared helper (not duplicated in a Grafana path) and MUST produce meaningful labels for Prometheus matrix results, including deduplicating repeated labels via a `label (n)` suffix.
### Requirement: SC-103 — Range query errors degrade gracefully
A Prometheus timeout, connection error, or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional.
### Requirement: SC-104 — Step is derived from the window preset
Given a window preset (1h / 6h / 24h / 7d), the backend MUST derive a `step` that yields a reasonable number of points (target ~100300 points). Users do not configure `step` directly.
### Requirement: SC-105 — Chart widget moves from grafana to prometheus
A widget kind named `chart` MUST be bound to the `prometheus` service type in both the backend registry and the frontend `SERVICE_REGISTRY`. The `grafana` service type MUST NOT offer a `chart` kind.
### Requirement: SC-106 — Chart renderer is reused unchanged
The recharts rendering (line chart, multi-series, axes, tooltip, `mergeSeries`, color tokens) MUST be preserved in the rebranded `PrometheusChartWidget`. The rename is structural; the rendering code is not rewritten.
### Requirement: SC-107 — Chart supports multiple series
The `chart` widget MUST render all series returned by the range query, each as its own line with a distinct color. There is no single-series restriction on `chart`.
### Requirement: SC-108 — Chart window is a preset
The `chart` widget config MUST expose the time window as a preset selector (`1h`, `6h`, `24h`, `7d`), not raw `from`/`to`/`step` fields. The preset is stored in widget config and resolved to `start`/`end` server-side.
### Requirement: SC-109 — Gauge renders an instant scalar
A widget kind named `gauge` MUST be bound to the `prometheus` service. Its data fetch MUST run an instant PromQL query (`/api/v1/query`) and return the scalar result for rendering as a gauge.
### Requirement: SC-110 — Gauge supports configurable threshold bands
The `gauge` widget config MUST accept optional threshold values (e.g. `warn_at`, `crit_at`) and the renderer MUST display green / amber / red bands accordingly. When thresholds are omitted, the gauge renders with a single neutral color and no bands.
### Requirement: SC-111 — Gauge is scalar-only
The `gauge` widget MUST render exactly one scalar value. If the instant query returns multiple series, the adapter MUST return `{ "error": str }` (not silently pick one), directing the user to refine the PromQL.
### Requirement: SC-112 — Mean computes client-side over a window
A widget kind named `mean` MUST be bound to the `prometheus` service. Its data fetch MUST run a range query over the configured window preset and return the arithmetic mean of all non-null point values as a single scalar.
### Requirement: SC-113 — Mean uses plain PromQL + window preset
The `mean` widget config MUST accept a plain PromQL expression (no requirement to wrap in `avg_over_time`) plus a window preset. Users do not write range-vector functions.
### Requirement: SC-114 — Mean is scalar-only
The `mean` widget MUST render exactly one scalar value. If the range query returns multiple series, the adapter MUST return `{ "error": str }` (not silently aggregate across series).
### Requirement: SC-115 — No grafana references in backend source
After the change, `grep -ri grafana backend/src --include='*.py'` MUST return no matches (excluding comments/changelog that are explicitly about the removal, if any are retained — but ideally zero).
### Requirement: SC-116 — No grafana references in frontend source
After the change, `grep -ri grafana frontend/src` MUST return no matches, **excluding** (a) test fixtures where "Grafana" appears as a user-authored dashboard *shortcut label* unrelated to the grafana service type (e.g. `Dashboard.test.tsx`), and (b) `LinksTab.tsx` / `service-tabs/index.ts` lines that are themselves being deleted as part of SC-118. (Source finding: `ObservabilityPage.tsx` was refactored into `service-tabs/`.)
### Requirement: SC-117 — Grafana service type is gone from registries
Neither the backend `SERVICE_DEFINITIONS` / `SERVICE_ADAPTERS` nor the frontend `SERVICE_REGISTRY` / `BUILTIN_WIDGETS` MUST contain a `grafana` entry. The `integrations/grafana.py` file MUST be deleted.
### Requirement: SC-118 — Grafana status checks and UI sections are removed
The `get_grafana_status` endpoint and its frontend hook (`useGrafanaStatus`) MUST be removed. The UI surface previously in `ObservabilityPage.tsx` has been refactored into a per-service-type `service-tabs/` architecture; the Grafana removal targets are therefore `service-tabs/LinksTab.tsx` + its test, the `grafana` case in `service-tabs/index.ts`, the grafana entry in `integrations/navEntries.ts`, the `grafana` member of `Dashboard.tsx`'s `OBSERVABILITY_TYPES` set, and any Grafana empty-state copy in `ServicesPage.tsx`. The literal `ObservabilityPage.tsx` no longer exists; SC-118's *intent* (no Grafana UI surface) is what is verified.
### Requirement: SC-119 — Grafana widget instances degrade gracefully
An existing persisted widget row referencing a `grafana` service MUST NOT crash the dashboard. It resolves to the existing "unknown widget" error state and surfaces a clear message; the operator can then delete it.
### Requirement: SC-120 — Grafana tests are removed
All Grafana-specific tests (backend and frontend) MUST be deleted; no test references grafana.
### Requirement: SC-121 — config.yaml matches implementation
`openspec/config.yaml` MUST NOT contain the stale claims "Do NOT re-implement charting in-app" or "No recharts/d3 is in use." It MUST reflect that in-app charting via `recharts` is the sanctioned approach for Prometheus-backed series, and MUST NOT reference Grafana as a chart path.
### Requirement: SC-122 — CHANGELOG documents the migration
`CHANGELOG.md` MUST include an entry instructing operators to delete existing Grafana service instances and recreate them as Prometheus services, noting that configured `grafana/chart` widgets must be recreated as `prometheus/chart` widgets.
### Requirement: SC-123 — Backend tests pass
`pytest` run from `backend/` MUST pass, including new tests covering: Prom range query → `{series}` normalization, gauge scalar-only enforcement, mean client-side aggregation, and the shared label helper.
### Requirement: SC-124 — Frontend typechecks, builds, and lints
`npm run build` (which runs `tsc -b` + `vite build`) and `npm run lint` from `frontend/` MUST pass.
### Requirement: SC-125 — New widget kinds have tests
`PrometheusChartWidget`, `PrometheusGaugeWidget`, and `PrometheusMeanWidget` MUST each have a frontend test covering at least: loading state, error state, and a rendered data case.
### Requirement: SC-126 — No silent data migration
The change MUST NOT attempt to auto-migrate existing `grafana` service rows into `prometheus` rows (URLs differ; true migration is impossible). Migration is operator-driven per the CHANGELOG note.
### Requirement: SC-127 — Non-blocking on the service-storage-harness change
This change MUST NOT depend on the `service-storage-harness` change. It is independently buildable, testable, and deployable. (The reverse dependency holds: the qBit speed widget depends on this change's chart capability.)
@@ -0,0 +1,168 @@
# Sync Report — `prometheus-direct-charting`
> Phase: **sync** · Change: `prometheus-direct-charting` · Repo: `/home/user/manage`
> Mode: file-backed (`artifactStore: openspec`). No source-code edits; only OpenSpec artifacts were
> written. Not committed (parent owns the commit). The change folder was **not** moved (that is
> `sdd-archive`'s job).
**Status: SYNCED.** A new canonical domain `openspec/specs/prometheus-charting/spec.md` was created
from the verified change, and the change-side domain delta spec that unblocks the native status
engine's `sync`/`archive` gates is also in place.
---
## 1. Executive summary
The `prometheus-direct-charting` change shipped a **complete but flat** `openspec/changes/prometheus-direct-charting/spec.md`
(27 requirements, SC-101 … SC-127) with **no** per-domain delta spec under
`openspec/changes/prometheus-direct-charting/specs/<domain>/`. `sdd-sync` requires a domain delta
spec; the flat spec alone does not satisfy the canonical-merge contract.
Verify already returned **PASS** (verdict in `verify-report.md`; all four gates green — backend
`pytest` 293 passed, `ruff` clean, frontend `npm run build` exit 0, `npm run lint` 0 errors).
Functional coverage was 26/27 fully PASS, with SC-125 PARTIAL on test coverage only (loading-state
test gap) — a non-blocking coverage finding, not a functional defect; the task framing treats it as
PARTIAL→PASS after coverage close. The verify report's single CRITICAL was an **archive** blocker
(19 unchecked task checkboxes + missing `apply-progress.md`); the `apply-progress.md` artifact now
exists and that condition does **not** block `sdd-sync` of the green code.
This sync **reconciles** the flat-spec-vs-domain-spec gap:
1. Authored the missing **change-side domain delta spec**
`openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md` — using a clean
`## ADDED Requirements` structure that preserves the exact requirement IDs (SC-101 … SC-127) and
text (including the patched SC-116 / SC-118 wording) from the verified flat `spec.md`. This is
what flips the native status engine's `specs` artifact from partial → done.
2. **Synced** the end-state into the **canonical store**
`openspec/specs/prometheus-charting/spec.md` — the actual sync target. Because the canonical
`prometheus-charting` domain did not previously exist, the native helper rule applies: *when the
canonical spec does not exist, the change spec becomes the new canonical spec.* The two files
therefore carry identical requirement bodies (delta under `## ADDED Requirements`; canonical
under `## Requirements`), verified byte-identical for the requirement region.
Domain name **`prometheus-charting`** was chosen (per the dispatch brief) because it covers the
full new model: direct Prometheus range-query charting, the gauge and mean modes, and the Grafana
removal that established the new direct-query model. It is distinct from the existing canonical
`web-ui` domain (MUI→shadcn migration — a different concern), which was **not touched**.
## 2. Structured status & actionContext findings
The native `gentle-pi.sdd-status` passed by the parent reports `changeName: null` with
`blockedReasons: ["Change selection is ambiguous: mobile-responsive-parity, prometheus-direct-charting,
prometheus-direct-charting, service-storage-harness, services-as-hub-ia."]` because the engine
auto-detected four active changes. This sync task was **explicitly assigned**
`prometheus-direct-charting`; the ambiguity is a parent-resolution artifact and does not block this
phase (`isNonAuthoritative: false`).
- `artifactStore: openspec`; change root `openspec/changes/prometheus-direct-charting/`.
- Artifacts present: `proposal.md`, `spec.md`, `design.md`, `tasks.md`, `verify-report.md`,
`apply-progress.md`.
- `verify: PASS` (verify-report verdict; gates green at `67ca0fc`).
- `actionContext`: `mode: repo-local`, `workspaceRoot: /home/user/manage`,
`allowedEditRoots: ["/home/user/manage"]`, `warnings: []`. All three files written are inside the
authoritative workspace / allowed edit roots. ✓
- `relationships.sameDomainActiveChanges: []`, `collisions: []` — **no active same-domain
collisions**, so no archive/sync ordering decision was required.
- The new `prometheus-charting` domain is distinct from the existing `web-ui` canonical domain;
`openspec/specs/web-ui/spec.md` was left untouched.
**Post-sync structural change:** `openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md`
now exists (`hasDomainSpecs` → true), resolving the missing-domain-spec condition that gated sync.
The flat `spec.md` is intentionally **left in place** as the authoritative planning artifact the
work was built against (the archive convention keeps flat specs too); it no longer triggers the
"flat spec without domain specs" condition now that a domain delta sits alongside it.
## 3. Domains synced & canonical files updated
| Domain | Change-side delta (source) | Canonical (sync target) | Action |
|---|---|---|---|
| `prometheus-charting` | `openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md` | `openspec/specs/prometheus-charting/spec.md` | **NEW domain**`## ADDED Requirements` copied into canonical as a new spec |
- **Canonical file created:** `openspec/specs/prometheus-charting/spec.md` (27 requirements).
- **Change-side delta created:** `openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md`
(27 requirements, all `## ADDED Requirements`).
## 4. Requirement delta (ADDED / MODIFIED / REMOVED)
- **ADDED (27)** — all to the new `prometheus-charting` domain (canonical did not exist pre-change).
IDs and text preserved verbatim from the verified flat `spec.md`. Grouped logically:
- *Direct Prometheus range query path* — SC-101, SC-102, SC-103, SC-104
- *Prometheus chart widget (rebrand + rebind)* — SC-105, SC-106, SC-107, SC-108
- *Prometheus gauge widget* — SC-109, SC-110, SC-111
- *Prometheus mean widget* — SC-112, SC-113, SC-114
- *Grafana removal* — SC-115, SC-116, SC-117, SC-118, SC-119, SC-120
- *Configuration documentation accuracy* — SC-121, SC-122
- *Test and build greenness* — SC-123, SC-124, SC-125
- *Migration guidance* — SC-126, SC-127
- **MODIFIED (0)** — none (new domain; no pre-existing canonical requirements to replace).
- **REMOVED (0)** — none.
- **RENAMED (0)** — none (RENAMED is intentionally unsupported by the native delta helper; not used).
## 5. Guardrails, approvals & destructive-sync assessment
- **Same-domain collisions:** none (`sameDomainActiveChanges: []`, `collisions: []`). The new
`prometheus-charting` domain does not overlap the existing `web-ui` canonical domain. No ordering
decision was needed.
- **Destructive sync:** **not applicable.** There are zero REMOVED requirements and zero large
MODIFIED blocks (new domain; everything is ADDED). No destructive-sync parent approval was
required for this sync beyond the explicit reconciliation instruction in the task.
- **Legacy flat spec:** detected pre-sync; resolved by adding the domain delta spec alongside it
(the block condition is specifically "flat spec *without* domain specs"). The flat spec was left
in place as a planning artifact.
- **`web-ui` canonical isolation:** the existing `openspec/specs/web-ui/spec.md` (the MUI→shadcn
rework) was **not modified** — verified untouched by `git status` (mtime `2026-06-17T19:11`,
not in the modified set). The two domains are independent.
## 6. Validation / checks performed (file-backed, read-only)
Run from `/home/user/manage` (no source edits, no test re-runs — those are owned by verify and were
already green at `67ca0fc`):
| Check | Command | Result |
|---|---|---|
| Canonical store populated | `ls openspec/specs/prometheus-charting/spec.md` | present ✓ |
| Change-side domain spec present | `ls openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md` | present ✓ |
| Requirement-ID parity (flat ↔ delta ↔ canonical) | `grep -oE 'SC-[0-9]+'` all three files, `sort -u` | **27 == 27 == 27**, identical IDs SC-101…SC-127 ✓ |
| Body-text parity (delta ↔ canonical) | `diff` of the `^### Requirement:` region of both files | **identical** ✓ |
| Patched SC-116 wording present | `grep "service-tabs/"` both delta + canonical | present in both ✓ |
| Patched SC-118 wording present | `grep "service-tabs/LinksTab.tsx"` both delta + canonical | present in both ✓ |
| Delta is pure ADDED | count `## ADDED/MODIFIED/REMOVED/RENAMED Requirements` | ADDED=1, MODIFIED=0, REMOVED=0, RENAMED=0 ✓ (no destructive sync) |
| `web-ui` canonical untouched | `git status --porcelain openspec/specs/web-ui/spec.md` | empty (not modified) ✓ |
| No edits outside openspec | `git status --porcelain` (filtered) | only `openspec/specs/prometheus-charting/`, `openspec/changes/prometheus-direct-charting/specs/`, and this report added; pre-existing dirty/untracked items unrelated to this sync unchanged ✓ |
| Markdown validity | write-time lint | all three files "Markdown clean" ✓ |
## 7. Carry-over items for the archive summary
These verify-phase findings are non-blocking for sync and should land in the archive summary:
1. **[INFO] SC-125 was PARTIAL in verify** (no explicit `isLoading:true` loading-state test in any of
the three new widget test files — coverage gap, not a functional defect). The task framing treats
this as PARTIAL→PASS after coverage close; if a loading-state case per widget has not been added,
`sdd-archive` may want to confirm or note it.
2. **[CRITICAL-process, archive-only] Unchecked task checkboxes.** At verify time, 19 implementation
/ verification task checkboxes (Slice 3 §3.13.14 and Integration §4.14.5) were unchecked and
`apply-progress.md` was missing. `apply-progress.md` now exists (created after the verify pass);
`sdd-archive` should re-scan the native status engine to confirm `tasks: done` / `applyProgress:
present` before moving the change to archive, and tick any remaining unchecked boxes if needed.
3. **[INFO] Stale generated `.pi-map.md`** files still reference Grafana / `ObservabilityPage.tsx`
(generated artifacts, not deliverable source; ignored by SC-115/116). Regenerate via
`project_map_patch` / `project_map_validate`.
4. **[INFO] Slice-2 review-budget variance** (~707 insertions vs ~310400 forecast) — additive
feature code + tests; boundary is the gauge+mean feature, not scope creep. Non-blocking; record
in the archive summary.
## 8. Next recommended phase
**`sdd-archive`** (clean). Confirm the native status re-scan reports `specs: done` / `sync: ready`
/ `archive: ready`, then move the change to
`openspec/changes/archive/YYYY-MM-DD-prometheus-direct-charting`, carrying over the items in §7 into
the archive summary. Do **not** commit or push — the parent owns the commit with explicit paths.
---
### Appendix — Files written by this sync (OpenSpec only; no source code)
- `openspec/changes/prometheus-direct-charting/specs/prometheus-charting/spec.md` — **change-side
domain delta (`## ADDED Requirements`), 27 requirements SC-101…SC-127.**
- `openspec/specs/prometheus-charting/spec.md` — **canonical spec (new domain), 27 requirements.**
- `openspec/changes/prometheus-direct-charting/sync-report.md` — this report.
@@ -0,0 +1,318 @@
# SDD Tasks: Prometheus Direct Charting (drop Grafana middleman)
**Change:** `prometheus-direct-charting`
**Phase:** tasks
**Date:** 2026-07-08
## Review Workload Forecast
| Field | Value |
|-------|-------|
| Estimated changed lines | ~8501,110 (sum of three implementation slices) |
| 400-line budget risk | Medium |
| Chained PRs recommended | Yes |
| Suggested split | PR 1: Prom range path + chart rebrand → PR 2: gauge + mean widgets → PR 3: Grafana removal + config rewrite |
| Delivery strategy | auto-chain |
| Chain strategy | stacked-to-main |
```text
Decision needed before apply: No
Chained PRs recommended: Yes
Chain strategy: stacked-to-main
400-line budget risk: Medium
```
> Each slice individually lands under the 400-line review budget. Slices are ordered S1 → S2 → S3; S1 and S2 are independently shippable, S3 must follow S1 (it removes the grafana chart binding S1 replaces). Per `openspec/config.yaml` rules, each slice leaves `npm run build` (tsc -b + vite build), `npm run lint`, and backend `pytest` green.
---
## Slice ordering rationale (critical)
**Slice 1 ADDS the prometheus chart capability without removing anything from grafana.** After S1:
- `prometheus` binding has `metric` + `chart`; `grafana` binding has only `link` (chart kind moved away).
- `GrafanaWidgetSource` still exists (serves the `link` kind; its `_fetch_chart` is now dead but harmless).
- `GrafanaLinkWidget` and the `grafana` service type are untouched.
- A `prometheus/chart` widget renders from a direct `/api/v1/query_range` call.
This ordering ensures the chart capability is proven against Prometheus before the grafana surface is removed in S3, so the two risks (new query path + grafana removal) never compound in a single slice.
---
## Slice 1: Prometheus range query path + shared helper + chart rebrand
**Goal:** Make a `prometheus/chart` widget render multi-series line charts from a direct `/api/v1/query_range` call, reusing the existing recharts renderer. Extract the series-normalization and step-derivation helpers into a testable module. Move the `chart` kind from `grafana` to `prometheus` in both registries. Do NOT yet remove the grafana service type, `GrafanaWidgetSource`, or `GrafanaLinkWidget`.
**Satisfies:** SC-101, SC-102, SC-103, SC-104, SC-105, SC-106, SC-107, SC-108.
- [x] **1.1 Create shared Prometheus range helpers module**
- Files: `backend/src/media_library_viewer_api/widgets/prometheus_range.py` (new)
- Lines: ~60
- Dependencies: none
- Details: Implement `WINDOW_PRESETS = {"1h": 3600, "6h": 21600, "24h": 86400, "7d": 604800}`, `step_for_window(window_seconds, target_points=200) -> int` returning `max(15, round(window_seconds / target_points))`, and `normalize_prometheus_matrix(result: list[dict]) -> list[dict]` that converts a Prometheus `/api/v1/query_range` `data.result` matrix into `{label, points:[{t:int, v:float|None}]}` series. Label rule: drop `__name__` from metric labels; join remaining as `k=v`; fall back to `"value"`; dedup collisions with `(n)` suffix. Null handling: `"NaN"`, `"+Inf"`, `"-Inf"`, `None``v: None`.
- [x] **1.2 Add backend unit tests for range helpers**
- Files: `backend/tests/test_prometheus_range.py` (new)
- Lines: ~70
- Dependencies: 1.1
- Details: Test `step_for_window` for all four presets asserts result yields 100300 points. Test `normalize_prometheus_matrix`: feed a two-entry sample matrix (one with `__name__`, one colliding label) → assert `{series}` shape, label dedup `(1)` suffix, null handling for `"NaN"` string.
- [x] **1.3 Add `_fetch_chart` + shared range-query plumbing to `PrometheusWidgetSource`**
- Files: `backend/src/media_library_viewer_api/widgets/sources.py` (modify)
- Lines: ~55
- Dependencies: 1.1
- Details: Import `WINDOW_PRESETS`, `step_for_window`, `normalize_prometheus_matrix` from `prometheus_range`. Add a private `_range_query(base_url, timeout, promql, window) -> dict` returning `{"matrix": result}` or `{"error": ...}` (shared by chart now and mean in S2). Add `_fetch_chart(self, base_url, timeout, config)` calling `_range_query` and returning `{"series": normalize_prometheus_matrix(matrix)}`. Dispatch `widget_kind == "chart"` in `.fetch()`. Existing `metric` path stays byte-for-byte unchanged. Errors (timeout, `RequestException`) → `{"error": str}`, never raise.
- [x] **1.4 Declare `chart` widget kind in Prometheus integration**
- Files: `backend/src/media_library_viewer_api/integrations/prometheus.py` (modify)
- Lines: ~15
- Dependencies: 1.3
- Details: Add `PrometheusChartWidgetConfig(WidgetConfigBase)` with `promql: str` and `window: str = "1h"`. Add a `widget_kind(...)` entry for `chart` (refresh 60s, default config `{"promql": "", "window": "1h"}`). Leave `metric` kind untouched.
- [x] **1.5 Rename `GrafanaChartWidget``PrometheusChartWidget`**
- Files: `frontend/src/widgets/GrafanaChartWidget.tsx``frontend/src/widgets/PrometheusChartWidget.tsx` (git mv)
- Lines: ~5 changed (rename export, fix empty-state copy)
- Dependencies: none
- Details: `git mv` to preserve history. Rename exported function `GrafanaChartWidget``PrometheusChartWidget`. The recharts body (`LineChart`, `Line`, `XAxis`, `YAxis`, `CartesianGrid`, `Tooltip`, `ResponsiveContainer`, `mergeSeries`, `CHART_COLORS`, `formatTime`) is preserved unchanged (SC-106). Fix empty-state copy: "Check your query and datasource_uid" → "Check your PromQL query and window."
- [x] **1.6 Rename chart widget test**
- Files: `frontend/src/widgets/__tests__/GrafanaChartWidget.test.tsx``frontend/src/widgets/__tests__/PrometheusChartWidget.test.tsx` (git mv)
- Lines: ~10 changed (import path, component name, error-message assertion)
- Dependencies: 1.5
- Details: `git mv`. Update import to `PrometheusChartWidget`. Update error-state assertion: the old Grafana-specific error string (`"Grafana api_key is required"`) → a Prom error string (e.g. `"promql is required"`). Keep loading + rendered-data test cases.
- [x] **1.7 Rebind `chart` from grafana to prometheus in frontend registry**
- Files: `frontend/src/integrations/registry.ts` (modify)
- Lines: ~30
- Dependencies: 1.5
- Details: Import `PrometheusChartWidget`. Add a `chart` entry to the `prometheus` binding's `widgets` array (kind `chart`, refresh 60s, configSchema with `promql` + `window`). Remove the `chart` entry from the `grafana` binding's `widgets` array (leave `link` intact). Do NOT delete the `grafana` binding itself.
- [x] **1.8 Update frontend widgets barrel export**
- Files: `frontend/src/widgets/index.ts` (modify)
- Lines: ~2
- Dependencies: 1.5
- Details: Rename the `GrafanaChartWidget` export to `PrometheusChartWidget`. Leave `GrafanaLinkWidget` export intact.
- [x] **1.9 Update registry tests for chart rebind**
- Files: `frontend/src/integrations/registry.test.ts` (modify)
- Lines: ~15
- Dependencies: 1.7
- Details: Assert `prometheus` binding has `metric` + `chart` kinds. Assert `grafana` binding has only `link` (no `chart`).
- [x] **1.10 Verify Slice 1 (build + lint + test)**
- Run: `cd backend && PYTHONPATH=src pytest tests/test_prometheus_range.py tests/test_widgets.py && cd ../frontend && npm run build && npm run lint`
- Verify: helpers tests pass; existing widget tests pass (grafana link adapter still wired); frontend typechecks and lints; `prometheus/chart` widget resolves to `PrometheusChartWidget`.
**Slice 1 total:** ~260330 changed lines.
---
## Slice 2: Gauge + mean widgets
**Goal:** Add `gauge` and `mean` widget kinds to the `prometheus` service. Gauge renders an instant scalar with configurable threshold bands. Mean renders a single value computed client-side over a range-query window. Both are scalar-only.
**Satisfies:** SC-109, SC-110, SC-111, SC-112, SC-113, SC-114.
- [x] **2.1 Extract shared `_instant_query` helper + add `_fetch_gauge` to `PrometheusWidgetSource`**
- Files: `backend/src/media_library_viewer_api/widgets/sources.py` (modify)
- Lines: ~40
- Dependencies: Slice 1 (1.3)
- Details: Extract the instant-query HTTP call from the existing `metric` path into a private `_instant_query(base_url, timeout, promql) -> dict` returning `{"result": [...]}` or `{"error": ...}`. Refactor the `metric` path to use it (behavior unchanged). Add `_fetch_gauge(self, base_url, timeout, config)` using `_instant_query`: assert `len(result) == 1` (scalar-only, SC-111); parse `float(result[0]["value"][1])`; return `{"value": float, "warn_at": config.get("warn_at"), "crit_at": config.get("crit_at"), "min": config.get("min"), "max": config.get("max"), "unit": config.get("unit")}`. Dispatch `widget_kind == "gauge"` in `.fetch()`.
- [x] **2.2 Add `_fetch_mean` to `PrometheusWidgetSource`**
- Files: `backend/src/media_library_viewer_api/widgets/sources.py` (modify)
- Lines: ~30
- Dependencies: 2.1, Slice 1 (1.3 for `_range_query`)
- Details: Add `_fetch_mean(self, base_url, timeout, config)` using the shared `_range_query` from S1. Assert `len(result) == 1` (scalar-only, SC-114). Collect non-null numeric values from the single series; compute arithmetic mean; return `{"value": mean, "unit": config.get("unit")}`. If no numeric samples → `{"error": "..."}`. Dispatch `widget_kind == "mean"` in `.fetch()`.
- [x] **2.3 Declare `gauge` + `mean` widget kinds in Prometheus integration**
- Files: `backend/src/media_library_viewer_api/integrations/prometheus.py` (modify)
- Lines: ~25
- Dependencies: 2.1, 2.2
- Details: Add `PrometheusGaugeWidgetConfig` (`promql: str`, `warn_at: float|None`, `crit_at: float|None`, `min: float|None`, `max: float|None`, `unit: str|None`) and `PrometheusMeanWidgetConfig` (`promql: str`, `window: str = "1h"`, `unit: str|None`). Add `widget_kind(...)` entries: `gauge` (refresh 30s), `mean` (refresh 60s).
- [x] **2.4 Add backend tests for gauge + mean adapters**
- Files: `backend/tests/test_widgets.py` (modify) or `backend/tests/test_prometheus_range.py` (modify)
- Lines: ~60
- Dependencies: 2.1, 2.2
- Details: Mock `requests.get` for gauge: instant query returning 1 series → assert `{value, ...}` shape; returning 2 series → assert `{"error": ...}` (SC-111). Mock for mean: range query returning 1 series with known values `[1.0, 2.0, 3.0]` → assert mean `2.0`; returning 2 series → assert `{"error": ...}` (SC-114). Test timeout/RequestException → `{"error": ...}`.
- [x] **2.5 Create `PrometheusGaugeWidget` component**
- Files: `frontend/src/widgets/PrometheusGaugeWidget.tsx` (new)
- Lines: ~90
- Dependencies: Slice 1 (1.5 for widget pattern)
- Details: Render via recharts `RadialBarChart` (no new dep; SC-110). Threshold bands: three stacked `RadialBar` track cells (green `0→warn`, amber `warn→crit`, red `crit→max`) + a value cell. When `warn_at`/`crit_at` absent → single neutral-color track. `min`/`max` default to `0`/`max(value, 1)`. Reuse `SectionCard` + `Alert`/`Skeleton` for loading/error states. Consume `data?.data?.value`, `warn_at`, etc. off `useWidgetData`.
- [x] **2.6 Create `PrometheusMeanWidget` component**
- Files: `frontend/src/widgets/PrometheusMeanWidget.tsx` (new)
- Lines: ~50
- Dependencies: Slice 1
- Details: Single-value display reusing the `MetricCard` pattern (big number + optional `unit` suffix + subtext "mean over last {window}"). Loading/error/empty via `Skeleton`/`Alert`. No charting library — it's a number (SC-112).
- [x] **2.7 Create gauge + mean frontend tests**
- Files: `frontend/src/widgets/__tests__/PrometheusGaugeWidget.test.tsx` (new), `frontend/src/widgets/__tests__/PrometheusMeanWidget.test.tsx` (new)
- Lines: ~60
- Dependencies: 2.5, 2.6
- Details: Each covers loading, error, and rendered-data case (SC-125). Gauge test: render with bands (`warn_at`/`crit_at` set) and without (single color). Mean test: render with `value` + `unit`.
- [x] **2.8 Add gauge + mean bindings to frontend registry**
- Files: `frontend/src/integrations/registry.ts` (modify)
- Lines: ~35
- Dependencies: 2.5, 2.6
- Details: Import `PrometheusGaugeWidget` + `PrometheusMeanWidget`. Add `gauge` (refresh 30s, configSchema with `promql`, `warn_at`, `crit_at`, `min`, `max`, `unit`) and `mean` (refresh 60s, configSchema with `promql`, `window`, `unit`) entries to the `prometheus` binding's `widgets` array alongside `metric` and `chart`.
- [x] **2.9 Update widgets barrel + registry tests**
- Files: `frontend/src/widgets/index.ts` (modify), `frontend/src/integrations/registry.test.ts` (modify)
- Lines: ~10
- Dependencies: 2.5, 2.6, 2.8
- Details: Export `PrometheusGaugeWidget` + `PrometheusMeanWidget`. Assert `prometheus` binding has `metric`, `chart`, `gauge`, `mean` (four kinds).
- [x] **2.10 Verify Slice 2 (build + lint + test)**
- Run: `cd backend && PYTHONPATH=src pytest tests/test_prometheus_range.py tests/test_widgets.py && cd ../frontend && npm run build && npm run lint`
- Verify: gauge/mean adapter tests pass; frontend typechecks and lints; all four prometheus widget kinds resolve.
**Slice 2 total:** ~310400 changed lines.
---
## Slice 3: Grafana removal + config rewrite + changelog
**Goal:** Remove the entire Grafana surface (service type, link widget, chart source, status checks, nav entries, UI tabs). Rewrite `config.yaml` to match reality. Add CHANGELOG migration note. Leave the app grep-clean of grafana service references.
**Satisfies:** SC-115, SC-116, SC-117, SC-118, SC-119, SC-120, SC-121, SC-122, SC-126.
> **Spec-text drift note:** SC-118 literally names `ObservabilityPage.tsx`, which was refactored into `service-tabs/`. The *intent* (no Grafana UI surface) is verified by the removals below. `Dashboard.test.tsx` contains "Grafana" as a user-authored shortcut label (unrelated to the grafana service) — SC-116 grep should not flag it; it is left intact.
- [x] **3.1 Delete backend Grafana integration module**
- Files: `backend/src/media_library_viewer_api/integrations/grafana.py` (delete)
- Lines: ~20 (deletion)
- Dependencies: Slice 1 (chart kind already moved to prometheus)
- Details: Delete the file. It contains `GrafanaConfig` + `GrafanaLinkWidgetConfig`.
- [x] **3.2 Remove Grafana from backend integration registry**
- Files: `backend/src/media_library_viewer_api/integrations/registry.py` (modify)
- Lines: ~3
- Dependencies: 3.1
- Details: Drop `from ...grafana import DEFINITION as GRAFANA` and the `GRAFANA.service_type: GRAFANA` entry from `SERVICE_DEFINITIONS`.
- [x] **3.3 Remove `GrafanaWidgetSource` + adapter registration**
- Files: `backend/src/media_library_viewer_api/widgets/sources.py` (modify)
- Lines: ~100 (deletion of `GrafanaWidgetSource` class + `_fetch_chart`)
- Dependencies: Slice 1 (normalization logic already extracted to `prometheus_range.py`)
- Details: Delete the `GrafanaWidgetSource` class entirely (including `_fetch_chart` — its logic was extracted to `normalize_prometheus_matrix` in S1). Remove `"grafana": GrafanaWidgetSource()` from `SERVICE_ADAPTERS`.
- [x] **3.4 Remove `get_grafana_status` endpoint**
- Files: `backend/src/media_library_viewer_api/routers/monitoring.py` (modify)
- Lines: ~25
- Dependencies: none
- Details: Delete the `@router.get("/grafana-status")` endpoint and its helper. Leave `get_prometheus_status` / `get_alertmanager_status` intact.
- [x] **3.5 Remove Grafana backend tests**
- Files: `backend/tests/test_widgets.py` (modify), `backend/tests/test_api.py` (modify), `backend/tests/test_services.py` (modify)
- Lines: ~40
- Dependencies: 3.3
- Details: Delete grafana adapter tests (`test_grafana_adapter_*`), grafana service fixtures, and the `TestGrafanaStatus` test class. SC-120.
- [x] **3.6 Delete frontend `GrafanaLinkWidget` + barrel export**
- Files: `frontend/src/widgets/GrafanaLinkWidget.tsx` (delete), `frontend/src/widgets/index.ts` (modify)
- Lines: ~35
- Dependencies: none
- Details: Delete the file. Remove the `GrafanaLinkWidget` export from `widgets/index.ts`.
- [x] **3.7 Remove `grafana` binding from frontend registry**
- Files: `frontend/src/integrations/registry.ts` (modify)
- Lines: ~30
- Dependencies: 3.6
- Details: Delete the entire `grafana` key from `SERVICE_REGISTRY`. Drop the `GrafanaLinkWidget` import. SC-117.
- [x] **3.8 Remove Grafana nav entry**
- Files: `frontend/src/integrations/navEntries.ts` (modify)
- Lines: ~5
- Dependencies: none
- Details: Drop the `grafana` entry from `SERVICE_TYPE_NAV_ENTRIES` and any now-unused icon import (e.g. `Link2`).
- [x] **3.9 Remove Grafana status hook + API client function + type**
- Files: `frontend/src/hooks/useObservability.ts` (modify), `frontend/src/api/client.ts` (modify), `frontend/src/types/index.ts` (modify)
- Lines: ~15
- Dependencies: 3.4
- Details: Drop `useGrafanaStatus` + its `fetchGrafanaStatus` import from `useObservability.ts`. Drop `fetchGrafanaStatus` from `client.ts`. Drop `GrafanaStatus` interface from `types/index.ts`.
- [x] **3.10 Delete Grafana service tab + remove from tab index**
- Files: `frontend/src/pages/service-tabs/LinksTab.tsx` (delete), `frontend/src/pages/service-tabs/__tests__/LinksTab.test.tsx` (delete), `frontend/src/pages/service-tabs/index.ts` (modify)
- Lines: ~80
- Dependencies: none
- Details: Delete `LinksTab.tsx` (grafana-specific per its docstring) and its test. Remove the `LinksTab` import and `case "grafana":` from `service-tabs/index.ts`.
- [x] **3.11 Remove Grafana from Dashboard + ServicesPage**
- Files: `frontend/src/pages/Dashboard.tsx` (modify), `frontend/src/pages/ServicesPage.tsx` (modify)
- Lines: ~5
- Dependencies: none
- Details: Drop `"grafana"` from `OBSERVABILITY_TYPES` set in `Dashboard.tsx`. Update `ServicesPage.tsx` empty-state copy: "Add a Grafana, Prometheus, …" → "Add a Prometheus, …".
- [x] **3.12 Rewrite `openspec/config.yaml` thin-dashboard rule**
- Files: `openspec/config.yaml` (modify)
- Lines: ~8
- Dependencies: none
- Details: Remove "Do NOT re-implement charting in-app" and "No recharts/d3 is in use" claims. Replace with accurate wording: in-app charting via `recharts` is the sanctioned approach for Prometheus-backed series; Grafana is no longer referenced. SC-121.
- [x] **3.13 Add CHANGELOG migration note**
- Files: `CHANGELOG.md` (modify)
- Lines: ~8
- Dependencies: none
- Details: Add entry under an appropriate version heading: instruct operators to delete existing Grafana service instances and recreate them as Prometheus services; note that `grafana/chart` widgets must be recreated as `prometheus/chart` widgets. SC-122, SC-126.
- [x] **3.14 Verify Slice 3 (grep-clean + build + lint + test)**
- Run: `grep -ri grafana backend/src --include='*.py'` → expect zero matches (SC-115)
- Run: `grep -ri grafana frontend/src` → expect zero matches except `Dashboard.test.tsx` shortcut fixture (SC-116)
- Run: `cd backend && PYTHONPATH=src pytest && cd ../frontend && npm run build && npm run lint`
- Verify: no grafana service references remain; all tests pass; frontend builds and lints.
**Slice 3 total:** ~250350 changed lines (mostly deletions).
---
## Integration and acceptance verification
- [x] **4.1 Full backend test run**
- Run: `cd backend && PYTHONPATH=src pytest`
- Verify: all tests pass; no grafana test references remain; prometheus range/gauge/mean tests pass.
- [x] **4.2 Full frontend build + lint**
- Run: `cd frontend && npm run build && npm run lint`
- Verify: TypeScript compiles; no lint failures; no grafana imports unresolved.
- [x] **4.3 Grep-clean verification**
- Run: `grep -ri grafana backend/src --include='*.py'` → zero matches
- Run: `grep -ri grafana frontend/src` → zero matches excluding `Dashboard.test.tsx` shortcut fixture
- Verify: SC-115, SC-116 satisfied.
- [x] **4.4 config.yaml accuracy check**
- Verify: `openspec/config.yaml` does not contain "Do NOT re-implement charting" or "No recharts/d3"; reflects recharts as sanctioned renderer (SC-121).
- [x] **4.5 CHANGELOG check**
- Verify: `CHANGELOG.md` documents the grafana→prometheus migration (SC-122).
---
## Total estimate
| Slice | Changed lines | Satisfies |
|-------|---------------|-----------|
| Slice 1: Prom range path + chart rebrand + helper | ~260330 | SC-101…108 |
| Slice 2: Gauge + mean widgets | ~310400 | SC-109…114 |
| Slice 3: Grafana removal + config + changelog | ~250350 | SC-115…122, 126 |
| Integration verification | ~0 | SC-123…125, 127 |
| **Total** | **~8201,080** | **SC-101…127** |
Each slice is under the 400-line review budget. Use three chained PRs (S1 → S2 → S3), each independently buildable and green.
---
## Risk flags for the apply phase
1. **`git mv` for chart widget rename** — use `git mv` (not delete+create) to preserve file history (design §9).
2. **recharts `RadialBarChart` gauge** — try recharts first; if rendering proves fiddly, the proposal sanctions a ~50-line SVG fallback (no new dep). Decision at apply time.
3. **`Dashboard.test.tsx` "Grafana" literal** — this is a user-authored shortcut label in a test fixture, not a grafana service reference. SC-116 grep should not flag it. Flagged for reviewer awareness.
4. **SC-118 textual drift**`ObservabilityPage.tsx` no longer exists (refactored to `service-tabs/`). The *intent* is satisfied by removing `get_grafana_status` + `useGrafanaStatus` + `LinksTab`. Verify against intent, not literal filename.
5. **`_range_query` / `_instant_query` refactoring timing** — `_range_query` is created in S1 (1.3) for `_fetch_chart`; `_instant_query` is extracted in S2 (2.1) when `_fetch_gauge` needs it. Both shared helpers must be in place before S3 (which deletes `GrafanaWidgetSource` and its private chart logic).
6. **Stale project map** — the pi-map references `ObservabilityPage.tsx` and 7 service types (actual: 8, including `authentik` + `backups`). Trust source, not the map. Run `project_map_patch` after source edits and `project_map_validate` before final handoff.
@@ -0,0 +1,336 @@
# Verify Report — prometheus-direct-charting
> Phase: **verify** · Change: `prometheus-direct-charting` · Repo: `/home/user/manage`
> FRESH-CONTEXT adversarial read-only verification of the change against
> `proposal.md`, `spec.md`, `design.md`, and `tasks.md`. **No source edits.**
> This verify report is the only file written.
**Head commit verified:** `67ca0fc` (`feat(prometheus-direct-charting): slice 3 — remove grafana + config rewrite + changelog`).
Three implementation slices are committed:
- `5dad982` slice 1 — prom range query + chart rebrand
- `65bae95` slice 2 — gauge + mean widgets
- `67ca0fc` slice 3 — grafana removal + config rewrite + changelog
> NOTE: the dispatch brief cited slice hashes `58be6e0` / `ba94317` (an earlier
> amend state). The actual landed commits are `65bae95` / `67ca0fc`. Content of
> all three slices matches the spec/design/tasks; this is informational, not a defect.
---
## 0. Executive summary / verdict
**VERDICT: PASS — implementation complete and green; archive BLOCKED on a
task-hygiene / missing-`apply-progress` issue (reconcilable without code).**
Every functional requirement **SC-101 … SC-127** was checked against source and
**passes**. Grafana is fully excised from the live code paths (service type,
adapters, widgets, hook, API client, type, status endpoint, nav entry, service
tab). Prometheus is the direct chart source: `chart` (multi-series recharts line
chart via `/api/v1/query_range`), `gauge` (instant scalar + threshold bands via
recharts `RadialBarChart`), and `mean` (client-side average) are wired in both
registries with consistent schemas. The shared `normalize_prometheus_matrix` /
`step_for_window` helpers are extracted and unit-tested. All four gates are
green: backend `pytest` (**293 passed**), `ruff` (**clean**), frontend
`npm run build` (**exit 0**), `npm run lint` (**0 errors**).
Findings:
- **[CRITICAL — archive blocker, NOT a code defect]** **19 unchecked
implementation/verification task checkboxes** remain in `tasks.md` (all of
Slice 3 §3.13.14 and Integration §4.14.5), and **`apply-progress.md` does not
exist** to reconcile them. The underlying work *is* done and verified complete
against source; the blocker is that the task tracker was never updated and no
apply-progress artifact was produced. Reconciliation = tick the boxes + write
`apply-progress.md` (no code change). See §4.
- **[WARNING]** SC-125 is **PARTIAL**: none of the three new widget tests
(`PrometheusChartWidget` / `PrometheusGaugeWidget` / `PrometheusMeanWidget`)
exercises an explicit **loading** state (`isLoading:true` → Skeleton). Error
and rendered-data cases exist for all three. The loading branch exists in each
component; the gap is test coverage, not functionality. See §5.
- **[INFO]** Stale generated `.pi-map.md` files still reference Grafana /
`ObservabilityPage.tsx`; these are not deliverable source and are ignored by
> SC-115/116, but should be regenerated (`project_map_patch`/`validate`). The
> `config.yaml` context block also still names `ObservabilityPage.tsx` (refactored
away into `service-tabs/`) — not an SC-121 criterion, minor doc staleness.
- **[INFO]** Working tree is not pristine: an uncommitted cosmetic reformat of
`frontend/src/pages/service-tabs/MediaTab.tsx` (unrelated to this change) plus
untracked `.pi-tmp/*` and `openspec/changes/service-storage-harness/` (the
proposal for the *separate* change SC-127 requires independence from).
---
## 1. Structured status & actionContext findings
The native `gentle-pi.sdd-status` reports `changeName: null` /
`blockedReasons: ["Change selection is ambiguous: …"]` because the engine
auto-detected four active changes. This verify task was **explicitly assigned**
`prometheus-direct-charting`; the ambiguity is a parent-resolution artifact and
does not block this phase.
- `artifactStore: openspec`; change root
`openspec/changes/prometheus-direct-charting/`.
- Artifacts present: `proposal.md`, `spec.md`, `design.md`, `tasks.md`.
- **`apply-progress.md`: MISSING** (confirmed: directory contains only the four
planning docs + this report). This is the root cause of the §4 archive blocker.
- `actionContext`: `mode: repo-local`, `workspaceRoot: /home/user/manage`,
`allowedEditRoots: ["/home/user/manage"]`, `warnings: []`. Implementation
ownership and all target files are provably inside the authoritative workspace. ✓
- The `service-storage-harness` change exists only as an untracked proposal
folder; SC-127 (independence) holds — this change builds/tests green without it.
## 2. Gate results (actual output, run at `67ca0fc`)
| Gate | Command | Result | Evidence |
|------|---------|--------|----------|
| Backend tests | `cd backend && PYTHONPATH=src python3 -m pytest -q` | **PASS** | **293 passed, 2 warnings** in 36.26s. Includes `test_prometheus_range.py` (9) + chart/gauge/mean adapter tests. |
| Backend lint | `cd backend && PYTHONPATH=src python3 -m ruff check src tests` | **PASS** | `All checks passed!` |
| Frontend build | `cd frontend && npm run build` (`tsc -b` + `vite build`) | **PASS** exit 0 | `✓ built in 1.08s`; 2543 modules transformed. Non-fatal `>500 kB` chunk-size warning (pre-existing). |
| Frontend lint | `cd frontend && npm run lint` (`eslint .`) | **PASS** exit 0 | `0 errors, 1 warning`. The warning is `react-hooks/exhaustive-deps` in `WidgetConfigDialog.tsx:370`**pre-existing, untouched by this change** (no slice modified that file). |
### Grep gates (SC-115 / SC-116 / SC-117 / SC-120)
```
grep -rin grafana backend/src --include='*.py'
→ 3 matches, ALL in prometheus_range.py docstrings/comments describing the
extraction/removal (explicitly allowed by SC-115). No live grafana code.
grep -rin grafana frontend/src
→ matches ONLY in: stale generated .pi-map.md / .pi-map.index.md files
(ignored — generated artifacts), and Dashboard.test.tsx lines 108/115
(the whitelisted user-authored shortcut *label* "Grafana"). No live
grafana code, no grafana widget/component/hook.
grep -rin grafana backend/tests --include='*.py'
→ ZERO matches (SC-120). (.pyc cache + .pi-map.md are stale; .py sources clean.)
```
---
## 3. Spec coverage (SC-101 … SC-127)
| SC | Requirement | Verdict | Evidence |
|----|-------------|---------|----------|
| SC-101 | range query → `{series}` shape | **PASS** | `sources.py:_fetch_chart``_range_query` hits `/api/v1/query_range` with `query/start/end/step`; returns `{"series": normalize_prometheus_matrix(matrix)}` = `{label, points:[{t:int, v:float\|null}]}`. Test `test_prometheus_chart_adapter_runs_range_query` asserts endpoint + params + shape. |
| SC-102 | shared, Prometheus-native label normalization | **PASS** | Single helper `normalize_prometheus_matrix` in `prometheus_range.py`; drops `__name__`/`__*`, joins `k=v` (sorted), falls back to `"value"`, dedups with `(n)`. No duplication. `test_prometheus_range.py` covers all rules. |
| SC-103 | range-query errors degrade gracefully | **PASS** | `_range_query`/`_instant_query` catch `asyncio.TimeoutError` + `requests.RequestException``{"error": ...}`; `.fetch()` wraps the whole body in `try/except → {"error"}`. Never raises. Test `test_prometheus_chart_adapter_degrades_on_http_error`. |
| SC-104 | step derived from window (100300 pts) | **PASS** | `step_for_window = max(15, round(window/200))`. Parametrized test asserts 100300 points for `1h/6h/24h/7d` + 15s floor + custom target. |
| SC-105 | `chart` rebind grafana→prometheus (both registries) | **PASS** | Backend `prometheus.py` declares `chart`; frontend `registry.ts` `prometheus` binding has `chart`. Grafana is gone entirely (so trivially offers no `chart`). |
| SC-106 | chart renderer reused unchanged | **PASS** | `PrometheusChartWidget.tsx` preserves `LineChart/Line/XAxis/YAxis/CartesianGrid/Tooltip/ResponsiveContainer/mergeSeries/CHART_COLORS/formatTime`; rename-only + empty-state copy fix. |
| SC-107 | chart multi-series | **PASS** | `series.map((s,i) => <Line dataKey={s.label} .../>)` renders every series; no single-series restriction. |
| SC-108 | chart window is a preset | **PASS** | chart config = `{promql, window}`; `window ∈ {1h,6h,24h,7d}` resolved via `WINDOW_PRESETS` server-side; no raw from/to/step. |
| SC-109 | gauge = instant scalar | **PASS** | `_fetch_gauge` → instant `/api/v1/query` → parses `result[0]["value"][1]``{"value":...}`. Bound to `prometheus` in both registries. |
| SC-110 | gauge threshold bands | **PASS** | `PrometheusGaugeWidget` (recharts `RadialBarChart`) renders green/amber/red track cells when `warn_at`+`crit_at` set, single neutral track otherwise; `min/max` default to `0`/`max(value,1)`. Config accepts `warn_at/crit_at/min/max/unit`. |
| SC-111 | gauge scalar-only | **PASS** | `len(result) != 1 → {"error": "Gauge requires a single-series query; refine your PromQL"}`. Test `test_prometheus_gauge_adapter_rejects_multi_series`. |
| SC-112 | mean = client-side mean over window | **PASS** | `_fetch_mean``_range_query` → averages non-null samples of the single series → `{"value": mean, "unit?"}`. Test asserts mean of `[1,2,3]=2.0`, NaN-skip `[2,NaN,4]=3.0`. |
| SC-113 | mean = plain PromQL + preset | **PASS** | mean config = `{promql, window, unit?}`; user supplies plain PromQL (no `avg_over_time`); window is a preset. |
| SC-114 | mean scalar-only | **PASS** | `len(result) != 1 → {"error": ...}`. Test `test_prometheus_mean_adapter_rejects_multi_series`. |
| SC-115 | no grafana in backend src | **PASS** | grep → only `prometheus_range.py` docstrings (allowed). `integrations/grafana.py` deleted; no `GrafanaWidgetSource`. |
| SC-116 | no grafana in frontend src | **PASS** | grep → only `Dashboard.test.tsx` whitelisted shortcut fixture + stale `.pi-map.md` (ignored). No live code. |
| SC-117 | grafana gone from registries | **PASS** | No `grafana` in backend `SERVICE_DEFINITIONS`/`SERVICE_ADAPTERS` or frontend `SERVICE_REGISTRY`/`BUILTIN_WIDGETS`; `integrations/grafana.py` deleted. `registry.test.ts` asserts 5 service types (no grafana). |
| SC-118 | grafana status + UI removed | **PASS (intent)** | `get_grafana_status` removed (`monitoring.py`); `useGrafanaStatus`/`fetchGrafanaStatus`/`GrafanaStatus` type removed; `LinksTab.tsx`+test deleted; `service-tabs/index.ts` grafana case removed; `navEntries.ts` grafana entry removed; `Dashboard.tsx` `OBSERVABILITY_TYPES` = `{alertmanager, prometheus}`; `ServicesPage.tsx` copy updated. `ObservabilityPage.tsx` was already refactored into `service-tabs/` (spec drift acknowledged in design §0 / spec SC-118 note). |
| SC-119 | orphaned grafana widget degrades | **PASS** | `WidgetInstanceCard`: `resolveWidget``undefined` for a grafana-bound widget (no binding) → renders `"Unknown widget: <kind> (service-bound)"` `Alert`. No crash. |
| SC-120 | grafana tests removed | **PASS** | No grafana in backend test `.py` or frontend widget tests. Only stale `.pyc` cache + `.pi-map.md`. |
| SC-121 | config.yaml accurate | **PASS** | Stale "Do NOT re-implement charting in-app" / "No recharts/d3" claims gone; OBSERVABILITY MODEL states "Manage renders Prometheus-backed metrics directly via recharts … Grafana is no longer integrated." No grafana-as-chart-path. |
| SC-122 | CHANGELOG migration note | **PASS** | `[Unreleased]` has "Direct Prometheus charting" + "**BREAKING** — Grafana service type removed" with migration: delete grafana instances → recreate as Prometheus; `grafana/chart` widgets → `prometheus/chart`. |
| SC-123 | backend tests pass | **PASS** | `pytest` → 293 passed; covers range→`{series}`, gauge scalar-only, mean aggregation, shared label helper. |
| SC-124 | frontend build + lint | **PASS** | `npm run build` exit 0; `npm run lint` 0 errors (1 pre-existing warning). |
| SC-125 | new widget kinds have tests | **PARTIAL** | All three test files exist with **error + rendered** cases (gauge has with/without bands; mean has with/without unit). **No explicit loading-state case** (`isLoading:true`) in any of the three — see §5. |
| SC-126 | no silent data migration | **PASS** | No grafana→prometheus row transformation code anywhere; only docstring references to manual migration. `settings_store.py` migration is unrelated (`jellyseerr→jellyfin`). |
| SC-127 | independent of service-storage-harness | **PASS** | No reference to it in source (only a `prometheus_range.py` docstring noting future reuse); builds/tests green without it; that change exists only as a proposal folder. |
**Functional spec coverage: 26/27 fully PASS, 1 PARTIAL (SC-125, test-coverage only).**
---
## 4. Task completion status — ⚠ archive blocker (reconcilable)
`tasks.md` checkbox state:
- **Slice 1 (§1.11.10): all 10 `[x]`**
- **Slice 2 (§2.12.10): all 10 `[x]`**
- **Slice 3 (§3.13.14): all 14 `[ ]` — UNCHECKED**
- **Integration (§4.14.5): all 5 `[ ]` — UNCHECKED**
Total: 20 checked, **19 unchecked**.
**This is a CRITICAL completeness issue for the archive gate per the verify
contract.** However — and this is the important reconciliation — **the Slice 3
and Integration work is verifiably DONE against source**:
| Unchecked task | Actual state (verified) |
|----------------|-------------------------|
| 3.1 delete `integrations/grafana.py` | **deleted** (`ls` → ENOENT; slice-3 diff `73 ------`) |
| 3.2 drop grafana from backend registry | **done** (`registry.py` has no grafana import/entry) |
| 3.3 remove `GrafanaWidgetSource` + adapter | **done** (`sources.py` has no class; `SERVICE_ADAPTERS` no grafana; diff `113 ---------`) |
| 3.4 remove `get_grafana_status` | **done** (`monitoring.py` grep clean; diff `-25`) |
| 3.5 remove grafana backend tests | **done** (`.py` test grep clean) |
| 3.6 delete `GrafanaLinkWidget` + barrel | **done** (file gone; `index.ts` export dropped) |
| 3.7 remove grafana frontend binding | **done** (`registry.ts` no grafana key; registry.test asserts) |
| 3.8 remove grafana nav entry | **done** (`navEntries.ts` grep clean) |
| 3.9 remove hook + api client + type | **done** (`useObservability.ts`/`client.ts`/`types/index.ts` grep clean) |
| 3.10 delete `LinksTab` + tab-index case | **done** (file + test deleted; `index.ts` grep clean) |
| 3.11 Dashboard + ServicesPage | **done** (`OBSERVABILITY_TYPES` = `{alertmanager,prometheus}`; ServicesPage copy updated) |
| 3.12 rewrite `config.yaml` | **done** (stale claims gone) |
| 3.13 CHANGELOG note | **done** (migration entry present) |
| 3.14 verify slice 3 (grep + build + lint + test) | **done** (all green — §2) |
| 4.1 full backend pytest | **done** (293 passed) |
| 4.2 frontend build + lint | **done** (exit 0 / 0 errors) |
| 4.3 grep-clean | **done** (SC-115/116 satisfied) |
| 4.4 config.yaml check | **done** (SC-121 satisfied) |
| 4.5 CHANGELOG check | **done** (SC-122 satisfied) |
The unchecked boxes are **stale** (work performed, tracker not updated), and
**no `apply-progress.md` exists** to serve as the stale-checkbox reconciliation
record the contract permits. Resolution is a **documentation-only** step:
tick §3.13.14 and §4.14.5, and author `apply-progress.md` describing the three
landed slices. **No code change is required.**
> Per the verify contract, an unchecked implementation-task line is an archive
> blocker until reconciled. Because the implementation is verified complete, this
> blocks **archive** but does **not** block `sdd-sync` of the green code.
---
## 5. TDD compliance & assertion-quality assessment
Strict-TDD was **not** declared active for this change in `config.yaml` /
parent prompt / (absent) `apply-progress.md`, so the formal TDD-cycle-evidence
check is **not applicable**. Assertion quality was still audited adversarially.
**Backend assertions — GENUINELY BEHAVIORAL (good).** Spot-checked:
- `test_prometheus_range.py`: asserts exact label strings (`instance=h:9100 mode=idle`), dedup suffix (`job=x (1)`), null sentinels → `None`, malformed-timestamp drop, point-count band per preset, 15s floor. No tautologies.
- `test_widgets.py` adapters: assert the **called URL** (`endswith("/api/v1/query_range")` / `/api/v1/query"`), the **params** (`{start,end,step} ⊆ params`, `query=="up"`), the **return shape** (`result["series"][0]["label"]`, `result["value"]==0.75`), multi-series → `{"error" ... "single-series"}`, and mean arithmetic (`2.0`, NaN-skip `3.0`). These verify real behavior, not smoke.
**Frontend assertions — adequate, one coverage gap.**
- Gauge test asserts the **rendered value text** (`0.75 %`, `42 req/s`) and threshold-label presence/absence — meaningful.
- Mean test asserts the **rendered value** (`23.5 %`, `1500`) — meaningful.
- Chart rendered-case asserts only the **SectionCard title** (`CPU Usage`), not that recharts drew the series SVG — a weak/title-only assertion (acceptable as a "rendered data case" since the component mounts with series data, but it does not prove the lines rendered).
**[WARNING] SC-125 loading-state gap (see §3).** None of the three widget tests
sets `isLoading:true`. The `mockData` helper hard-codes `isLoading:false`, and no
case asserts the `<Skeleton>` loading branch. Each component's loading branch
exists and is structurally identical to sibling widgets, so this is a
**test-coverage gap, not a functional defect**. Recommend adding one
`isLoading:true` case per widget to fully satisfy SC-125's enumerated
"loading state" requirement. Non-blocking.
No ghost loops, no type-only assertions, no implementation-detail CSS assertions
found. No test mocks grafana anywhere (only the whitelisted `Dashboard.test.tsx`
shortcut label).
## 6. Review-workload / PR-boundary findings
Per-slice changed lines (numstat, excluding `tasks.md` doc churn):
| Commit | Slice | Source Δ | Over 400? | Verdict |
|--------|-------|----------|-----------|---------|
| `5dad982` | 1 range+rebrand+helper | ~333 ins / ~56 del | under (337 net w/ test) | **OK** |
| `65bae95` | 2 gauge+mean | ~707 ins / ~32 del | **over** (largely new widgets+tests: gauge 137, mean 59, tests 124, backend tests 197) | **OK** — additive feature code+tests; forecast rated "Medium"; boundary is the feature, not a missed split |
| `67ca0fc` | 3 grafana removal+config+changelog | ~123 ins / **~926 del** | under (net negative) | **OK** — mostly deletions |
The `tasks.md` Review Workload Forecast (`stacked-to-main`, 3 slices, ~8501,110
total) was followed: S1→S2→S3, each independently green. Slice 2's +707 insertions
exceed the 400-line *added* budget but are dominated by two new components + their
tests + new adapter tests (no `size:exception` flag was recorded, and the forecast
itself rated slice 2 "~310400 changed lines" which under-counts the test volume).
This is a **minor forecast-vs-actual variance on an additive slice**, not scope
creep — the boundary is exactly the gauge+mean feature, no unrelated files touched.
Recommend recording the slice-2 actual in the archive summary. **Non-blocking.**
Scope was honored: no backend API/type-contract widening, no `service-storage-harness`
coupling, `metric` path preserved (`PrometheusMetricWidget` reads `data?.data?.result`
as the PromQL data object; `_instant_query` returns `{"result": payload.get("data",{})}`
— identical shape).
## 7. Adversarial checks
- **Dead imports after grafana removal?** None. Backend `ruff` (catches unused
imports) is clean; `sources.py` imports `WINDOW_PRESETS/normalize_prometheus_matrix/
step_for_window` — all used. Frontend `tsc`/`eslint` clean (unresolved imports
would fail the build).
- **`PrometheusWidgetSource.fetch` dispatch cross-contamination?** None. Clean
`widget_kind` dispatch: `chart``{series}`, `gauge``{value,...}`, `mean``{value}`,
default→`{result}` (metric). Distinct shapes; shared `_range_query`/`_instant_query`
only do HTTP + error mapping.
- **Config-schema consistency (backend models vs frontend registry)?** Consistent.
gauge: backend `{promql, warn_at, crit_at, min, max, unit}` ↔ frontend configSchema
`{promql(req), warn_at, crit_at, min, max, unit}`. chart: `{promql, window}`
`{promql(req), window}`. mean: `{promql, window, unit}``{promql(req), window, unit}`.
- **Lingering grafana mocks?** None in `.py`/`.tsx` source. Only stale `.pyc`
bytecode cache + `.pi-map.md`.
## 8. Residual risks / non-blocking findings
1. **[CRITICAL-process] 19 unchecked tasks + missing `apply-progress.md`** (§4) — archive blocker; reconciliation is doc-only.
2. **[WARNING] SC-125 loading-state tests missing** (§5) — coverage gap, not a defect.
3. **[INFO] Stale generated `.pi-map.md`** files reference Grafana / `ObservabilityPage.tsx` / `TestGrafanaStatus` / `GrafanaLinkWidget`. Not deliverable source; ignored by SC-115/116. Regenerate via `project_map_patch`/`project_map_validate` (the project-map protocol flags these `dirty`).
4. **[INFO] `config.yaml` context** still names `frontend/src/components/ObservabilityPage.tsx` (refactored away into `service-tabs/`). Not an SC-121 criterion (which targets the charting/grafana claims, which are fixed); minor doc staleness.
5. **[INFO] Uncommitted `MediaTab.tsx`** cosmetic reformat (Prettier line-wrap of a ternary) — unrelated to this change, predates/orthogonal. Dirty working tree; no files are staged.
6. **[INFO] Chunk-size build warning** (~1.1 MB JS) — non-fatal, pre-existing, orthogonal.
7. **No browser/visual smoke** performed (out of scope); the recharts `RadialBarChart` gauge and `LineChart` rendering are only structurally tested.
## 9. Exact blockers
- **BLOCKER (archive only, doc-reconcilable):** 19 unchecked implementation/verification
tasks (§3.13.14, §4.14.5) and absent `apply-progress.md`. Implementation is
verified complete; resolution = tick boxes + write `apply-progress.md`.
No code-level blockers. All functional requirements SC-101…SC-127 pass (SC-125
PARTIAL on test coverage only). All four gates green. **Code is ready for
`sdd-sync`; archive requires the checkbox/apply-progress reconciliation.**
## 10. Recommended next phase
**`sdd-sync`** (code PASS). Concurrently/after: author `apply-progress.md`
documenting the three landed slices, and tick §3.13.14 + §4.14.5 in
`tasks.md` to clear the archive blocker. Optionally add three `isLoading:true`
widget tests to move SC-125 PARTIAL→PASS, and regenerate the stale `.pi-map.md`.
---
### Appendix A — Verification commands run (at `67ca0fc`)
```
cd backend && PYTHONPATH=src python3 -m pytest -q → 293 passed (2 warnings)
cd backend && PYTHONPATH=src python3 -m ruff check src tests → All checks passed!
cd frontend && npm run build → exit 0 (✓ built; >500kB warning pre-existing)
cd frontend && npm run lint → exit 0 (0 errors, 1 pre-existing warning)
grep -rin grafana backend/src --include='*.py' → 3 docstring hits in prometheus_range.py (allowed)
grep -rin grafana frontend/src → Dashboard.test.tsx fixture + stale .pi-map.md only
grep -rin grafana backend/tests --include='*.py' → ZERO
ls backend/src/media_library_viewer_api/integrations/grafana.py → ENOENT (deleted)
grep -nE '^\s*- \[ \]' tasks.md → 19 unchecked (§3.13.14, §4.14.5)
```
### Appendix B — Files substantively changed
**Backend**
- `widgets/prometheus_range.py` (new) — `WINDOW_PRESETS`, `step_for_window`, `normalize_prometheus_matrix`.
- `widgets/sources.py` — dropped `GrafanaWidgetSource`; `PrometheusWidgetSource` gains `_fetch_chart/_fetch_gauge/_fetch_mean` + shared `_range_query/_instant_query`.
- `integrations/prometheus.py``metric`+`chart`+`gauge`+`mean` widget kinds + config models.
- `integrations/registry.py` — grafana import/entry removed.
- `integrations/grafana.py`**deleted**.
- `routers/monitoring.py``get_grafana_status` removed.
- `tests/test_prometheus_range.py` (new), `tests/test_widgets.py` (chart/gauge/mean adapter tests; grafana tests removed), `tests/test_api.py` + `tests/test_services.py` (grafana fixtures/tests removed).
**Frontend**
- `widgets/PrometheusChartWidget.tsx` (renamed from GrafanaChartWidget), `PrometheusGaugeWidget.tsx` (new), `PrometheusMeanWidget.tsx` (new) + their tests.
- `widgets/GrafanaLinkWidget.tsx`**deleted**; `widgets/index.ts` barrel updated.
- `integrations/registry.ts` — grafana binding removed; chart/gauge/mean added to prometheus; `registry.test.ts` updated.
- `integrations/navEntries.ts` — grafana entry removed.
- `hooks/useObservability.ts`, `api/client.ts`, `types/index.ts` — grafana hook/client/type removed.
- `pages/service-tabs/LinksTab.tsx` (+test) — **deleted**; `service-tabs/index.ts` grafana case removed.
- `pages/Dashboard.tsx` (`OBSERVABILITY_TYPES`), `pages/ServicesPage.tsx` (copy) — grafana removed.
**Docs**
- `openspec/config.yaml` — stale thin-dashboard/no-recharts claims repealed.
- `CHANGELOG.md` — direct-Prometheus-charting + grafana-removal migration note.
@@ -0,0 +1,55 @@
# Apply Progress: Service Storage Harness
**Change:** `service-storage-harness`
**Phase:** apply-progress
**Date:** 2026-07-09
**Status:** complete — all 35 tasks done, all gates green, verified (see `verify-report.md`)
## Slices delivered
Four slices, each its own commit, each leaving `pytest` / `npm run build` / `npm run lint` / `ruff` green.
### Slice 1 — Harness + qBit store + client + integration (commit `e7bd0af`, amended)
- `services/service_data.py``ServiceDataHarness`: lifecycle-only (concern registration via dataclass, idempotent `run_migrations` catching "duplicate column name" per-statement, `cascade_delete(service_id)` iterating owned tables). No generic data ops (SS-101..104).
- `services/qbittorrent_store.py``QbittorrentSampleStore`: `qbittorrent_speed_samples(service_id, ts, dl_speed, up_speed)` + index in dedicated `qbittorrent.db`; `append/window/prune`, MAX_SAMPLES=120; registered as a harness concern (SS-105..107).
- `clients/qbittorrent.py``QbittorrentClient`: cookie login via `/api/v2/auth/login`, 403 re-login+retry, `maindata()` via `/api/v2/sync/maindata` (SS-108..110).
- `integrations/qbittorrent.py` + registry entry — config (base_url, timeout_seconds) + secret (username, password) schema; 3 widget kinds declared.
- Initialized in `main.py` lifespan.
- Tests: `test_service_data.py` (harness lifecycle), `test_qbittorrent_store.py`, `test_qbittorrent_client.py`.
### Slice 2 — qBit widgets + LineSeriesChart extract (commit `8b0e7ea`, amended)
- `QbittorrentWidgetSource` in `widgets/sources.py` — 3 branches: `totals` (item count from maindata), `active` (filter state ∈ {downloading, uploading}), `speed` (append sample + return `{series}` from `.window()`, ts×1000 for JS ms). Errors → `{error}`. Registered in `SERVICE_ADAPTERS` (SS-111..115).
- `frontend/src/components/LineSeriesChart.tsx` — shared recharts renderer extracted from `PrometheusChartWidget` (~40 lines, props `{series, height?}`); `PrometheusChartWidget` becomes a thin wrapper. **Extraction non-regressive: 4 PrometheusChartWidget tests stay green** (SS-118).
- `QbittorrentTotalsWidget` (count), `QbittorrentActiveTorrentsWidget` (list), `QbittorrentSpeedWidget` (uses LineSeriesChart). Registry binding + barrel + tests (SS-116..117).
### Slice 3 — MediaIndex migration (commit `c87f398`) — LOAD-BEARING
- Idempotent harness migration: `ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT ''`. `media_index.db` file location UNCHANGED. Existing rows backfill to `service_id=''` via DEFAULT (SS-119, SS-120, SS-124).
- **Scoped `replace_items`**`DELETE FROM media_items WHERE service_id = ?` replaces the prior global `DELETE FROM media_items`. **FIXES the latent global-clear bug** where rebuilding for one Jellyfin wiped another's rows. `service_id` stamped into inserted rows. Regression test `test_replace_scoped_by_service_id_preserves_other_services` proves svc-A survives svc-B's rebuild (SS-121).
- `query(service_id="")` shows all rows (backward-compat); `query(service_id="X")` scopes. **Existing MediaIndex + API tests pass unchanged** (62 passed) (SS-122).
- Worker threads the real `service_id` (already plumbed via `--service-id`) into `replace_items` so new rows are stamped (SS-123).
### Slice 4 — Cascade-delete wiring (commit `75c949a`)
- `settings_store.delete_service` calls `ServiceDataHarness.cascade_delete(service_id)` after existing cleanup, best-effort try/except (failure logs, doesn't crash the delete) (SS-125).
- Integration test proves end-to-end cascade across BOTH concerns (qBit samples + media items) with multi-instance preservation (SS-126).
## Deviations from tasks.md
- **Slice 2 over the 400-line review budget** (verify-report flagged +644 source lines). The slice is additive (3 new widgets + extraction + tests), no scope creep, but the per-slice budget from `openspec/config.yaml` was exceeded. Retrospectively this could have been split (extraction in one slice, qBit widgets in another). No code defect; recorded here as a process note for future slicing. The verify agent flagged it WARNING, not blocking.
## Final gate results
| Gate | Result |
|---|---|
| `backend && PYTHONPATH=src python3 -m pytest -q` | **322 passed**, 2 warnings (pre-existing pythonjsonlogger DeprecationWarning) |
| `backend && PYTHONPATH=src python3 -m ruff check src tests` | **All checks passed** |
| `frontend && npm run build` | **exit 0** (pre-existing chunk-size warning) |
| `frontend && npm run lint` | **0 errors**, 1 pre-existing warning (`WidgetConfigDialog.tsx`, untouched) |
| `frontend && npx vitest run PrometheusChartWidget.test.tsx` | **4 passed** (extraction non-regression confirmed) |
## Verification
See `verify-report.md` — adversarial fresh-context review: **28/28 PASS**. No blocking code findings. Archive blocker is doc-only (this file + the ticked tasks.md clear it).
@@ -0,0 +1,220 @@
# Archive Report — `service-storage-harness`
> Phase: **archive** · Change: `service-storage-harness` · Repo: `/home/user/manage`
> Mode: file-backed (`artifactStore: openspec`). No source-code edits; only OpenSpec artifacts
> were touched. **Not committed** — the parent/orchestrator owns the archive commit. No push, no `gh`.
**Status: ARCHIVED.** All eight lifecycle phases are complete (proposal → spec → design → tasks →
apply → verify → sync → **archive**). Every archive precondition is verified PASS (see §2). The
canonical `openspec/specs/service-storage/spec.md` (created by `sdd-sync`) remains in place as the
durable end-state spec and is **not** moved (archive never moves canonical specs). The change folder
was moved to `openspec/changes/archive/2026-07-09-service-storage-harness/` via `git mv` to preserve
history (the renames were then unstaged so the parent commits from a clean index with explicit paths).
---
## 0. Archive disposition
- **Disposition: `archived`.** The folder move was performed inline as instructed (unlike a
`documented-pending-manual` outcome): the parent explicitly requested the `git mv` and owns the
commit, so the move is executed here and left as working-tree changes for the parent's
explicit-path commit.
- **Archive convention:** OpenSpec SDD archive contract for `openspec` mode — completed file-backed
sync → write the in-folder archive report → move the change folder to
`openspec/changes/archive/YYYY-MM-DD-{change}/`. No standalone manifest/index exists under
`openspec/` (only `config.yaml`, `changes/`, `specs/`), so the folder move **is** the archive
mechanism. No `rules.archive` override exists in `openspec/config.yaml`.
- **Target archived path:** `openspec/changes/archive/2026-07-09-service-storage-harness/`
- **Archive date:** `2026-07-09` (ISO).
- **Canonical spec left in place (not moved):** `openspec/specs/service-storage/spec.md`
28 requirements (SS-101 … SS-128). Verified present and byte-identical before/after the move
(sha256 `6629e307…` unchanged). The other canonical domains (`web-ui`, `prometheus-charting`)
were also left untouched.
- **Audit-trail integrity:** the change folder was moved as a whole, including the legacy flat
`spec.md` and the per-domain delta `specs/service-storage/spec.md`, which travel with the
record. Nothing was silently deleted or rewritten. The flat `spec.md` is retained as the
authoritative planning artifact the work was built against.
## 1. Native `sdd-status` read & discrepancy statement
The native `gentle-pi.sdd-status` engine supplied by the parent reports **non-actionable state for
this archive** because it was resolved without a change context: `changeName: null`,
`artifacts: all missing`, `applyState: blocked`, `dependencies.archive: blocked`,
`blockedReasons: ["Change selection is ambiguous: mobile-responsive-parity, service-storage-harness,
services-as-hub-ia."]`, `isNonAuthoritative: false`. This is a **parent-resolution artifact**: the
engine auto-detected three active changes and could not pick one. The ambiguity does **not** reflect
the state of `service-storage-harness`, which this archive task was **explicitly assigned**.
**Discrepancy with the parent's authoritative confirmed state — RESOLVED in favor of the parent.**
The parent physically verified (and this executor re-confirmed directly against the filesystem in
§2) that all 35 tasks are ticked, `apply-progress.md` exists and records the four landed slices, the
verify report is clearly passing (28/28 PASS, all four gates green), and `sync-report.md` records a
completed sync. Per the archive contract's non-authoritative-store carve-out guidance and the
parent's explicit instruction ("DISREGARD; PROCEED"), the stale `archive: blocked` / "ambiguous"
labels are **disregarded** and the archive **proceeds**.
Direct filesystem re-validation (§2) is the source of truth for this report.
## 2. Archive preconditions (validated directly against the filesystem)
| Precondition | Evidence | Result |
|---|---|---|
| Verify report present | `verify-report.md` | ✓ verdict **PASS** |
| Verify clearly passing — no unresolved `FAIL`/`BLOCKED`/`CRITICAL` | verify-report: **28/28 PASS** (SS-101…SS-128); the sole CRITICAL was an archive-only checkbox/apply-progress gap (now reconciled) | ✓ |
| Sync report present & successful | `sync-report.md`**Status: SYNCED** | ✓ |
| Canonical spec exists (sync target) | `openspec/specs/service-storage/spec.md` (28 requirements) | ✓ |
| Change-side domain delta exists | `specs/service-storage/spec.md` | ✓ |
| Delta op-class = pure `## ADDED` (non-destructive) | ADDED=1, MODIFIED=0, REMOVED=0, RENAMED=0 (new domain) | ✓ |
| Requirement-ID parity (flat ↔ delta ↔ canonical) | 28 == 28 == 28, identical IDs SS-101…SS-128 | ✓ |
| proposal / design / tasks artifacts present | all populated | ✓ |
| **Final Task Completion Gate — zero unchecked `- [ ]`** | `grep -nE '^\s*- \[ \]' tasks.md`**NONE**; `grep -cE '^\s*- \[x\]'`**35** | ✓ |
| `apply-progress.md` present & records the work | present; status "complete — all 35 tasks done", 4 slices + final gates documented | ✓ |
| No active same-domain (`service-storage`) collision | only this change carries a `service-storage` delta; `web-ui`/`prometheus-charting` untouched | ✓ |
| Canonical untouched by the move | sha256 of `openspec/specs/service-storage/spec.md` identical before/after the `git mv` | ✓ |
**Stale-checkbox reconciliation note.** At verify time, 30 implementation/verification checkboxes
(Slices §1.11.9, §2.12.9, §3.13.9, §4.14.3 and Integration §5.15.5) were unchecked and
`apply-progress.md` did not exist. That condition was reconciled **before** archive: the boxes are
now all ticked (re-confirmed directly: `grep -nE '^\s*- \[ \]' tasks.md` → none; 35 `[x]`) and
`apply-progress.md` was authored documenting the four landed slices. `apply-progress.md` plus the
verify report prove every previously-unchecked task complete. No archive-time mechanical checkbox
repair was needed — the gate now passes on the persisted `tasks.md` as-is. No partial-archive
approval applies.
## 3. Artifacts read (archive preflight)
- `openspec/changes/service-storage-harness/proposal.md`
- `openspec/changes/service-storage-harness/spec.md` (flat, authoritative planning artifact — 28 requirements)
- `openspec/changes/service-storage-harness/specs/service-storage/spec.md` (change-side domain delta)
- `openspec/changes/service-storage-harness/design.md`
- `openspec/changes/service-storage-harness/tasks.md`
- `openspec/changes/service-storage-harness/apply-progress.md`
- `openspec/changes/service-storage-harness/verify-report.md`
- `openspec/changes/service-storage-harness/sync-report.md`
- `openspec/specs/service-storage/spec.md` (canonical, sync target — verified present and byte-identical after the move)
- `openspec/config.yaml` (rules: proposal/tasks; no `rules.archive` override)
> The legacy flat `spec.md` is **not** the only spec artifact: a per-domain delta
> (`specs/service-storage/spec.md`) and a canonical spec both exist, so the "legacy flat spec as the
> *only* artifact" archive-block condition does not apply. The flat spec travels with the archived
> folder as part of the audit trail.
## 4. Domains synced & requirement delta
| Domain | Change-side delta | Canonical | Action |
|---|---|---|---|
| `service-storage` | `specs/service-storage/spec.md` | `openspec/specs/service-storage/spec.md` | **NEW domain** — pure ADDED (28 requirements) |
- **ADDED (28)** — all to the new `service-storage` domain (canonical did not exist pre-change). IDs
and text preserved verbatim from the verified flat `spec.md`. Grouped logically:
- *ServiceDataHarness (lifecycle layer)* — SS-101, SS-102, SS-103, SS-104
- *QbittorrentSampleStore* — SS-105, SS-106, SS-107
- *QbittorrentClient* — SS-108, SS-109, SS-110
- *qBittorrent widget source adapter* — SS-111, SS-112, SS-113, SS-114, SS-115
- *qBittorrent frontend widgets* — SS-116, SS-117, SS-118
- *MediaIndex migration onto harness* — SS-119, SS-120, SS-121, SS-122, SS-123, SS-124
- *Cascade-delete wiring* — SS-125, SS-126
- *Test and build greenness* — SS-127, SS-128
- **MODIFIED (0)** · **REMOVED (0)** · **RENAMED (0)** — new domain; nothing destructive.
> No destructive-merge guard or parent approval was triggered (zero REMOVED / zero MODIFIED). The
> new `service-storage` domain is distinct from the existing `web-ui` and `prometheus-charting`
> canonical domains, neither of which was touched.
## 5. Final lifecycle status (all 8 phases done)
| Phase | Status | Evidence |
|---|---|---|
| Proposal | ✅ done | `proposal.md` |
| Spec | ✅ done | flat `spec.md` (28) + domain delta `specs/service-storage/spec.md` (28 ADDED) |
| Design | ✅ done | `design.md` |
| Tasks | ✅ done | `tasks.md`**35/35** checked, zero `- [ ]` |
| Apply | ✅ done | 4 slices delivered (`e7bd0af`, `1fb12b8`, `c87f398`, `75c949a`) |
| Verify | ✅ PASS | `verify-report.md` — 28/28 PASS; gates green |
| Sync | ✅ done | `sync-report.md` — SYNCED; canonical `service-storage` domain created |
| Archive | ✅ done | this report + folder move performed |
## 6. Gate results (per verify-report; head `c9404f0`)
| Gate | Command | Result |
|---|---|---|
| Backend tests | `cd backend && PYTHONPATH=src python3 -m pytest -q` | **PASS** — 322 passed (2 pre-existing warnings) |
| Backend lint | `cd backend && PYTHONPATH=src python3 -m ruff check src tests` | **PASS** — All checks passed |
| Frontend build | `cd frontend && npm run build` | **PASS** — exit 0 (pre-existing chunk-size warning) |
| Frontend lint | `cd frontend && npm run lint` | **PASS** — 0 errors (1 pre-existing warning) |
| Extraction non-regression | `npx vitest run …/PrometheusChartWidget.test.tsx` | **PASS** — 4 passed (SS-118) |
| New FE widget/chart tests | `npx vitest run …/widgets/__tests__ …/LineSeriesChart.test.tsx` | **PASS** — 8 files, 31 tests passed |
## 7. Carry-over follow-ups (non-blocking; recorded per verify/apply-progress)
1. **[WARNING] Slice-2 review-budget variance** — slice 2 (`1fb12b8`) lands **~644 non-test source
lines** (qBit widget adapter + `LineSeriesChart` extraction + 3 FE widgets), above the 400-line
per-slice budget (`openspec/config.yaml`) and above the slice-2 "~350400" forecast. No
`size:exception` was recorded. This is a forecast-vs-actual variance on an **additive** slice —
the boundary is exactly the qBit-widget feature (no unrelated files, no scope creep). Non-blocking;
recorded for the record (could have split extraction into its own slice). The verify agent rated
it WARNING, not blocking.
2. **[INFO] Slice-2 commit-hash drift** — the brief/apply-progress cited `8b0e7ea` (an earlier amend
state); the actual landed commit is `1fb12b8`. Content matches spec/design/tasks. Informational.
3. **[INFO] Stale generated `.pi-map.md`** — generated project-map artifacts (`.pi-map.md` /
`.pi-map.index.md`) predate the new modules (`service_data.py`, `qbittorrent_store.py`,
`clients/qbittorrent.py`, `Qbittorrent*.tsx`, `LineSeriesChart.tsx`). These are **generated
artifacts, not deliverable source**, and are out of scope for this change — regenerate via
`project_map_patch` / `project_map_validate` in a separate housekeeping pass; the project-map
protocol already flags these `dirty`.
4. **[INFO] `LineSeriesChart.test.tsx` smoke-only** — asserts the component mounts
(`container.firstChild` non-null) but does not assert the recharts `<Line>` SVG series rendered.
Acceptable as a crash-guard; non-blocking coverage note.
5. **[INFO] qBit store prune edge case** — `QbittorrentSampleStore.append` prunes via
`ts NOT IN (SELECT ts … LIMIT 120)`; in the degenerate case of two samples sharing an identical
`ts` the keep-set could exceed `MAX_SAMPLES=120`. At a 5 s poll the probability is effectively nil
and the per-service cap holds in all realistic operation. Non-blocking.
6. **[INFO] Orthogonal dirty working-tree items** — uncommitted cosmetic reformat of
`frontend/src/pages/service-tabs/MediaTab.tsx` and untracked `.pi-tmp/*` (predating/orthogonal to
this change) were **not touched** by this archive.
## 8. Residual risks & destructive-merge statement
- **Destructive sync / merge:** **not applicable.** Zero REMOVED and zero MODIFIED requirements (new
`service-storage` domain; pure ADDED). No destructive-merge guard or parent approval was triggered.
- **Backend / data-contract impact:** none beyond the planned change. The `ServiceDataHarness` is
lifecycle-only (provisioning, idempotent migrations, `service_id` cascade-delete) with **no**
generic value table or CRUD. The load-bearing MediaIndex migration (SS-119..SS-124) is correct:
`replace_items` is scoped `WHERE service_id = ?` (fixing the latent global-clear bug), with a real
regression test; the `media_index.db` file location is unchanged; existing callers are
backward-compatible via the `service_id=""` default. Archive touched only OpenSpec docs + the folder
move.
- **No critical verification issues** remain (CRITICAL issues are non-overridable; the one verify
CRITICAL was the reconcilable checkbox/apply-progress gap, now resolved).
- **No browser/visual smoke** was performed (out of scope); the qBit speed recharts line and the
Prometheus/gauge/mean charts are only structurally tested.
- **Memory observation IDs:** none — `artifactStore: openspec`; traceability lives in the filesystem
archive + canonical spec.
## 9. Move performed
```
git mv openspec/changes/service-storage-harness openspec/changes/archive/2026-07-09-service-storage-harness
```
- **All 9 artifacts confirmed present at the archived path:** `proposal.md`, `spec.md` (flat),
`specs/service-storage/spec.md` (delta), `design.md`, `tasks.md`, `apply-progress.md`,
`verify-report.md`, `sync-report.md`, `archive-report.md` (this file).
- Of the 9, **8 were git-tracked** (renamed by `git mv`, preserving history via git rename
detection) and **1** (`archive-report.md`) is newly written and traveled with the directory move.
- **Canonical `openspec/specs/service-storage/spec.md` remains in place** — verified byte-identical
(sha256 `6629e307…`) before and after the move. `openspec/specs/web-ui/spec.md` and
`openspec/specs/prometheus-charting/spec.md` also untouched.
- **Not committed / not pushed** — the parent owns the commit with explicit paths.
---
### Appendix — Files written/moved by this archive (OpenSpec only; no source code)
- **Written:** `openspec/changes/service-storage-harness/archive-report.md` (this file) — at the
active path before the move; travels with the move into the archive.
- **Moved (via `git mv`):** the entire
`openspec/changes/service-storage-harness/` directory →
`openspec/changes/archive/2026-07-09-service-storage-harness/`.
- **Left in place (durable canonical):** `openspec/specs/service-storage/spec.md`.
- **Not committed / not pushed** — the parent owns the commit with explicit paths.
@@ -0,0 +1,863 @@
# SDD Design: Service Storage Harness (with qBittorrent widgets + MediaIndex migration)
**Change:** `service-storage-harness`
**Phase:** design
**Date:** 2026-07-09
> Grounded in `proposal.md` (Q1Q5 resolved) and the already-archived
> `prometheus-direct-charting` change. No source changes in this phase.
## 0. Source findings (read before anything else)
The proposal was written against a stale project map. Reading actual source
surfaced three findings that shape the design. Trust source, not the map.
### 0.1 The media worker ALREADY threads `service_id` — but it is NOT persisted in rows
`workers/media_index_worker.py` already accepts `--service-id` via argparse
(`run_build(final_index_path, staging_index_path, service_id="")`), and
`_resolve_jellyfin(service_id)` resolves the Jellyfin client + user_id from the
settings store. The media router's `_start_worker(index, service_id)` and
`post_build_index` already pass `jellyfin_service_id` through to the worker.
**But:** `service_id` is used ONLY to pick the Jellyfin connection. It is never
stored in the `media_items` table (which has no `service_id` column). The
migration must close this gap: `replace_items` must scope its DELETE+INSERT by
`service_id`, and `query` must filter by it. The plumbing to get `service_id`
into the worker already exists — only the storage layer is missing.
### 0.2 `media_items` is cleared globally on every build
`replace_items` does `DELETE FROM media_items` (no WHERE clause). This means a
build for Jellyfin instance A wipes instance B's rows. After migration this
becomes `DELETE FROM media_items WHERE service_id = ?`.
### 0.3 The `delete_service` cascade already exists — the harness hooks into it
`SettingsStore.delete_service` already cascade-deletes `dashboard_widgets WHERE
service_id = ?` (with a PRAGMA-guarded column check). The harness cascade-delete
hooks into the same place: after the service row is deleted, iterate registered
concerns and delete that `service_id` from each owned table.
### 0.4 DB path is a module constant, not in config.py
`DEFAULT_INDEX_PATH = Path(".cache/media_library_viewer/media_index.sqlite")` in
`media_index_impl.py`. The file stays at this path (per the locked topology
decision). The harness must not move it.
---
## 1. Architecture overview
```
┌──────────────────────────────────────────────────────────────────┐
│ ServiceDataHarness │
│ (services/service_data.py — LIFECYCLE ONLY) │
│ │
│ • register_concern(db_filename, migrations[], tables[], │
│ service_id_column="service_id") │
│ • run_migrations() — on startup, per concern DB │
│ • cascade_delete(service_id) — iterate concerns, DELETE rows │
│ • connect(db_filename) → sqlite3.Connection (per-concern) │
└──────────────┬───────────────────────────┬───────────────────────┘
│ │
┌──────────▼──────────┐ ┌─────────▼──────────────┐
│ QbittorrentStore │ │ MediaIndex │
│ (services/ │ │ (services/ │
│ qbittorrent_store) │ │ media_index_impl) │
│ │ │ │
│ qbittorrent.db │ │ media_index.sqlite │
│ └ qbittorrent_ │ │ └ media_items │
│ speed_samples │ │ (+ service_id col) │
│ (service_id, ts, │ │ └ index_metadata │
│ dl_speed, │ │ │
│ up_speed) │ │ bespoke: replace_items │
│ │ │ (scoped), query │
│ bespoke: append, │ │ (scoped), status │
│ window, prune │ │ │
└──────────────────────┘ └────────────────────────┘
┌──────────▼──────────┐
│ QbittorrentClient │
│ (clients/ │
│ qbittorrent) │
│ │
│ login → cookie │
│ sync/maindata │
│ (totals + active + │
│ speeds) │
└──────────────────────┘
```
**Key constraints carried from the proposal:**
- Harness is lifecycle-only: migrations, service_id scoping, cascade-delete. No generic value table, no generic CRUD (D2).
- Per-concern DB files: `media_index.sqlite` stays put; new `qbittorrent.db` (D4).
- MediaIndex migration is sequenced after harness + qBit are proven (D3).
- qBit speed chart reuses the `PrometheusChartWidget` recharts renderer fed from an InService data path returning `{series}` (Q1).
- Totals = count of listed items, NOT transfer bytes (Q2).
- Active = state downloading|uploading (Q3). N instances (Q4). Username/password → cookie (Q5).
---
## 2. Backend design
### 2.1 `ServiceDataHarness` (`services/service_data.py`)
A lifecycle-only registry of storage concerns. Each concern declares its own DB
filename, ordered migrations, owned tables, and the column used for service
scoping.
```python
@dataclass(frozen=True)
class StorageConcern:
"""A per-integration storage namespace registered with the harness."""
concern_key: str # e.g. "qbittorrent", "media_index"
db_filename: str # e.g. "qbittorrent.db", "media_index.sqlite"
migrations: list[str] # ordered CREATE/ALTER statements (idempotent)
tables: list[str] # tables owned by this concern (for cascade)
service_id_column: str = "service_id"
class ServiceDataHarness:
"""Lifecycle-only registry of per-concern storage.
Owns: DB provisioning, per-concern migrations, service_id cascade-delete.
Does NOT own: data operations (each store keeps bespoke append/window/query/etc.).
"""
def __init__(self, base_dir: Path) -> None:
self._base_dir = Path(base_dir)
self._concerns: dict[str, StorageConcern] = {}
def register(self, concern: StorageConcern) -> None:
"""Register a storage concern. Called at module import / startup."""
self._concerns[concern.concern_key] = concern
def db_path(self, concern_key: str) -> Path:
"""Return the absolute path to a concern's DB file."""
concern = self._concerns[concern_key]
return self._base_dir / concern.db_filename
def connect(self, concern_key: str) -> sqlite3.Connection:
"""Open a WAL-mode connection to a concern's DB."""
path = self.db_path(concern_key)
path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path, timeout=30)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA busy_timeout=30000")
return conn
def run_migrations(self) -> None:
"""Run pending migrations for every registered concern."""
for concern in self._concerns.values():
path = self.db_path(concern.concern_key)
path.parent.mkdir(parents=True, exist_ok=True)
with sqlite3.connect(path, timeout=30) as conn:
conn.execute("PRAGMA journal_mode=WAL")
conn.executescript(";".join(concern.migrations))
def cascade_delete(self, service_id: str) -> None:
"""Delete all rows for a service_id across every concern's tables.
Called from SettingsStore.delete_service after the service row is removed.
"""
for concern in self._concerns.values():
col = concern.service_id_column
with sqlite3.connect(self.db_path(concern.concern_key), timeout=30) as conn:
for table in concern.tables:
cols = {row[1] for row in conn.execute(f"PRAGMA table_info({table})").fetchall()}
if col in cols:
conn.execute(f"DELETE FROM {table} WHERE {col} = ?", (service_id,))
```
**Module-level singleton + registration:**
```python
_HARNESS: ServiceDataHarness | None = None
def get_service_data_harness() -> ServiceDataHarness:
global _HARNESS
if _HARNESS is None:
base_dir = Path(os.environ.get("BACKEND_CACHE_DIR", ".cache/media_library_viewer"))
_HARNESS = ServiceDataHarness(base_dir)
# Register built-in concerns (each store module calls register on import)
_HARNESS.register(QBITTORRENT_CONCERN)
_HARNESS.register(MEDIA_INDEX_CONCERN)
_HARNESS.run_migrations()
return _HARNESS
```
**Startup hook:** called from `main.py` lifespan alongside `get_settings_store().ensure_defaults()`. The harness is lazy-initialized on first access (like `SettingsStore`), so tests can override the base dir via env.
### 2.2 `QbittorrentSampleStore` (`services/qbittorrent_store.py`)
Speed-sample storage for the qBit speed widget. Registered as a concern with the
harness.
```python
QBITTORRENT_CONCERN = StorageConcern(
concern_key="qbittorrent",
db_filename="qbittorrent.db",
migrations=[
"""
CREATE TABLE IF NOT EXISTS qbittorrent_speed_samples (
service_id TEXT NOT NULL,
ts INTEGER NOT NULL,
dl_speed INTEGER NOT NULL DEFAULT 0,
up_speed INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS idx_qbit_samples_service_ts
ON qbittorrent_speed_samples(service_id, ts);
"""
],
tables=["qbittorrent_speed_samples"],
)
MAX_SAMPLES = 120 # ~2 min at 1s poll, ~4 min at 2s poll
class QbittorrentSampleStore:
"""Bespoke speed-sample store for qBittorrent widgets."""
def __init__(self, harness: ServiceDataHarness | None = None) -> None:
self._harness = harness or get_service_data_harness()
def append(self, service_id: str, ts: int, dl_speed: int, up_speed: int) -> None:
"""Append a sample and prune old entries beyond MAX_SAMPLES."""
with self._harness.connect("qbittorrent") as conn:
conn.execute(
"INSERT INTO qbittorrent_speed_samples (service_id, ts, dl_speed, up_speed) VALUES (?, ?, ?, ?)",
(service_id, ts, dl_speed, up_speed),
)
# Prune: keep only the most recent MAX_SAMPLES rows for this service
conn.execute(
"""DELETE FROM qbittorrent_speed_samples
WHERE service_id = ? AND ts NOT IN (
SELECT ts FROM qbittorrent_speed_samples
WHERE service_id = ?
ORDER BY ts DESC LIMIT ?
)""",
(service_id, service_id, MAX_SAMPLES),
)
def window(self, service_id: str, since_ts: int | None = None) -> list[dict[str, Any]]:
"""Return all samples for a service since a timestamp (or all if None)."""
with self._harness.connect("qbittorrent") as conn:
if since_ts is not None:
rows = conn.execute(
"SELECT ts, dl_speed, up_speed FROM qbittorrent_speed_samples WHERE service_id = ? AND ts >= ? ORDER BY ts ASC",
(service_id, since_ts),
).fetchall()
else:
rows = conn.execute(
"SELECT ts, dl_speed, up_speed FROM qbittorrent_speed_samples WHERE service_id = ? ORDER BY ts ASC",
(service_id,),
).fetchall()
return [{"ts": r[0], "dl_speed": r[1], "up_speed": r[2]} for r in rows]
```
### 2.3 `QbittorrentClient` (`clients/qbittorrent.py`)
Cookie-session HTTP client modeled on `JellyfinClient`'s session pattern.
```python
class QbittorrentClient:
"""Minimal qBittorrent Web API client (read-only: sync/maindata only)."""
def __init__(self, base_url: str, username: str, password: str, timeout: int = 10) -> None:
self.base_url = base_url.rstrip("/")
if not self.base_url.endswith("/api/v2"):
self.base_url += "/api/v2"
self._username = username
self._password = password
self._timeout = timeout
self._session = requests.Session()
self._logged_in = False
def _login(self) -> None:
"""POST username/password to /auth/login; store the SID cookie."""
resp = self._session.post(
f"{self.base_url}/auth/login",
data={"username": self._username, "password": self._password},
timeout=self._timeout,
headers={"Referer": self.base_url},
)
resp.raise_for_status()
if resp.text.strip() != "Ok.":
raise RuntimeError(f"qBittorrent login failed: {resp.text.strip()}")
self._logged_in = True
def _get(self, path: str, **params: Any) -> dict[str, Any]:
"""GET with auto re-login on 403."""
if not self._logged_in:
self._login()
url = f"{self.base_url}{path}"
resp = self._session.get(url, params=params, timeout=self._timeout)
if resp.status_code == 403:
self._logged_in = False
self._login()
resp = self._session.get(url, params=params, timeout=self._timeout)
resp.raise_for_status()
return resp.json()
def maindata(self) -> dict[str, Any]:
"""Fetch /sync/maindata — returns server_state + torrents dict.
server_state contains: dl_info_speed (bytes/s), up_info_speed (bytes/s), etc.
torrents is a dict of {hash: {name, state, progress, ...}}.
"""
return self._get("/sync/maindata")
```
**Endpoints used (all from `/sync/maindata` — single call covers all three widgets):**
| Widget | Data extracted from `maindata()` |
|---|---|
| **totals** | `len(response["torrents"])` — count of all listed torrents |
| **active** | `filter(t for t in response["torrents"].values() if t["state"] in {"downloading","uploading"})` |
| **speed** | `response["server_state"]["dl_info_speed"]` + `["up_info_speed"]` (current instant speed, appended to store) |
**Note:** `/api/v2/transfer/info` is NOT needed (totals = item count per Q2, not transfer bytes).
### 2.4 Widget source adapter — `QbittorrentWidgetSource`
Lives in `widgets/sources.py`, implements `WidgetSource.fetch(service, widget_kind, config)`.
Resolves the client inline from `ServiceRecord` (like `PrometheusWidgetSource` does — no new
dependency-injection helper needed).
```python
class QbittorrentWidgetSource:
"""Fetch qBittorrent data for totals, active, and speed widgets."""
async def fetch(self, service: ServiceRecord | None, widget_kind: str, config: dict[str, Any]) -> dict[str, Any]:
try:
if service is None:
return {"error": "qBittorrent widget is missing its service"}
base_url = str(service.config.get("base_url") or "")
username = str(service.secrets.get("username") or "")
password = str(service.secrets.get("password") or "")
timeout = int(service.config.get("timeout_seconds") or 10)
if not base_url or not username or not password:
return {"error": "qBittorrent service is missing base_url, username, or password"}
client = QbittorrentClient(base_url, username, password, timeout)
data = await asyncio.wait_for(asyncio.to_thread(client.maindata), timeout=timeout)
server_state = data.get("server_state", {})
torrents = data.get("torrents", {})
if widget_kind == "totals":
# Q2: count of listed items, broken down by state
by_state: dict[str, int] = {}
for t in torrents.values():
state = str(t.get("state", "unknown"))
by_state[state] = by_state.get(state, 0) + 1
return {"total": len(torrents), "by_state": by_state}
if widget_kind == "active":
# Q3: downloading or uploading only
active = [
{"name": t.get("name"), "state": t.get("state"),
"size": t.get("size"), "progress": t.get("progress"),
"dl_speed": t.get("dlspeed"), "up_speed": t.get("upspeed")}
for t in torrents.values()
if str(t.get("state", "")) in {"downloading", "uploading"}
]
return {"torrents": active}
if widget_kind == "speed":
# Q1: append sample + return window as {series} shape
# matching PrometheusChartWidget's expected format
dl = int(server_state.get("dl_info_speed", 0))
up = int(server_state.get("up_info_speed", 0))
ts = int(time.time())
store = QbittorrentSampleStore()
store.append(service.id, ts, dl, up)
samples = store.window(service.id)
series = [
{"label": "download",
"points": [{"t": s["ts"] * 1000, "v": s["dl_speed"]} for s in samples]},
{"label": "upload",
"points": [{"t": s["ts"] * 1000, "v": s["up_speed"]} for s in samples]},
]
return {"series": series}
return {"error": f"Unknown qBittorrent widget kind: {widget_kind}"}
except asyncio.TimeoutError:
return {"error": "qBittorrent data fetch timed out"}
except Exception as exc:
logger.exception("qbittorrent adapter failed")
return {"error": f"qBittorrent fetch failed: {exc}"}
```
**Registered in `SERVICE_ADAPTERS`:**
```python
SERVICE_ADAPTERS: dict[str, WidgetSource] = {
"prometheus": PrometheusWidgetSource(),
"qbittorrent": QbittorrentWidgetSource(), # NEW
"alertmanager": AlertmanagerWidgetSource(),
"jellyfin": JellyfinWidgetSource(),
"ssh_tasks": SshTaskWidgetSource(),
}
```
### 2.5 Integration registration — `integrations/qbittorrent.py`
Models config + secret schema on `prometheus.py`.
```python
class QbittorrentConfig(ServiceConfigBase):
base_url: ServiceBaseUrl
timeout_seconds: int = 10
# No per-widget config needed for any of the three kinds
# (all derive from the service connection).
class QbittorrentWidgetConfig(WidgetConfigBase):
pass
DEFINITION = ServiceDefinition(
service_type="qbittorrent",
name="qBittorrent",
description="Torrent client activity, speeds, and item counts.",
config_model=QbittorrentConfig,
secret_fields=[
SecretField(key="username", label="Username", required=True),
SecretField(key="password", label="Password", required=True, helper="Stored encrypted"),
],
widget_kinds=[
widget_kind(kind="totals", name="Totals",
description="Count of all listed torrents, broken down by state.",
model_cls=QbittorrentWidgetConfig, default_config={},
refresh_interval_ms=30_000),
widget_kind(kind="active", name="Active torrents",
description="Torrents currently downloading or uploading.",
model_cls=QbittorrentWidgetConfig, default_config={},
refresh_interval_ms=15_000),
widget_kind(kind="speed", name="Speed chart",
description="Live download/upload speed over a short window.",
model_cls=QbittorrentWidgetConfig, default_config={},
refresh_interval_ms=5_000),
],
)
```
**Registered in `integrations/registry.py`:**
```python
from media_library_viewer_api.integrations.qbittorrent import DEFINITION as QBITTORRENT
SERVICE_DEFINITIONS["qbittorrent"] = QBITTORRENT
```
### 2.6 Cascade-delete wiring
`SettingsStore.delete_service` gains a harness call after the service row is
deleted:
```python
def delete_service(self, service_id: str) -> None:
self.init_schema()
with self.connect() as conn:
# existing widget cascade ...
conn.execute("DELETE FROM services WHERE id = ?", (service_id,))
# NEW: cascade-delete harness-managed data
try:
from media_library_viewer_api.services.service_data import get_service_data_harness
get_service_data_harness().cascade_delete(service_id)
except Exception:
logger.exception("harness cascade-delete failed for service %s", service_id)
```
The try/except guard prevents a harness failure from blocking service deletion
(data cleanup is best-effort; the service row is already gone).
---
## 3. MediaIndex migration design (load-bearing)
### 3.1 Schema migration: add `service_id` column
The `init_schema` method's `CREATE TABLE IF NOT EXISTS` gains the new column.
For existing databases, a migration adds it:
```sql
-- In MEDIA_INDEX_CONCERN.migrations (runs via harness on startup):
-- The init_schema already creates the table for new installs WITH service_id.
-- This migration handles existing DBs that lack the column.
-- media_index_impl.py init_schema: add service_id to the CREATE TABLE columns.
-- Harness migration (runs on existing DBs):
ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT '';
```
The `DEFAULT ''` backfills all existing rows to empty string (the legacy
sentinel — see §3.3 for how this is resolved).
### 3.2 `replace_items` — scoped delete + insert
```python
def replace_items(self, rows: Iterable[dict[str, Any]], service_id: str = "") -> int:
self.init_schema()
row_list = list(rows)
# ... columns list gains "service_id" ...
with self.connect() as conn:
# SCOPED: only delete this service's rows
conn.execute("DELETE FROM media_items WHERE service_id = ?", (service_id,))
conn.executemany(
f"INSERT OR REPLACE INTO media_items ({','.join(columns)}) VALUES ({placeholders})",
[[service_id] + [row.get(column) for column in columns_without_service_id] for row in row_list],
)
# ... metadata ...
return len(row_list)
```
### 3.3 `query` — scoped filter
```python
def query(self, service_id: str = "", ...) -> tuple[list[dict[str, Any]], int]:
# ... existing where clauses ...
# Add service_id filter: if non-empty, scope; if empty (legacy), show all
# (backward-compatibility for the period before multi-instance is wired in UI)
if service_id:
where.append("service_id = ?")
params.append(service_id)
# ... rest unchanged ...
```
**Backfill semantics:** existing rows get `service_id = ''` (empty string). When
`service_id` is empty string in the query, the filter is skipped, so the Media
page shows all items (backward-compatible behavior). When a specific Jellyfin
service triggers a rebuild, `replace_items(rows, service_id=that_service)` scopes
the delete + insert. New builds set the real service_id; legacy rows remain
visible until a rebuild replaces them.
### 3.4 `build_media_index` — thread service_id
`build_media_index` already receives no `service_id` today. Add it as a
parameter and pass it through to `replace_items`:
```python
def build_media_index(
client, user_id, libraries, index=None, page_size=500,
media_root="", fallback_prefix="",
progress_callback=None, should_cancel=None,
service_id: str = "", # NEW
) -> int:
# ... existing logic ...
# Pass service_id to replace_items:
processed_total = index.replace_items(normalized_rows, service_id=service_id)
```
### 3.5 Worker — already threads `service_id`, just pass it to `build_media_index`
The worker's `run_build(final_index_path, staging_index_path, service_id="")`
already receives `service_id` from argparse. The only change: pass it to
`build_media_index(..., service_id=service_id)`.
### 3.6 Media router — thread `service_id` into `query_media`
`query_media` already resolves `client: JellyfinClient = Depends(get_jellyfin_client)`.
The service_id is available via request query param (the dependency layer resolves
it from `?jellyfin_service_id=...`). Add it to the query call:
```python
@router.get("/query")
def query_media(
...,
jellyfin_service_id: str | None = None, # already available pattern
index: MediaIndex = Depends(get_media_index),
) -> dict[str, Any]:
...
rows, total = index.query(service_id=jellyfin_service_id or "", ...)
```
### 3.7 MediaIndex concern registration
```python
MEDIA_INDEX_CONCERN = StorageConcern(
concern_key="media_index",
db_filename="media_index.sqlite", # SAME FILE, unchanged path
migrations=[
"ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT ''",
# This ALTER is idempotent-safe: init_schema creates the table WITH
# service_id for new installs; this migration adds it to existing DBs.
# SQLite ALTER TABLE ADD COLUMN is a no-op if the column already exists
# (we guard with a PRAGMA check in run_migrations, or catch the error).
],
tables=["media_items"],
)
```
**IMPORTANT — ALTER TABLE idempotency:** SQLite raises an error if the column
already exists. The `run_migrations` method should catch this per-statement or
pre-check via `PRAGMA table_info`. Design choice: wrap each migration in a
try/except for "duplicate column name" errors:
```python
def run_migrations(self) -> None:
for concern in self._concerns.values():
path = self.db_path(concern.concern_key)
path.parent.mkdir(parents=True, exist_ok=True)
with sqlite3.connect(path, timeout=30) as conn:
conn.execute("PRAGMA journal_mode=WAL")
for stmt in concern.migrations:
try:
conn.executescript(stmt)
except sqlite3.OperationalError as exc:
if "duplicate column name" not in str(exc).lower():
raise
```
### 3.8 File location confirmation
`DEFAULT_INDEX_PATH = Path(".cache/media_library_viewer/media_index.sqlite")`
unchanged. The harness `base_dir` defaults to `Path(".cache/media_library_viewer")`
(the same parent). The `db_filename = "media_index.sqlite"` matches. No data move.
---
## 4. Frontend design
### 4.1 Three widget components
All three live in `frontend/src/widgets/`, modeled on existing patterns
(`AlertmanagerAlertsWidget`, `PrometheusChartWidget`, `MetricCard`).
**`QbittorrentTotalsWidget.tsx`** — MetricCard-style count tile:
```tsx
// Renders {total: number, by_state: {...}} from useWidgetData.
// Uses SectionCard + numeric display (like BackupsWidget / MetricCard).
// Shows total count prominently + state breakdown badges.
```
**`QbittorrentActiveTorrentsWidget.tsx`** — active torrents list:
```tsx
// Renders {torrents: [{name, state, size, progress, dl_speed, up_speed}]}.
// Uses SectionCard + a compact list/table (DataTable or manual Table rows).
// Shows name, state badge, progress bar, speeds. Max 10 rows with scroll.
```
**`QbittorrentSpeedWidget.tsx`** — speed chart reusing Change A's renderer:
```tsx
// Renders {series: [{label, points:[{t,v}]}]} — IDENTICAL to PrometheusChartWidget.
// Two options:
// (a) Import PrometheusChartWidget directly and pass props (if its props accept
// the series externally rather than via useWidgetData).
// (b) Extract the recharts rendering into a shared <LineSeriesChart series={...} />
// component that both PrometheusChartWidget and QbittorrentSpeedWidget use.
//
// RECOMMENDED: option (b) — extract a shared LineSeriesChart component (~40 lines)
// into frontend/src/components/LineSeriesChart.tsx. Both widgets call useWidgetData
// independently (different refresh intervals) but share the renderer.
```
**`LineSeriesChart.tsx`** (shared renderer extraction):
```tsx
// Extracts: mergeSeries, formatTime, CHART_COLORS, and the <ResponsiveContainer>
// + <LineChart> JSX from PrometheusChartWidget.
// Props: { series: ChartSeries[], height?: number }
// PrometheusChartWidget becomes a thin wrapper: useWidgetData → <LineSeriesChart series={data.data.series} />
// QbittorrentSpeedWidget: same pattern, different refresh interval (5s vs 60s).
```
This keeps the recharts rendering in ONE place (no duplication) while letting each
widget own its polling lifecycle.
### 4.2 Frontend registry binding (`integrations/registry.ts`)
```typescript
qbittorrent: {
serviceType: "qbittorrent",
name: "qBittorrent",
description: "Torrent client activity, speeds, and item counts.",
widgets: [
{ kind: "totals", name: "Totals", description: "...",
refreshIntervalMs: 30_000, defaultConfig: {},
configSchema: { type: "object", properties: {}, required: [] },
component: QbittorrentTotalsWidget },
{ kind: "active", name: "Active torrents", description: "...",
refreshIntervalMs: 15_000, defaultConfig: {},
configSchema: { type: "object", properties: {}, required: [] },
component: QbittorrentActiveTorrentsWidget },
{ kind: "speed", name: "Speed chart", description: "...",
refreshIntervalMs: 5_000, defaultConfig: {},
configSchema: { type: "object", properties: {}, required: [] },
component: QbittorrentSpeedWidget },
],
},
```
### 4.3 Types
No new TypeScript types needed for widget payloads — data flows through the
existing `WidgetDataResponse` + `useWidgetData` polling. The series shape
(`{series:[{label,points:[{t,v}]}]}`) is already used by `PrometheusChartWidget`.
---
## 5. Tests
### 5.1 Backend tests
| Test file | Coverage |
|---|---|
| `backend/tests/test_service_data.py` (NEW) | Harness: register, run_migrations (creates tables), cascade_delete (removes rows by service_id), migration idempotency (ALTER doesn't crash on re-run) |
| `backend/tests/test_qbittorrent_store.py` (NEW) | Store: append + prune (MAX_SAMPLES cap), window (returns samples in order), service_id isolation (two services don't cross-contaminate) |
| `backend/tests/test_qbittorrent_client.py` (NEW) | Client: login flow (POST /auth/login → Ok.), cookie reuse, 403 → re-login, maindata parsing, timeout handling |
| `backend/tests/test_widgets.py` (extend) | `QbittorrentWidgetSource`: totals (counts all torrents), active (filters state), speed (appends sample + returns {series}), missing-service error, timeout error |
### 5.2 Frontend tests
| Test file | Coverage |
|---|---|
| `QbittorrentTotalsWidget.test.tsx` (NEW) | Loading skeleton, error alert, rendered count + state badges |
| `QbittorrentActiveTorrentsWidget.test.tsx` (NEW) | Loading, error, rendered torrent rows |
| `QbittorrentSpeedWidget.test.tsx` (NEW) | Loading, error, rendered chart (series present) |
| `LineSeriesChart.test.tsx` (NEW) | Renders lines from series data, empty state |
### 5.3 Existing MediaIndex tests must stay green
`backend/tests/test_media_index.py` exercises `replace_items`, `query`, `status`.
After migration, these call with the new `service_id=""` default (backward-compatible).
The tests pass unchanged because empty-string service_id shows all rows.
---
## 6. Slice plan (for tasks.md)
Four slices, each ≤400 changed lines, each leaving `pytest` + `npm run build` +
`npm run lint` green. Slices 12 prove the harness; Slice 3 migrates MediaIndex;
Slice 4 wires cascade-delete end-to-end.
### Slice 1: Harness + QbittorrentSampleStore + QbittorrentClient + integration (~280350 lines)
**Files:**
- `backend/src/media_library_viewer_api/services/service_data.py` (NEW — harness + StorageConcern)
- `backend/src/media_library_viewer_api/services/qbittorrent_store.py` (NEW — store + concern)
- `backend/src/media_library_viewer_api/clients/qbittorrent.py` (NEW — client)
- `backend/src/media_library_viewer_api/integrations/qbittorrent.py` (NEW — definition)
- `backend/src/media_library_viewer_api/integrations/registry.py` (MODIFY — add qbittorrent)
- `backend/src/media_library_viewer_api/main.py` (MODIFY — call harness init in lifespan)
- `backend/tests/test_service_data.py` (NEW)
- `backend/tests/test_qbittorrent_store.py` (NEW)
- `backend/tests/test_qbittorrent_client.py` (NEW)
**Exit gate:** harness creates tables + runs migrations + cascade_delete works in
tests. qBit client login/maindata tested with mocked HTTP. No frontend changes yet.
### Slice 2: Widget adapter + frontend widgets + registry binding (~320400 lines)
**Files:**
- `backend/src/media_library_viewer_api/widgets/sources.py` (MODIFY — add QbittorrentWidgetSource + SERVICE_ADAPTERS entry)
- `backend/tests/test_widgets.py` (EXTEND — qBit adapter tests)
- `frontend/src/components/LineSeriesChart.tsx` (NEW — shared renderer extracted from PrometheusChartWidget)
- `frontend/src/widgets/PrometheusChartWidget.tsx` (MODIFY — use LineSeriesChart)
- `frontend/src/widgets/QbittorrentTotalsWidget.tsx` (NEW)
- `frontend/src/widgets/QbittorrentActiveTorrentsWidget.tsx` (NEW)
- `frontend/src/widgets/QbittorrentSpeedWidget.tsx` (NEW)
- `frontend/src/widgets/index.ts` (MODIFY — barrel exports)
- `frontend/src/integrations/registry.ts` (MODIFY — add qbittorrent binding)
- `frontend/src/widgets/__tests__/QbittorrentTotalsWidget.test.tsx` (NEW)
- `frontend/src/widgets/__tests__/QbittorrentActiveTorrentsWidget.test.tsx` (NEW)
- `frontend/src/widgets/__tests__/QbittorrentSpeedWidget.test.tsx` (NEW)
- `frontend/src/components/__tests__/LineSeriesChart.test.tsx` (NEW)
**Exit gate:** qBit widgets render loading/error/data states; speed widget shows
a recharts line chart from {series}; LineSeriesChart is shared with PrometheusChartWidget.
### Slice 3: MediaIndex migration onto harness (~200280 lines)
**Files:**
- `backend/src/media_library_viewer_api/services/service_data.py` (MODIFY — register MEDIA_INDEX_CONCERN)
- `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY — add service_id to schema, replace_items, query)
- `backend/src/media_library_viewer_api/services/media_index.py` (re-export unchanged)
- `backend/src/media_library_viewer_api/workers/media_index_worker.py` (MODIFY — pass service_id to build_media_index)
- `backend/src/media_library_viewer_api/routers/media.py` (MODIFY — thread jellyfin_service_id into query_media)
- `backend/tests/test_media_index.py` (EXTEND — service_id-scoped replace_items + query)
**Exit gate:** all existing MediaIndex tests pass unchanged (empty-string default);
new tests verify scoped delete/insert/query. Media page works identically.
### Slice 4: Cascade-delete end-to-end + integration test (~80120 lines)
**Files:**
- `backend/src/media_library_viewer_api/services/settings_store.py` (MODIFY — call harness.cascade_delete in delete_service)
- `backend/tests/test_services.py` (EXTEND — verify cascade-delete removes qBit samples when service is deleted)
**Exit gate:** deleting a qBittorrent service removes its speed samples; deleting
a Jellyfin service removes its media items.
---
## 7. Key design decisions summary
| # | Decision | Rationale |
|---|---|---|
| D1 | Harness is lifecycle-only (migrations, service_id, cascade) | n=2 justifies abstraction, but data-ops differ wildly (append/window vs replace_all/query). Generalize only the shared lifecycle. |
| D2 | Per-concern DB files via harness `base_dir` + `db_filename` | media_index.sqlite stays put (no data move); qbittorrent.db is new. Separate writers. |
| D3 | `run_migrations` catches "duplicate column name" per-statement | SQLite ALTER TABLE ADD COLUMN is not idempotent; migrations must not crash on re-run. |
| D4 | Speed widget extracts a shared `LineSeriesChart` component | Both PrometheusChartWidget and QbittorrentSpeedWidget need the same recharts renderer; extract it once (DRY) rather than duplicating or tightly coupling. |
| D5 | qBit adapter resolves client inline from ServiceRecord (like PrometheusWidgetSource) | No new dependency-injection helper; the adapter gets config+secrets from the service record. |
| D6 | MediaIndex backfill uses `service_id = ''` (empty string) sentinel | Backward-compatible: existing tests + Media page work unchanged; empty-string queries skip the filter (show all). Real service_ids overwrite on next rebuild. |
| D7 | `replace_items` deletes `WHERE service_id = ?` instead of all | Fixes the existing global-clear bug (building for one Jellyfin wipes others). |
| D8 | Cascade-delete is best-effort (try/except) in delete_service | Harness failure must not block service deletion; data cleanup is non-critical. |
| D9 | Speed sample timestamps multiplied by 1000 for frontend | PrometheusChartWidget expects `t` in milliseconds (JS epoch); SQLite stores Unix seconds. |
| D10 | MAX_SAMPLES = 120 cap with per-append prune | ~2 min history at 1s poll; bounded DB growth; single-statement prune. |
---
## 8. Risks and mitigations
| Risk | Mitigation |
|---|---|
| **MediaIndex migration breaks existing tests.** | `service_id` defaults to `""`; empty-string queries skip the filter → all rows visible. Existing tests pass unchanged. |
| **ALTER TABLE fails on fresh installs** where init_schema already created the column. | `run_migrations` catches "duplicate column name" errors per-statement. |
| **Harness lazy-init hides migration failures.** | `run_migrations` runs on first access (startup lifespan); failures raise (not swallowed) except the expected duplicate-column case. |
| **Speed widget poll interval (5s) too aggressive.** | 5s is the default; configurable via widget config if needed. MAX_SAMPLES cap bounds storage. |
| **qBit cookie expiry between polls.** | Client re-logins on 403 transparently; persistent auth failure surfaces as widget error state. |
| **LineSeriesChart extraction breaks PrometheusChartWidget.** | Slice 2 includes tests for both; extraction is mechanical (move JSX + helpers, pass series as prop). |
| **Review budget (>400 lines).** | Four slices, each ≤400 lines. Slices 12 are independently shippable (qBit works without MediaIndex migration). |
---
## 9. Data flow diagrams
### 9.1 Speed widget data flow (InService path)
```
Dashboard poll (useWidgetData, 5s)
└► GET /api/widgets/instances/{id}/data
└► QbittorrentWidgetSource.fetch(service, "speed", {})
├► QbittorrentClient.maindata()
│ └► /api/v2/sync/maindata → {server_state:{dl_info_speed, up_info_speed}}
├► QbittorrentSampleStore.append(service.id, ts, dl, up)
│ └► INSERT + prune (keep 120)
├► QbittorrentSampleStore.window(service.id)
│ └► SELECT ts, dl_speed, up_speed → [{ts, dl_speed, up_speed}]
└► return {series: [{label:"download", points:[{t,v}]}, {label:"upload", points:[{t,v}]}]}
└► Frontend: QbittorrentSpeedWidget → <LineSeriesChart series={...} />
```
### 9.2 Cascade-delete data flow
```
DELETE /api/services/instances/{id}
└► SettingsStore.delete_service(id)
├► DELETE FROM dashboard_widgets WHERE service_id = ?
├► DELETE FROM services WHERE id = ?
└► ServiceDataHarness.cascade_delete(id)
├► DELETE FROM qbittorrent_speed_samples WHERE service_id = ?
└► DELETE FROM media_items WHERE service_id = ?
```
@@ -0,0 +1,148 @@
# SDD Proposal: Service Storage Harness (with qBittorrent widgets + MediaIndex migration)
**Change:** `service-storage-harness`
**Phase:** proposal
**Date:** 2026-07-08
## 1. Problem / Why Now
Two things are happening at once, and this change addresses both:
1. **Feature request — qBittorrent widgets.** The operator wants at-a-glance qBittorrent visibility on the Manage dashboard: active downloads/uploads, total bytes transferred, and a download/upload speed indicator. qBittorrent is currently a blind spot — it is not modeled in the service registry and exposes no widgets.
2. **Platform gap — services have no owned persistence.** Manage already has a service registry (machines, Jellyfin, Grafana, Prometheus, Alertmanager, Jellyseerr, ssh_tasks) and a configurable widget system. But when a service needs to *remember operational data over time*, there is no shared answer. `MediaIndex` is a one-off: it owns its own `media_index.db`, runs its own schema, and is built by a dedicated subprocess worker. It is a "special snowflake." Rather than add a *second* snowflake for qBittorrent speed history, we extract the shared lifecycle into a small **`ServiceDataHarness`** and prove it with qBittorrent, then migrate `MediaIndex` onto it so the pattern has two evidence-based consumers.
The two are bundled because the second service (qBittorrent) is the one that justifies generalizing from the first (`MediaIndex`) — n=2 is what makes the abstraction worth its cost.
## 2. Target Users and Situations
- **Primary users:** Homelab operators running qBittorrent alongside Manage, who want download/upload activity visible without opening the qBittorrent UI.
- **Workflow moments:**
- Glance at the dashboard: "is a download running, how fast, how much has been transferred?"
- Decide whether qBittorrent is healthy without leaving Manage.
- Browse the Jellyfin media catalog (unchanged UX) — which now runs on the same storage harness, validating the abstraction.
- **Urgency:** Medium. The feature is valuable but not breaking; the platform refactor is opportunistic (do it now while only two consumers exist, before a third snowflake appears).
## 3. Product Outcome
After this change, an authenticated user can:
- Register one or more **qBittorrent service instances** in the existing Services UI (URL + username + password, stored as encrypted secrets — same posture as Grafana/Prometheus).
- Place three discrete **qBittorrent widget kinds** on the dashboard:
1. **Totals tile** — count of currently-listed torrents (total items in the qBittorrent list, which may exceed the active count because rate/connection limits leave some torrents non-transferring). NOT cumulative bytes transferred.
2. **Active torrents list** — torrents whose state is `downloading` or `uploading`.
3. **Speed chart** — live download/upload speed over a short rolling window, rendered with the **`PrometheusChartWidget` recharts renderer established by the `prometheus-direct-charting` change** (fed from the `QbittorrentSampleStore` via an InService-style data path returning the same `{series}` shape). No hand-rolled SVG sparkline; the thin-dashboard rule was already repealed for recharts charting by the foundational change.
- Continue using the Jellyfin **Media page** exactly as before; its underlying storage moves onto the harness transparently and additionally becomes **multi-instance capable** (scoped per Jellyfin service).
## 4. Scope Boundaries and Non-Goals
### In scope
- **`ServiceDataHarness`** — a general lifecycle layer: per-concern DB files, per-integration schema migrations, `service_id` scoping of all tables, and cascade-delete when a service instance is removed.
- **qBittorrent integration** — new `integrations/qbittorrent.py` (config + secret schema + widget kinds), new `clients/qbittorrent.py` (Web API client, cookie login), a `QbittorrentSampleStore` (speed samples), widget source adapter(s) in `widgets/sources.py`, registry entry.
- **qBittorrent frontend** — three widget components under `frontend/src/widgets/`, binding in `integrations/registry.ts`, types, API client functions.
- **MediaIndex migration**`MediaIndex` registered with the harness, `media_items` scoped by `service_id`, subprocess worker updated. The `media_index.db` file location is unchanged (only gains a column + harness registration) to minimize churn on a load-bearing feature.
- **Cascade-delete wiring** — removing a service instance cleans up its owned data in every harness-managed table.
### Non-goals (explicitly out of scope)
- **Torrent management UI** — no add/pause/delete/recheck/priority UI. Read-only visibility only.
- **A generic time-series database.** The harness owns *lifecycle*, not a generic `(service_id, key, ts, value)` table. Each integration owns its own schema and operations.
- **A generic CRUD/ORM layer.** Stores keep bespoke operations (`append/window` vs `replace_all/query`); only the lifecycle is shared.
- **Per-user or per-tenant storage partitioning.** Storage is scoped by `service_id` only.
- **Re-indexing/migrating existing media data.** The `media_index.db` file stays in place; the migration is a schema column add + harness registration, not a data move.
- **WebSocket / push updates.** Polling via existing `useWidgetData(widgetId, refreshIntervalMs)` is sufficient.
- **Transfer-byte totals.** The totals widget counts torrent *items*, not cumulative bytes uploaded/downloaded (per the §8 Q2 resolution). Byte totals are out of scope.
- **qBittorrent Prometheus exporter.** Not built; the speed chart is sourced from the local `QbittorrentSampleStore`, not Prometheus.
## 5. High-Level Approach
### 5.1 Architectural decisions (locked during grilling)
| Decision | Outcome |
|---|---|
| **Storage tech** | SQLite. Low volume; proven in stack. |
| **Abstraction level** | `ServiceDataHarness` owns the *lifecycle* (DB filename, migrations, `service_id` scoping, cascade-delete). Each integration owns its *operations* in a bespoke Store. General where shared; bespoke where not. |
| **MediaIndex migration** | Included in this change, but **sequenced**: harness + qBit proven first; MediaIndex folded in after. Inside one change, never two simultaneous risks. |
| **DB topology** | Per-concern DB files. `media_index.db` keeps its file (gains `service_id` + registration); new `qbittorrent.db`. |
### 5.2 Backend
1. **`ServiceDataHarness`** (new, `services/service_data.py` or similar):
- A registry of *concerns*: each integration declares a DB filename, an ordered list of migration SQL statements, the tables it owns, and a `service_id` column convention.
- On startup: runs pending migrations per concern DB.
- On service deletion: cascades — for each concern table, `DELETE FROM <table> WHERE service_id = ?`.
- Provides connection management per concern DB (separate connections → bulk media writes don't share a writer with chatty qBit appends).
2. **qBittorrent client** (`clients/qbittorrent.py`):
- Login via `/api/v2/auth/login` → cookie session; reuse cookie across calls; re-login on 403.
- Endpoints used: `/api/v2/transfer/info` (global totals), `/api/v2/sync/maindata` (active torrents + current speeds), `/api/v2/torrents/info` (filtered lists if needed).
- Timeout + reachability handling consistent with `JellyfinClient`/`GrafanaWidgetSource`.
3. **`QbittorrentSampleStore`** (new):
- Table `qbittorrent_speed_samples(service_id TEXT, ts INTEGER, dl_speed INTEGER, up_speed INTEGER)` in `qbittorrent.db`.
- Operations: `append(service_id, ts, dl, up)`, `window(service_id, since_ts)`, `prune(service_id, older_than_ts)`.
- Pruning runs on each append (cap retention to the configured window, e.g. 120 samples).
4. **Widget source adapter**`QbittorrentWidgetSource` in `widgets/sources.py` implementing the existing `WidgetSource.fetch(service, widget_kind, config)` contract. For the speed kind, `fetch` appends a sample to the store and returns the current window for rendering.
5. **Integration registration** — add `qbittorrent` to `integrations/registry.py` with config schema (base URL, timeout) and secret schema (username, password); declare widget kinds.
6. **MediaIndex migration** — register `MediaIndex` as a concern; add `service_id` column (backfill existing rows with the default/local Jellyfin service id); update `replace_items`/`query`/worker to scope by `service_id`; route through the harness connection.
### 5.3 Frontend
1. **Three widget components** under `frontend/src/widgets/`:
- `QbittorrentTotalsWidget.tsx` — numeric tiles for all-time bytes (reuses `MetricCard`/`SectionCard`).
- `QbittorrentActiveTorrentsWidget.tsx` — list/table of active torrents (reuses the already-migrated `DataTable` + `@tanstack/react-table`; **no new DataGrid migration risk**).
- `QbittorrentSpeedWidget.tsx` — visual form decided by §8 Q1 (numeric tiles + delta, or sparkline if an exception is granted, or a PrometheusMetricWidget binding if the exporter path is chosen).
2. **Registry binding** — add a `qbittorrent` entry to `SERVICE_REGISTRY` in `frontend/src/integrations/registry.ts` mapping the three widget kinds to components + config schemas.
3. **Types + API client**`frontend/src/types/index.ts` and `frontend/src/api/client.ts` gain qBittorrent-aware widget kinds only (data flows through the existing `useWidgetData` polling; no new endpoints beyond widget CRUD).
### 5.4 Type contracts
- Backend Pydantic models for qBittorrent config/secret schema in `integrations/qbittorrent.py`.
- Frontend TypeScript interfaces for qBittorrent widget payloads.
- Harness has no data-shape types of its own (it is lifecycle-only) — keeping the seam clean.
## 6. Success Criteria / Acceptance Criteria
1. A user can register a qBittorrent service instance (URL + username + password) and the three widget kinds appear as bindable on the dashboard.
2. Totals tile shows all-time bytes from `transferInfo`; active list shows downloading/uploading torrents; speed indicator reflects current rates.
3. `ServiceDataHarness` runs migrations on startup and cascades deletes across qBittorrent samples *and* media items when a service is removed.
4. `MediaIndex` continues to power the Media page identically (all existing Media tests green) and is now scoped by `service_id`.
5. Removing a Jellyfin service removes only that service's media rows; removing a qBittorrent service removes only that service's speed samples.
6. A misconfigured/unreachable qBittorrent instance degrades gracefully per-widget (error state), the rest of the dashboard renders.
7. Backend tests (`pytest`) and frontend `npm run build` + `npm run lint` stay green.
8. No secrets land in widget `config_json`; qBittorrent password is encrypted via the existing Fernet path.
9. No new charting dependency is added unless §8 Q1 grants an explicit exception.
## 7. Risks and Mitigations
| Risk | Mitigation |
|------|------------|
| **Compound risk: new abstraction × load-bearing refactor.** A wrong harness shape breaks the Media page. | Sequence inside the change: harness + qBit land and pass tests first; MediaIndex migrates only after the harness shape is settled in code. |
| **Harness over-generalization.** Building a meta-framework from n=2. | Keep the harness lifecycle-only; do NOT add a generic value table or generic CRUD. Each store keeps bespoke operations. Extract further only when a third shape appears. |
| **Thin-dashboard rule collision (§8 Q1).** In-app speed graph violates the stated "no in-app charting" rule. | Resolve in the question round before spec; default to the rule-compliant numeric-tile + Grafana deep-link option unless an explicit exception is granted. |
| **qBittorrent auth lifecycle.** Cookie expiry / 403 handling. | Re-login transparently on 403; short request timeout; surface persistent auth failure as widget error state. |
| **MediaIndex backfill correctness.** Adding `service_id` to existing rows. | Backfill all existing rows to the default/local Jellyfin service id; migration is additive; existing Media tests must pass unchanged. |
| **Review budget (>400 changed lines).** Bundled scope is large. | Slice into chained PRs (Slice 1: harness + qBit; Slice 2: MediaIndex migration). Each slice leaves `npm run build` + `npm run lint` + `pytest` green. |
| **DataGrid migration (config rule callout).** | Not a risk here: the active-torrents list reuses the already-migrated `DataTable` (`@tanstack/react-table`). No new DataGrid migration is introduced. The key technical risks are the harness abstraction and the MediaIndex refactor, above. |
## 8. Resolved Questions (question round complete)
All five product/semantic questions resolved during grilling + the Change A (`prometheus-direct-charting`) lifecycle:
- **Q1 — Speed visualization.** RESOLVED: the thin-dashboard rule was repealed by Change A; in-app recharts charting is now sanctioned. The qBit speed widget reuses the `PrometheusChartWidget` recharts renderer, fed from `QbittorrentSampleStore.window()` via an InService data path returning the same `{series}` shape. No SVG sparkline, no Grafana dependency.
- **Q2 — "Totals" semantics.** RESOLVED: count of currently-listed torrent items (total in the qBittorrent list), which may exceed the active count due to rate/connection limits. NOT transfer bytes, NOT per-session counters.
- **Q3 — "Active" definition.** RESOLVED: torrents in state `downloading` or `uploading` only.
- **Q4 — Instances.** RESOLVED: support N qBittorrent instances, each a service row, independently scoped by `service_id`.
- **Q5 — Auth model.** RESOLVED: username/password login → cookie session, encrypted via Fernet. No reverse-proxy no-auth flag in this pass.
## 9. Future Phases
1. **Third service consumer** — when a service with a genuinely new storage shape arrives, reconsider promoting the harness toward a broader abstraction (evidence-based, n=3).
2. **Torrent management** — add/pause/delete actions (would require write endpoints + confirmation UX).
3. **Storage admin UI** — surface harness-managed table sizes and a "clear cached data" action per service.
4. **Reverse-proxy / no-auth qBittorrent flag** — for setups behind Authentik where native login is bypassed.
5. **Transfer-byte totals** — if desired later, add a separate widget using `transferInfo.globalUploaded/Downloaded`.
@@ -0,0 +1,155 @@
# SDD Spec: Service Storage Harness (qBittorrent widgets + MediaIndex migration)
**Change:** `service-storage-harness`
**Phase:** spec
**Date:** 2026-07-09
This spec defines the acceptance requirements for the change, derived from the reconciled `proposal.md` and `design.md`. Requirements are testable.
## Requirement categories
1. ServiceDataHarness (lifecycle layer)
2. QbittorrentSampleStore
3. QbittorrentClient
4. qBittorrent widget source adapter
5. qBittorrent frontend widgets
6. MediaIndex migration onto harness
7. Cascade-delete wiring
8. Test + build greenness
---
## 1. ServiceDataHarness (lifecycle layer)
### SS-101 — Harness is lifecycle-only
A `ServiceDataHarness` class exists in `backend/src/media_library_viewer_api/services/service_data.py` that owns ONLY lifecycle concerns: per-concern DB provisioning, per-integration migrations, `service_id` cascade-delete. It MUST NOT provide generic data operations (no generic value table, no generic CRUD).
### SS-102 — Concern registration
An integration/concern registers via a dataclass carrying: DB filename, ordered migration SQL list, owned-tables list, and `service_id` column name. The harness stores registered concerns.
### SS-103 — Idempotent migrations
`run_migrations` runs each concern's migration statements and MUST be idempotent — specifically, re-running `ALTER TABLE ... ADD COLUMN` on an already-migrated DB MUST NOT raise (the harness catches "duplicate column name" per-statement).
### SS-104 — Cascade-delete across concerns
`cascade_delete(service_id)` iterates every registered concern and, for each owned table, executes `DELETE FROM <table> WHERE <service_id_column> = ?`. It MUST cover every registered concern (qBittorrent samples + media items after this change).
## 2. QbittorrentSampleStore
### SS-105 — Schema
`QbittorrentSampleStore` owns a `qbittorrent_speed_samples` table with columns `(service_id, ts, dl_speed, up_speed)` and an index on `(service_id, ts)`, in a dedicated `qbittorrent.db` file (per-concern topology).
### SS-106 — append/window/prune operations
The store exposes `append(service_id, ts, dl_speed, up_speed)`, `window(service_id, since_ts)` returning ordered rows, and prunes per-append to `MAX_SAMPLES = 120`.
### SS-107 — Registered as a harness concern
The qBittorrent sample store is registered with the harness so its table participates in cascade-delete (SS-104).
## 3. QbittorrentClient
### SS-108 — Cookie login
`QbittorrentClient` authenticates via `POST /api/v2/auth/login` with username/password, stores the resulting cookie, and reuses it for subsequent requests. Credentials are resolved from the `ServiceRecord` secrets (Fernet-encrypted at rest).
### SS-109 — 403 re-login
On HTTP 403 the client MUST transparently re-login once and retry the request.
### SS-110 — maindata fetch
The client exposes `maindata()` calling `/api/v2/sync/maindata` and returning its dict. Errors (timeout, connection, non-2xx) propagate as exceptions for the adapter to catch.
## 4. qBittorrent widget source adapter
### SS-111 — Three widget kinds dispatched
`QbittorrentWidgetSource.fetch(service, widget_kind, config)` dispatches on `widget_kind` ∈ {`totals`, `active`, `speed`}, resolving the client inline from `ServiceRecord` (same pattern as `PrometheusWidgetSource`).
### SS-112 — totals = item count
`totals` returns the count of currently-listed torrents from `maindata()` as `{total: int}`. It is NOT cumulative transfer bytes.
### SS-113 — active = downloading/uploading filter
`active` returns the subset of torrents whose `state` is `downloading` or `uploading` as `{torrents: [...]}`.
### SS-114 — speed appends sample + returns series
`speed` reads the current dl/up speeds from `maindata()`, appends a sample via `QbittorrentSampleStore.append`, and returns `{series: [{label: "download", points: [...]}, {label: "upload", points: [...]}]}` from `.window()` — the exact shape `LineSeriesChart` consumes (timestamps in JS milliseconds).
### SS-115 — Errors degrade gracefully
Adapter errors (auth failure, timeout, connection) return `{error: str}` and MUST NOT raise.
## 5. qBittorrent frontend widgets
### SS-116 — Three widget components
`QbittorrentTotalsWidget`, `QbittorrentActiveTorrentsWidget`, `QbittorrentSpeedWidget` exist under `frontend/src/widgets/` and are bound to the `qbittorrent` service binding in `integrations/registry.ts` with their respective kinds.
### SS-117 — Speed widget reuses shared renderer
`QbittorrentSpeedWidget` renders via the shared `LineSeriesChart` component (extracted from `PrometheusChartWidget`). No new charting code or charting dependency.
### SS-118 — LineSeriesChart extraction is non-regressive
The extraction of the recharts body into `frontend/src/components/LineSeriesChart.tsx` MUST leave `PrometheusChartWidget`'s behavior and tests green; `PrometheusChartWidget` becomes a thin wrapper.
## 6. MediaIndex migration onto harness
### SS-119 — service_id column added
`media_items` gains a `service_id TEXT NOT NULL DEFAULT ''` column via an idempotent harness migration. The `media_index.db` file location is UNCHANGED.
### SS-120 — Existing rows backfill
All pre-existing `media_items` rows receive `service_id = ''` (via the column DEFAULT), preserving their visibility.
### SS-121 — replace_items is scoped (bug fix)
`replace_items(service_id=...)` deletes only `WHERE service_id = ?` (not a global `DELETE FROM media_items`). This FIXES the latent global-clear bug where rebuilding for one Jellyfin instance wiped another's rows. A regression test MUST prove service A's rows survive service B's rebuild.
### SS-122 — query is backward-compatible
`query(service_id="")` returns all rows (no filter); `query(service_id="X")` scopes to service X. Existing tests that pass no `service_id` MUST continue to pass unchanged.
### SS-123 — Worker threads service_id into rows
The media index worker (which already receives `--service-id`) passes it into `replace_items` so newly-built rows are stamped with the real service_id.
### SS-124 — Registered as a harness concern
MediaIndex registers `media_items` as a harness-owned table so it participates in cascade-delete.
## 7. Cascade-delete wiring
### SS-125 — delete_service triggers harness cascade
`settings_store.delete_service` calls `ServiceDataHarness.cascade_delete(service_id)` after its existing cleanup, in a best-effort try/except (a cascade failure MUST NOT crash the service deletion; it logs and continues).
### SS-126 — End-to-end cascade across both concerns
Deleting a service removes both its qBittorrent samples AND its media rows. An integration test MUST prove this across both concerns, and MUST prove rows of OTHER services are preserved.
## 8. Test + build greenness
### SS-127 — Backend tests + lint green
`PYTHONPATH=src python3 -m pytest -q` and `PYTHONPATH=src python3 -m ruff check src tests` from `backend/` MUST pass, including new tests for: harness lifecycle, qBit store, qBit client (login/403/maindata), qBit widget adapter (3 branches), MediaIndex scoped replace_items regression, and cascade-delete integration.
### SS-128 — Frontend build + lint green
`npm run build` and `npm run lint` from `frontend/` MUST pass (0 errors). New widget tests cover loading/error/rendered states; PrometheusChartWidget tests stay green (SS-118).
---
## Notes for downstream phases
- **Verify** should confirm SS-101..SS-128 against source; the load-bearing items are SS-121 (scoped replace_items bug fix), SS-122/SS-123 (MediaIndex backward-compat), and SS-118 (non-regressive extraction).
- **Sync + archive** follow the Change A pattern (delta into a new canonical domain `service-storage`, then archive to `openspec/changes/archive/2026-07-09-service-storage-harness/`).
@@ -0,0 +1,139 @@
# Service Storage — Delta (`service-storage-harness`)
> Change: `service-storage-harness` · Domain: `service-storage` · Phase: **spec** (reconciled during `sdd-sync`).
> Distilled verbatim from the verified flat `spec.md` (28 requirements, SS-101 … SS-128) of change
> `service-storage-harness`, cross-referenced against `design.md` and `verify-report.md`. Captures
> the **durable, post-change end-state contracts** for the lifecycle-only `ServiceDataHarness`, the
> qBittorrent store/client/widget stack built on it, the MediaIndex migration onto the harness, and
> the cross-concern cascade-delete wiring.
## ADDED Requirements
> The canonical `openspec/specs/service-storage/spec.md` did not exist before this change. All
> requirements below are therefore **ADDED** to a new `service-storage` domain; `sdd-sync` copies
> them into the canonical spec (native helper rule: when the canonical spec does not exist, the
> change spec becomes the new canonical spec).
>
> Requirement IDs (SS-101 … SS-128) and body text are preserved **exactly** from the verified flat
> `spec.md`. Requirements are grouped logically and listed in the following group order:
>
> - **ServiceDataHarness (lifecycle layer)** — SS-101 … SS-104
> - **QbittorrentSampleStore** — SS-105 … SS-107
> - **QbittorrentClient** — SS-108 … SS-110
> - **qBittorrent widget source adapter** — SS-111 … SS-115
> - **qBittorrent frontend widgets** — SS-116 … SS-118
> - **MediaIndex migration onto harness** — SS-119 … SS-124
> - **Cascade-delete wiring** — SS-125 … SS-126
> - **Test and build greenness** — SS-127 … SS-128
### Requirement: SS-101 — Harness is lifecycle-only
A `ServiceDataHarness` class exists in `backend/src/media_library_viewer_api/services/service_data.py` that owns ONLY lifecycle concerns: per-concern DB provisioning, per-integration migrations, `service_id` cascade-delete. It MUST NOT provide generic data operations (no generic value table, no generic CRUD).
### Requirement: SS-102 — Concern registration
An integration/concern registers via a dataclass carrying: DB filename, ordered migration SQL list, owned-tables list, and `service_id` column name. The harness stores registered concerns.
### Requirement: SS-103 — Idempotent migrations
`run_migrations` runs each concern's migration statements and MUST be idempotent — specifically, re-running `ALTER TABLE ... ADD COLUMN` on an already-migrated DB MUST NOT raise (the harness catches "duplicate column name" per-statement).
### Requirement: SS-104 — Cascade-delete across concerns
`cascade_delete(service_id)` iterates every registered concern and, for each owned table, executes `DELETE FROM <table> WHERE <service_id_column> = ?`. It MUST cover every registered concern (qBittorrent samples + media items after this change).
### Requirement: SS-105 — Schema
`QbittorrentSampleStore` owns a `qbittorrent_speed_samples` table with columns `(service_id, ts, dl_speed, up_speed)` and an index on `(service_id, ts)`, in a dedicated `qbittorrent.db` file (per-concern topology).
### Requirement: SS-106 — append/window/prune operations
The store exposes `append(service_id, ts, dl_speed, up_speed)`, `window(service_id, since_ts)` returning ordered rows, and prunes per-append to `MAX_SAMPLES = 120`.
### Requirement: SS-107 — Registered as a harness concern
The qBittorrent sample store is registered with the harness so its table participates in cascade-delete (SS-104).
### Requirement: SS-108 — Cookie login
`QbittorrentClient` authenticates via `POST /api/v2/auth/login` with username/password, stores the resulting cookie, and reuses it for subsequent requests. Credentials are resolved from the `ServiceRecord` secrets (Fernet-encrypted at rest).
### Requirement: SS-109 — 403 re-login
On HTTP 403 the client MUST transparently re-login once and retry the request.
### Requirement: SS-110 — maindata fetch
The client exposes `maindata()` calling `/api/v2/sync/maindata` and returning its dict. Errors (timeout, connection, non-2xx) propagate as exceptions for the adapter to catch.
### Requirement: SS-111 — Three widget kinds dispatched
`QbittorrentWidgetSource.fetch(service, widget_kind, config)` dispatches on `widget_kind` ∈ {`totals`, `active`, `speed`}, resolving the client inline from `ServiceRecord` (same pattern as `PrometheusWidgetSource`).
### Requirement: SS-112 — totals = item count
`totals` returns the count of currently-listed torrents from `maindata()` as `{total: int}`. It is NOT cumulative transfer bytes.
### Requirement: SS-113 — active = downloading/uploading filter
`active` returns the subset of torrents whose `state` is `downloading` or `uploading` as `{torrents: [...]}`.
### Requirement: SS-114 — speed appends sample + returns series
`speed` reads the current dl/up speeds from `maindata()`, appends a sample via `QbittorrentSampleStore.append`, and returns `{series: [{label: "download", points: [...]}, {label: "upload", points: [...]}]}` from `.window()` — the exact shape `LineSeriesChart` consumes (timestamps in JS milliseconds).
### Requirement: SS-115 — Errors degrade gracefully
Adapter errors (auth failure, timeout, connection) return `{error: str}` and MUST NOT raise.
### Requirement: SS-116 — Three widget components
`QbittorrentTotalsWidget`, `QbittorrentActiveTorrentsWidget`, `QbittorrentSpeedWidget` exist under `frontend/src/widgets/` and are bound to the `qbittorrent` service binding in `integrations/registry.ts` with their respective kinds.
### Requirement: SS-117 — Speed widget reuses shared renderer
`QbittorrentSpeedWidget` renders via the shared `LineSeriesChart` component (extracted from `PrometheusChartWidget`). No new charting code or charting dependency.
### Requirement: SS-118 — LineSeriesChart extraction is non-regressive
The extraction of the recharts body into `frontend/src/components/LineSeriesChart.tsx` MUST leave `PrometheusChartWidget`'s behavior and tests green; `PrometheusChartWidget` becomes a thin wrapper.
### Requirement: SS-119 — service_id column added
`media_items` gains a `service_id TEXT NOT NULL DEFAULT ''` column via an idempotent harness migration. The `media_index.db` file location is UNCHANGED.
### Requirement: SS-120 — Existing rows backfill
All pre-existing `media_items` rows receive `service_id = ''` (via the column DEFAULT), preserving their visibility.
### Requirement: SS-121 — replace_items is scoped (bug fix)
`replace_items(service_id=...)` deletes only `WHERE service_id = ?` (not a global `DELETE FROM media_items`). This FIXES the latent global-clear bug where rebuilding for one Jellyfin instance wiped another's rows. A regression test MUST prove service A's rows survive service B's rebuild.
### Requirement: SS-122 — query is backward-compatible
`query(service_id="")` returns all rows (no filter); `query(service_id="X")` scopes to service X. Existing tests that pass no `service_id` MUST continue to pass unchanged.
### Requirement: SS-123 — Worker threads service_id into rows
The media index worker (which already receives `--service-id`) passes it into `replace_items` so newly-built rows are stamped with the real service_id.
### Requirement: SS-124 — Registered as a harness concern
MediaIndex registers `media_items` as a harness-owned table so it participates in cascade-delete.
### Requirement: SS-125 — delete_service triggers harness cascade
`settings_store.delete_service` calls `ServiceDataHarness.cascade_delete(service_id)` after its existing cleanup, in a best-effort try/except (a cascade failure MUST NOT crash the service deletion; it logs and continues).
### Requirement: SS-126 — End-to-end cascade across both concerns
Deleting a service removes both its qBittorrent samples AND its media rows. An integration test MUST prove this across both concerns, and MUST prove rows of OTHER services are preserved.
### Requirement: SS-127 — Backend tests + lint green
`PYTHONPATH=src python3 -m pytest -q` and `PYTHONPATH=src python3 -m ruff check src tests` from `backend/` MUST pass, including new tests for: harness lifecycle, qBit store, qBit client (login/403/maindata), qBit widget adapter (3 branches), MediaIndex scoped replace_items regression, and cascade-delete integration.
### Requirement: SS-128 — Frontend build + lint green
`npm run build` and `npm run lint` from `frontend/` MUST pass (0 errors). New widget tests cover loading/error/rendered states; PrometheusChartWidget tests stay green (SS-118).
@@ -0,0 +1,166 @@
# Sync Report — `service-storage-harness`
> Phase: **sync** · Change: `service-storage-harness` · Repo: `/home/user/manage`
> Mode: file-backed (`artifactStore: openspec`). No source-code edits; only OpenSpec artifacts were
> written. Not committed (parent owns the commit). The change folder was **not** moved (that is
> `sdd-archive`'s job).
**Status: SYNCED.** A new canonical domain `openspec/specs/service-storage/spec.md` was created
from the verified change, and the change-side domain delta spec that unblocks the native status
engine's `sync`/`archive` gates is also in place.
---
## 1. Executive summary
The `service-storage-harness` change shipped a **complete but flat** `openspec/changes/service-storage-harness/spec.md`
(28 requirements, SS-101 … SS-128) with **no** per-domain delta spec under
`openspec/changes/service-storage-harness/specs/<domain>/`. `sdd-sync` requires a domain delta
spec; the flat spec alone does not satisfy the canonical-merge contract.
Verify already returned **PASS** (verdict in `verify-report.md`; all four gates green — backend
`pytest` 322 passed, `ruff` clean, frontend `npm run build` exit 0, `npm run lint` 0 errors).
Functional coverage was **28/28 fully PASS**. The verify report's single CRITICAL was an **archive**
blocker (30 unchecked task checkboxes + missing `apply-progress.md`); `apply-progress.md` now exists
and reconciles the 35 tasks (the task tracker condition does **not** block `sdd-sync` of the green
code).
This sync **reconciles** the flat-spec-vs-domain-spec gap:
1. Authored the missing **change-side domain delta spec**
`openspec/changes/service-storage-harness/specs/service-storage/spec.md` — using a clean
`## ADDED Requirements` structure that preserves the exact requirement IDs (SS-101 … SS-128) and
text from the verified flat `spec.md`. This is what flips the native status engine's `specs`
artifact from partial → done.
2. **Synced** the end-state into the **canonical store**
`openspec/specs/service-storage/spec.md` — the actual sync target. Because the canonical
`service-storage` domain did not previously exist, the native helper rule applies: *when the
canonical spec does not exist, the change spec becomes the new canonical spec.* The two files
therefore carry identical requirement bodies (delta under `## ADDED Requirements`; canonical
under `## Requirements`).
Domain name **`service-storage`** was chosen (per the dispatch brief) because it covers the full
new model: the lifecycle-only `ServiceDataHarness` layer, the qBittorrent store + client + widget
stack, and the MediaIndex migration that established the per-service storage pattern. It is distinct
from the existing canonical domains `web-ui` (MUI→shadcn migration) and `prometheus-charting`
(direct Prometheus metric visualization), neither of which was **touched**.
## 2. Structured status & actionContext findings
The native `gentle-pi.sdd-status` passed by the parent reports `changeName: null` with
`blockedReasons: ["Change selection is ambiguous: mobile-responsive-parity, service-storage-harness,
services-as-hub-ia."]` because the engine auto-detected three active changes. This sync task was
**explicitly assigned** `service-storage-harness`; the ambiguity is a parent-resolution artifact
and does not block this phase (`isNonAuthoritative: false`).
- `artifactStore: openspec`; change root `openspec/changes/service-storage-harness/`.
- Artifacts present: `proposal.md`, `spec.md`, `design.md`, `tasks.md`, `verify-report.md`,
`apply-progress.md`.
- `verify: PASS` (verify-report verdict; gates green at `c9404f0`).
- `actionContext`: `mode: repo-local`, `workspaceRoot: /home/user/manage`,
`allowedEditRoots: ["/home/user/manage"]`, `warnings: []`. All three files written are inside the
authoritative workspace / allowed edit roots. ✓
- `relationships.sameDomainActiveChanges: []`, `collisions: []` — **no active same-domain
collisions**, so no archive/sync ordering decision was required.
- The new `service-storage` domain is distinct from the existing `web-ui` and `prometheus-charting`
canonical domains; both were left untouched.
**Post-sync structural change:** `openspec/changes/service-storage-harness/specs/service-storage/spec.md`
now exists (`hasDomainSpecs` → true), resolving the missing-domain-spec condition that gated sync.
The flat `spec.md` is intentionally **left in place** as the authoritative planning artifact the
work was built against (the archive convention keeps flat specs too); it no longer triggers the
"flat spec without domain specs" condition now that a domain delta sits alongside it.
## 3. Domains synced & canonical files updated
| Domain | Change-side delta (source) | Canonical (sync target) | Action |
|---|---|---|---|
| `service-storage` | `openspec/changes/service-storage-harness/specs/service-storage/spec.md` | `openspec/specs/service-storage/spec.md` | **NEW domain**`## ADDED Requirements` copied into canonical as a new spec |
- **Canonical file created:** `openspec/specs/service-storage/spec.md` (28 requirements).
- **Change-side delta created:** `openspec/changes/service-storage-harness/specs/service-storage/spec.md`
(28 requirements, all `## ADDED Requirements`).
## 4. Requirement delta (ADDED / MODIFIED / REMOVED)
- **ADDED (28)** — all to the new `service-storage` domain (canonical did not exist pre-change).
IDs and text preserved verbatim from the verified flat `spec.md`. Grouped logically:
- *ServiceDataHarness (lifecycle layer)* — SS-101, SS-102, SS-103, SS-104
- *QbittorrentSampleStore* — SS-105, SS-106, SS-107
- *QbittorrentClient* — SS-108, SS-109, SS-110
- *qBittorrent widget source adapter* — SS-111, SS-112, SS-113, SS-114, SS-115
- *qBittorrent frontend widgets* — SS-116, SS-117, SS-118
- *MediaIndex migration onto harness* — SS-119, SS-120, SS-121, SS-122, SS-123, SS-124
- *Cascade-delete wiring* — SS-125, SS-126
- *Test and build greenness* — SS-127, SS-128
- **MODIFIED (0)** — none (new domain; no pre-existing canonical requirements to replace).
- **REMOVED (0)** — none.
- **RENAMED (0)** — none (RENAMED is intentionally unsupported by the native delta helper; not used).
## 5. Guardrails, approvals & destructive-sync assessment
- **Same-domain collisions:** none (`sameDomainActiveChanges: []`, `collisions: []`). The new
`service-storage` domain does not overlap the existing `web-ui` or `prometheus-charting`
canonical domains. No ordering decision was needed.
- **Destructive sync:** **not applicable.** There are zero REMOVED requirements and zero large
MODIFIED blocks (new domain; everything is ADDED). No destructive-sync parent approval was
required beyond the explicit reconciliation instruction in the task.
- **Legacy flat spec:** detected pre-sync; resolved by adding the domain delta spec alongside it
(the block condition is specifically "flat spec *without* domain specs"). The flat spec was left
in place as a planning artifact.
- **`web-ui` / `prometheus-charting` canonical isolation:** the existing
`openspec/specs/web-ui/spec.md` (MUI→shadcn rework) and `openspec/specs/prometheus-charting/spec.md`
(direct Prometheus charting) were **not modified** — verified untouched by `git status`. The three
domains are independent.
## 6. Validation / checks performed (file-backed, read-only)
Run from `/home/user/manage` (no source edits, no test re-runs — those are owned by verify and were
already green at `c9404f0`):
| Check | Command | Result |
|---|---|---|
| Canonical store populated | `ls openspec/specs/service-storage/spec.md` | present ✓ |
| Change-side domain spec present | `ls openspec/changes/service-storage-harness/specs/service-storage/spec.md` | present ✓ |
| Requirement-ID parity (flat ↔ delta ↔ canonical) | `grep -oE 'SS-[0-9]+'` all three files, `sort -u` | **28 == 28 == 28**, identical IDs SS-101…SS-128 ✓ |
| Body-text parity (delta ↔ canonical) | `diff` of the `^### Requirement:` region of both files | **identical** ✓ |
| Delta is pure ADDED | count `## ADDED/MODIFIED/REMOVED/RENAMED Requirements` | ADDED=1, MODIFIED=0, REMOVED=0, RENAMED=0 ✓ (no destructive sync) |
| Other canonicals untouched | `git status --porcelain openspec/specs/web-ui openspec/specs/prometheus-charting` | empty (not modified) ✓ |
| No edits outside openspec | `git status --porcelain` (filtered) | only `openspec/specs/service-storage/`, `openspec/changes/service-storage-harness/specs/`, and this report added ✓ |
| Markdown validity | write-time lint | all three files "Markdown clean" ✓ |
## 7. Carry-over items for the archive summary
These verify-phase findings are non-blocking for sync and should land in the archive summary:
1. **[CRITICAL-process, archive-only] Unchecked task checkboxes.** At verify time, 30
implementation/verification task checkboxes (§1.15.5) were unchecked and `apply-progress.md`
was missing. `apply-progress.md` now exists (created after the verify pass, reconciling all 35
tasks). `sdd-archive` should re-scan the native status engine to confirm `tasks: done` /
`applyProgress: present` before moving the change to archive, and tick any remaining unchecked
boxes if needed.
2. **[WARNING] Slice 2 over the 400-line review budget** (~644 source insertions vs the 400-line
budget / ~350400 forecast). Additive feature slice (3 widgets + `LineSeriesChart` extraction +
tests); boundary is exactly the qBit-widget feature, no scope creep. No `size:exception`
recorded; non-blocking — record the actual in the archive summary.
3. **[INFO] `LineSeriesChart.test.tsx` is smoke-only** (asserts mount, not rendered `<Line>` series).
Non-blocking coverage note.
4. **[INFO] Stale generated `.pi-map.md`** files predate the new modules (`service_data.py`,
`qbittorrent_store.py`, `clients/qbittorrent.py`, `Qbittorrent*.tsx`, `LineSeriesChart.tsx`);
not deliverable source. Regenerate via `project_map_patch` / `project_map_validate`.
## 8. Next recommended phase
**`sdd-archive`** (clean). Confirm the native status re-scan reports `specs: done` / `sync: ready`
/ `archive: ready`, then move the change to
`openspec/changes/archive/2026-07-09-service-storage-harness`, carrying over the items in §7 into
the archive summary. Do **not** commit or push — the parent owns the commit with explicit paths.
---
### Appendix — Files written by this sync (OpenSpec only; no source code)
- `openspec/changes/service-storage-harness/specs/service-storage/spec.md` — **change-side
domain delta (`## ADDED Requirements`), 28 requirements SS-101…SS-128.**
- `openspec/specs/service-storage/spec.md` — **canonical spec (new domain), 28 requirements.**
- `openspec/changes/service-storage-harness/sync-report.md` — this report.
@@ -0,0 +1,293 @@
# SDD Tasks: Service Storage Harness (qBittorrent widgets + MediaIndex migration)
**Change:** `service-storage-harness`
**Phase:** tasks
**Date:** 2026-07-09
## Review Workload Forecast
| Field | Value |
|-------|-------|
| Estimated changed lines | ~1,1801,380 (sum of four implementation slices) |
| 400-line budget risk | High |
| Chained PRs recommended | Yes |
| Suggested split | PR 1: Harness + QbittorrentSampleStore + client + integration → PR 2: Widget adapter + LineSeriesChart extract + 3 FE widgets → PR 3: MediaIndex migration → PR 4: Cascade-delete wiring |
| Delivery strategy | auto-chain |
| Chain strategy | stacked-to-main |
```text
Decision needed before apply: No
Chained PRs recommended: Yes
Chain strategy: stacked-to-main
400-line budget risk: High
```
> **Slice ordering rationale:** Slices 12 prove the harness and deliver all qBittorrent functionality independently of the MediaIndex migration. Slice 3 migrates the load-bearing MediaIndex feature. Slice 4 wires cascade-delete end-to-end. This ordering ensures the two biggest risks (new abstraction × load-bearing refactor) never fire in the same slice — the harness shape is settled in code before MediaIndex touches it.
---
## Slice 1: ServiceDataHarness + QbittorrentSampleStore + QbittorrentClient + integration registration
**Goal:** Build the lifecycle-only storage harness, prove it with the qBittorrent speed-sample store, add the qBittorrent HTTP client and integration registration. No widgets or frontend yet — this slice is backend-only and proves the harness contract with a real consumer.
- [x] **1.1 Create `ServiceDataHarness` lifecycle module**
- Files: `backend/src/media_library_viewer_api/services/service_data.py` (NEW)
- Lines: ~100
- Dependencies: none
- Details: `StorageConcern` dataclass (`concern_key`, `db_filename`, `migrations: list[str]`, `tables: list[str]`, `service_id_column="service_id"`). `ServiceDataHarness` class with `register(concern)`, `db_path(concern_key)`, `connect(concern_key)` (WAL + busy_timeout 30s), `run_migrations()` (per-concern, per-statement try/except for "duplicate column name"), `cascade_delete(service_id)` (iterate concerns, PRAGMA-check column exists, DELETE WHERE col = ?). Module singleton `get_service_data_harness()` that lazy-inits with `base_dir = BACKEND_CACHE_DIR or ".cache/media_library_viewer"` and calls `run_migrations()`. NO generic value table, NO generic CRUD.
- [x] **1.2 Create `QbittorrentSampleStore` with concern registration**
- Files: `backend/src/media_library_viewer_api/services/qbittorrent_store.py` (NEW)
- Lines: ~80
- Dependencies: 1.1
- Details: Define `QBITTORRENT_CONCERN` (`db_filename="qbittorrent.db"`, migration creates `qbittorrent_speed_samples(service_id, ts, dl_speed, up_speed)` + index on `(service_id, ts)`, `tables=["qbittorrent_speed_samples"]`). `MAX_SAMPLES = 120`. `QbittorrentSampleStore` class: `append(service_id, ts, dl_speed, up_speed)` (INSERT + prune keeping most recent MAX_SAMPLES for this service_id), `window(service_id, since_ts=None)` (SELECT ordered ASC). Register `QBITTORRENT_CONCERN` with harness on import.
- [x] **1.3 Create `QbittorrentClient` HTTP client**
- Files: `backend/src/media_library_viewer_api/clients/qbittorrent.py` (NEW)
- Lines: ~90
- Dependencies: none
- Details: `QbittorrentClient(base_url, username, password, timeout=10)` modeled on `JellyfinClient`'s session pattern. `_login()` POSTs to `/auth/login` with `Referer` header, expects `"Ok."` response, stores SID cookie. `_get(path, **params)` auto-logins on first call, re-logins on 403. `maindata()` calls `/sync/maindata` returning `{server_state: {...}, torrents: {hash: {...}}}`. Base URL normalization: append `/api/v2` if not present. Timeout + `requests.RequestException` handling consistent with existing clients.
- [x] **1.4 Create qBittorrent integration definition**
- Files: `backend/src/media_library_viewer_api/integrations/qbittorrent.py` (NEW)
- Lines: ~55
- Dependencies: 1.1 (for `ServiceConfigBase`, `WidgetConfigBase`, `SecretField`, `ServiceDefinition`, `widget_kind` from `base.py`)
- Details: `QbittorrentConfig(ServiceConfigBase)` with `base_url: ServiceBaseUrl`, `timeout_seconds: int = 10`. `QbittorrentWidgetConfig(WidgetConfigBase)` empty (all three widget kinds derive from service connection). `DEFINITION = ServiceDefinition(service_type="qbittorrent", ...)` with `secret_fields=[username (required), password (required)]` and three `widget_kinds`: `totals` (30s refresh), `active` (15s), `speed` (5s). Model on `integrations/prometheus.py`.
- [x] **1.5 Register qBittorrent in backend integration registry**
- Files: `backend/src/media_library_viewer_api/integrations/registry.py` (MODIFY)
- Lines: ~5
- Dependencies: 1.4
- Details: Import `DEFINITION as QBITTORRENT` from `integrations/qbittorrent.py`; add `SERVICE_DEFINITIONS["qbittorrent"] = QBITTORRENT`. Verify `list_service_types()` now includes `qbittorrent`.
- [x] **1.6 Initialize harness in application lifespan**
- Files: `backend/src/media_library_viewer_api/main.py` (MODIFY, lifespan function ~line 41)
- Lines: ~5
- Dependencies: 1.1, 1.2
- Details: In `lifespan()` after `get_settings_store().ensure_defaults()`, add a try/except block calling `get_service_data_harness()` (triggers lazy init + migration run). Log on failure (same pattern as the settings seed block at lines 5154). Import is local inside lifespan to avoid circular import risk.
- [x] **1.7 Create backend tests for harness + store**
- Files: `backend/tests/test_service_data.py` (NEW)
- Lines: ~110
- Dependencies: 1.1, 1.2
- Details: Test `ServiceDataHarness`: register concern → `run_migrations()` creates table; `cascade_delete(service_id)` removes rows by service_id but leaves other services' rows; migration idempotency (re-run doesn't crash). Test `QbittorrentSampleStore`: `append` + prune (append 130 samples, assert only 120 remain), `window` returns ordered samples, two services don't cross-contaminate. Use `tmp_path` for base_dir isolation.
- [x] **1.8 Create backend tests for qBittorrent client**
- Files: `backend/tests/test_qbittorrent_client.py` (NEW)
- Lines: ~85
- Dependencies: 1.3
- Details: Test login flow (mock `requests.Session.post``"Ok."` + cookie), cookie reuse on subsequent calls, 403 → re-login flow (first GET returns 403, second returns 200 after re-login), `maindata()` parsing, base URL normalization (appends `/api/v2`), login failure (response ≠ "Ok." raises RuntimeError), timeout handling. Use `unittest.mock.patch` on the session.
- [x] **1.9 Verify Slice 1 (backend only)**
- Run: `cd backend && PYTHONPATH=src ruff check src tests && PYTHONPATH=src python -m pytest tests/test_service_data.py tests/test_qbittorrent_client.py -v`
- Verify: all new tests pass; existing tests unaffected.
**Slice 1 total:** ~310390 changed lines. Backend-only; no frontend changes.
---
## Slice 2: Widget source adapter + LineSeriesChart extraction + frontend widgets + registry binding
**Goal:** Wire qBittorrent data into the widget system, extract a shared chart renderer, and build the three frontend widgets. This slice touches `PrometheusChartWidget` (extraction) — its existing tests MUST stay green.
- [x] **2.1 Add `QbittorrentWidgetSource` adapter to `widgets/sources.py`**
- Files: `backend/src/media_library_viewer_api/widgets/sources.py` (MODIFY)
- Lines: ~75
- Dependencies: Slice 1 (1.3, 1.2)
- Details: Add `QbittorrentWidgetSource` implementing `WidgetSource.fetch(service, widget_kind, config)`. Resolves client inline from `ServiceRecord` config+secrets (like `PrometheusWidgetSource`). Three branches:
- `totals`: `len(torrents)` + `by_state` dict breakdown. Returns `{total, by_state}`.
- `active`: filter `state in {"downloading","uploading"}`. Returns `{torrents: [{name, state, size, progress, dl_speed, up_speed}]}`.
- `speed`: append sample to `QbittorrentSampleStore(service.id, ts, dl, up)`, read `window(service.id)`, return `{series: [{label:"download", points:[{t: ts*1000, v: dl}]}, {label:"upload", ...}]}` (timestamps ×1000 for JS epoch).
All branches wrap in try/except → `{"error": ...}`. Add `"qbittorrent": QbittorrentWidgetSource()` to `SERVICE_ADAPTERS`.
- [x] **2.2 Add backend tests for qBittorrent widget adapter**
- Files: `backend/tests/test_widgets.py` (EXTEND)
- Lines: ~90
- Dependencies: 2.1
- Details: Test `QbittorrentWidgetSource` with mocked `QbittorrentClient.maindata()`: totals counts all torrents + breaks down by state; active filters to DL/UL only (assert queued/stalled excluded); speed appends sample and returns `{series}` shape with two labeled series; missing service → `{"error": ...}`; timeout → `{"error": ...}`. Use `tmp_path` harness override for the store.
- [x] **2.3 Extract shared `LineSeriesChart` component from `PrometheusChartWidget`**
- Files: `frontend/src/components/LineSeriesChart.tsx` (NEW), `frontend/src/widgets/PrometheusChartWidget.tsx` (MODIFY)
- Lines: ~75 (new file) + ~20 (PrometheusChartWidget thin wrapper)
- Dependencies: none (refactor of existing code)
- Details: **RISK ITEM (a).** Extract `mergeSeries`, `formatTime`, `CHART_COLORS`, `ChartSeries`/`SeriesPoint` types, and the `<ResponsiveContainer>+<LineChart>+<XAxis>+<YAxis>+<CartesianGrid>+<Tooltip>+<Line>` JSX from `PrometheusChartWidget.tsx` into `components/LineSeriesChart.tsx`. Props: `{ series: ChartSeries[], height?: number }`. `PrometheusChartWidget` becomes: `useWidgetData``<LineSeriesChart series={data.data.series} />`. Export `ChartSeries` type from `LineSeriesChart` for reuse. **PrometheusChartWidget's existing tests MUST stay green — the extraction is mechanical (move + import), not a rewrite.**
- [x] **2.4 Create `QbittorrentTotalsWidget` component**
- Files: `frontend/src/widgets/QbittorrentTotalsWidget.tsx` (NEW)
- Lines: ~60
- Dependencies: 2.3
- Details: Renders `{total, by_state}` from `useWidgetData`. Uses `SectionCard` + prominent total count + state breakdown badges (e.g. `downloading: 3`, `uploading: 1`, `paused: 5`). Loading skeleton + error alert + empty state. Model on `BackupsWidget`/`MetricCard` patterns.
- [x] **2.5 Create `QbittorrentActiveTorrentsWidget` component**
- Files: `frontend/src/widgets/QbittorrentActiveTorrentsWidget.tsx` (NEW)
- Lines: ~80
- Dependencies: 2.3
- Details: Renders `{torrents: [...]}` from `useWidgetData`. Uses `SectionCard` + compact table/list (manual `Table` rows, NOT DataTable — keep it simple for ≤10 active torrents). Columns: name, state badge, progress bar, dl/up speed. Scrollable container for overflow. Loading + error + empty ("No active torrents") states. Reuses `humanSize`-style formatting for speeds (bytes/s → MB/s).
- [x] **2.6 Create `QbittorrentSpeedWidget` component**
- Files: `frontend/src/widgets/QbittorrentSpeedWidget.tsx` (NEW)
- Lines: ~40
- Dependencies: 2.3
- Details: Renders `{series}` from `useWidgetData` — IDENTICAL pattern to `PrometheusChartWidget`. Thin wrapper: `useWidgetData``<LineSeriesChart series={data.data.series} height={220} />`. Loading skeleton + error alert + empty state ("No speed data yet"). Refresh interval from registry binding (5s).
- [x] **2.7 Add qBittorrent binding to frontend registry**
- Files: `frontend/src/integrations/registry.ts` (MODIFY)
- Lines: ~40
- Dependencies: 2.4, 2.5, 2.6
- Details: Import the three new widget components. Add `qbittorrent` entry to `SERVICE_REGISTRY` with three widget kinds: `totals` (30s, `QbittorrentTotalsWidget`), `active` (15s, `QbittorrentActiveTorrentsWidget`), `speed` (5s, `QbittorrentSpeedWidget`). All with empty config schemas (`{type:"object", properties:{}, required:[]}`). Update barrel `widgets/index.ts` with the three new exports.
- [x] **2.8 Create frontend tests for new widgets + LineSeriesChart**
- Files: `frontend/src/widgets/__tests__/QbittorrentTotalsWidget.test.tsx` (NEW), `frontend/src/widgets/__tests__/QbittorrentActiveTorrentsWidget.test.tsx` (NEW), `frontend/src/widgets/__tests__/QbittorrentSpeedWidget.test.tsx` (NEW), `frontend/src/components/__tests__/LineSeriesChart.test.tsx` (NEW)
- Lines: ~110 (4 files × ~25-30 lines)
- Dependencies: 2.3, 2.4, 2.5, 2.6
- Details: Each widget test: loading skeleton (`data-slot="skeleton"`), error alert (destructive variant), rendered data case. `LineSeriesChart.test.tsx`: renders lines from series data, empty state (no series). Model on existing `PrometheusChartWidget.test.tsx` mock pattern (`vi.mocked(useWidgetData).mockReturnValue(...)`). **Verify `PrometheusChartWidget.test.tsx` still passes after extraction.**
- [x] **2.9 Verify Slice 2 (full stack)**
- Run: `cd backend && PYTHONPATH=src ruff check src tests && PYTHONPATH=src python -m pytest tests/test_widgets.py -v`
- Run: `cd frontend && npm run lint && npm run build && npx vitest run src/widgets/__tests__/Qbittorrent src/widgets/__tests__/PrometheusChart src/components/__tests__/LineSeriesChart`
- Verify: PrometheusChartWidget tests GREEN (extraction not a regression); qBit widget tests GREEN; build + lint GREEN.
**Slice 2 total:** ~350400 changed lines.
---
## Slice 3: MediaIndex migration onto harness (load-bearing refactor)
**Goal:** Register MediaIndex as a harness concern, add `service_id` scoping to the schema, fix the latent global-clear bug in `replace_items`, and thread `service_id` through query/build/worker. All existing MediaIndex tests MUST pass unchanged.
- [x] **3.1 Define `MEDIA_INDEX_CONCERN` and register with harness**
- Files: `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY — add concern constant near top), `backend/src/media_library_viewer_api/services/service_data.py` (MODIFY — register concern in `get_service_data_harness`)
- Lines: ~25
- Dependencies: Slice 1 (1.1)
- Details: `MEDIA_INDEX_CONCERN = StorageConcern(concern_key="media_index", db_filename="media_index.sqlite", migrations=["ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT ''"], tables=["media_items"])`. Register in harness singleton alongside `QBITTORRENT_CONCERN`. The `db_filename` MUST match the existing `DEFAULT_INDEX_PATH` leaf name (`media_index.sqlite`). The harness `base_dir` (`.cache/media_library_viewer`) is the same parent — no file move.
- [x] **3.2 Add `service_id` column to `init_schema` CREATE TABLE**
- Files: `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY, `init_schema` ~line 107)
- Lines: ~3
- Dependencies: 3.1
- Details: Add `service_id TEXT NOT NULL DEFAULT ''` as the last column in the `CREATE TABLE IF NOT EXISTS media_items` statement. New installs get the column from `init_schema`; existing DBs get it from the harness ALTER migration. Both paths converge on the same schema.
- [x] **3.3 Scope `replace_items` by `service_id` (FIXES latent global-clear bug)**
- Files: `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY, `replace_items` ~line 166)
- Lines: ~20
- Dependencies: 3.2
- Details: **RISK ITEM (b).** Add `service_id: str = ""` parameter. Add `"service_id"` to the columns list. Change `DELETE FROM media_items``DELETE FROM media_items WHERE service_id = ?` with `(service_id,)`. Change the `executemany` rows to prepend `service_id` to each row's values. This FIXES the latent bug where building for one Jellyfin wipes another's rows. Existing tests call `replace_items(rows)` with no `service_id` → defaults to `""` → deletes `WHERE service_id = ''` → backward-compatible (existing test data has `service_id=''` from backfill).
- [x] **3.4 Scope `query` by `service_id`**
- Files: `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY, `query` ~line 279)
- Lines: ~10
- Dependencies: 3.2
- Details: Add `service_id: str = ""` parameter. After existing WHERE clause building, add: `if service_id: where.append("service_id = ?"); params.append(service_id)`. When `service_id=""` (empty), skip the filter → shows all rows (backward-compatible). This means existing tests (which pass no `service_id`) see all rows unchanged.
- [x] **3.5 Thread `service_id` through `build_media_index``replace_items`**
- Files: `backend/src/media_library_viewer_api/services/media_index_impl.py` (MODIFY, `build_media_index` ~line 331)
- Lines: ~5
- Dependencies: 3.3
- Details: Add `service_id: str = ""` parameter to `build_media_index(...)`. Change the `count = index.replace_items(normalized_rows)` call (~line 458) to `count = index.replace_items(normalized_rows, service_id=service_id)`.
- [x] **3.6 Pass `service_id` from worker to `build_media_index`**
- Files: `backend/src/media_library_viewer_api/workers/media_index_worker.py` (MODIFY, `run_build` ~line 144, `build_media_index` call ~line 168)
- Lines: ~3
- Dependencies: 3.5
- Details: The worker's `run_build(final_index_path, staging_index_path, service_id="")` ALREADY receives `service_id` from argparse (`--service-id`). Change the `build_media_index(...)` call (~line 168) to pass `service_id=service_id`.
- [x] **3.7 Add `jellyfin_service_id` to `query_media` and pass to `query`**
- Files: `backend/src/media_library_viewer_api/routers/media.py` (MODIFY, `query_media` ~line 276)
- Lines: ~5
- Dependencies: 3.4
- Details: `query_media` does NOT currently have a `jellyfin_service_id` param (unlike `post_build_index` which does). Add `jellyfin_service_id: str | None = None` to the signature (after `offset`). Pass `service_id=jellyfin_service_id or ""` to the `index.query(...)` call (~line 310). Existing callers that don't send the param → `None``""` → all rows (backward-compatible).
- [x] **3.8 Add backend tests for scoped MediaIndex operations**
- Files: `backend/tests/test_media_index.py` (EXTEND)
- Lines: ~45
- Dependencies: 3.3, 3.4
- Details: Add tests for the NEW scoped behavior: `replace_items(rows, service_id="svc-a")` then `replace_items(other_rows, service_id="svc-b")` — assert svc-a rows survive svc-b's replace (proves the bug fix). `query(service_id="svc-a")` returns only svc-a rows. `query(service_id="")` returns all rows. `query()` (no arg) returns all rows. **Existing MediaIndex tests MUST pass unchanged** — verify they don't break by running the full file.
- [x] **3.9 Verify Slice 3 (backend, load-bearing)**
- Run: `cd backend && PYTHONPATH=src ruff check src tests && PYTHONPATH=src python -m pytest tests/test_media_index.py -v`
- Run: `cd backend && PYTHONPATH=src python -m pytest -q` (full suite — no regressions)
- Verify: ALL existing MediaIndex tests pass unchanged; new scoped tests pass; full suite green.
**Slice 3 total:** ~200260 changed lines. Backend-only; Media page UX unchanged.
---
## Slice 4: Cascade-delete wiring end-to-end + integration test
**Goal:** Hook `ServiceDataHarness.cascade_delete` into `SettingsStore.delete_service` so removing a service wipes its owned data across all harness-managed tables. End-to-end integration test.
- [x] **4.1 Wire harness cascade-delete into `delete_service`**
- Files: `backend/src/media_library_viewer_api/services/settings_store.py` (MODIFY, `delete_service` ~line 1742)
- Lines: ~10
- Dependencies: Slice 1 (1.1), Slice 3 (3.1)
- Details: After the existing `DELETE FROM services WHERE id = ?` (line 1754), add a try/except block: `from media_library_viewer_api.services.service_data import get_service_data_harness; get_service_data_harness().cascade_delete(service_id)`. Log on failure (`logger.exception(...)`) — cascade is best-effort (service row is already deleted; data cleanup must not block deletion). Local import to avoid circular dependency.
- [x] **4.2 Add integration test for cascade-delete**
- Files: `backend/tests/test_services.py` (EXTEND)
- Lines: ~45
- Dependencies: 4.1
- Details: Test end-to-end cascade: (1) create a qBittorrent service, append speed samples for it, delete the service → assert samples gone from `qbittorrent.db`. (2) Create two Jellyfin services, build media for both with distinct `service_id`s, delete one → assert only that service's media rows removed, other survives. Use `tmp_path` for both settings DB and harness base_dir. Verify `delete_service` doesn't raise even if harness fails (best-effort guard).
- [x] **4.3 Verify Slice 4 (full suite)**
- Run: `cd backend && PYTHONPATH=src ruff check src tests && PYTHONPATH=src python -m pytest -q`
- Run: `cd frontend && npm run lint && npm run build`
- Verify: full backend suite green (293+ tests + new); frontend unaffected (no FE changes in this slice).
**Slice 4 total:** ~80110 changed lines. Backend-only.
---
## Integration and acceptance verification
- [x] **5.1 Full backend test run**
- Run: `cd backend && PYTHONPATH=src python -m pytest -q`
- Verify: all existing tests pass + all new tests (harness, store, client, widget adapter, scoped MediaIndex, cascade-delete) pass.
- [x] **5.2 Full frontend build + lint**
- Run: `cd frontend && npm run lint && npm run build`
- Verify: no TypeScript errors; no new lint failures; PrometheusChartWidget tests still green after LineSeriesChart extraction.
- [x] **5.3 Verify qBittorrent end-to-end (manual or integration)**
- Verify: a qBittorrent service can be registered; three widget kinds are bindable; totals shows item count; active shows DL/UL torrents; speed shows a live recharts line chart.
- [x] **5.4 Verify MediaIndex backward-compat**
- Verify: existing Media page works identically (no `service_id` → sees all rows); building for a specific Jellyfin service scopes correctly.
- [x] **5.5 Verify cascade-delete**
- Verify: deleting a service removes its qBit samples + media items; deleting one of multiple Jellyfin services removes only its rows.
---
## Total estimate
| Slice | Changed lines | Risk |
|-------|---------------|------|
| Slice 1: Harness + store + client + integration | ~310390 | Low (new code, no existing behavior touched) |
| Slice 2: Widget adapter + LineSeriesChart + FE widgets | ~350400 | Medium (touches PrometheusChartWidget) |
| Slice 3: MediaIndex migration | ~200260 | **High** (load-bearing; must fix replace_items bug + keep tests green) |
| Slice 4: Cascade-delete wiring | ~80110 | Low (small, well-isolated) |
| Integration tests | ~45 | Low |
| **Total** | **~9851,205** | |
---
## Guard lines
```text
Decision needed before apply: No
Chained PRs recommended: Yes
Chain strategy: stacked-to-main
400-line budget risk: High
```
---
## Risk callouts (for reviewer + apply agents)
| # | Risk | Slice | Mitigation |
|---|------|-------|------------|
| **R1** | `LineSeriesChart` extraction breaks `PrometheusChartWidget` tests | S2 (2.3, 2.8) | Extraction is mechanical (move JSX + helpers, pass series as prop). S2 exit gate explicitly verifies `PrometheusChartWidget.test.tsx` still passes. |
| **R2** | MediaIndex migration breaks existing tests or introduces a regression | S3 (3.3, 3.8) | `service_id` defaults to `""`; empty-string queries skip the filter → all rows visible. `replace_items` bug fix changes `DELETE FROM media_items``DELETE WHERE service_id = ?` with `""` default → existing tests (which use `""`) are unaffected. S3 exit gate runs full backend suite. |
| **R3** | ALTER TABLE fails on fresh installs where `init_schema` already created the column | S3 (3.1) | `run_migrations` catches "duplicate column name" per-statement (designed in Slice 1). |
| **R4** | Harness lazy-init hides migration failures on startup | S1 (1.6) | `run_migrations` runs on first access in lifespan; failures raise (only "duplicate column name" is swallowed). |
| **R5** | Circular import between `settings_store` and `service_data` | S4 (4.1) | Cascade-delete uses local import inside `delete_service` method, not module-level. |
@@ -0,0 +1,353 @@
# Verify Report — service-storage-harness
> Phase: **verify** · Change: `service-storage-harness` · Repo: `/home/user/manage`
> FRESH-CONTEXT adversarial read-only verification of the change against
> `proposal.md`, `spec.md`, `design.md`, and `tasks.md`. **No source edits.**
> This verify report is the only file written.
**Head commit verified:** `c9404f0` (`spec(service-storage-harness): add spec`).
Four implementation slices are committed underneath it:
- `e7bd0af` slice 1 — harness + qbit store + client + integration
- `1fb12b8` slice 2 — qbit widgets + LineSeriesChart extract *(brief cited `8b0e7ea`,
an earlier amend state; the actual landed commit is `1fb12b8`. Content matches
the spec/design/tasks; informational, not a defect.)*
- `c87f398` slice 3 — migrate MediaIndex onto harness (scoped replace_items, +service_id)
- `75c949a` slice 4 — cascade-delete wiring + integration test
---
## 0. Executive summary / verdict
**VERDICT: PASS — implementation complete and green; archive BLOCKED on a
task-hygiene / missing-`apply-progress` issue (reconcilable without code).**
Every functional requirement **SS-101 … SS-128** was checked against source and
**passes**. The `ServiceDataHarness` is genuinely lifecycle-only (provisioning,
idempotent migrations, `service_id` cascade-delete) with **no** generic value
table or CRUD. The qBittorrent stack (store + cookie-reusing client + 3-branch
adapter + 3 frontend widgets) is wired consistently in **both** registries with
matching kinds/names/refresh intervals. The **load-bearing** MediaIndex migration
(SS-119..SS-124) is correct: `replace_items` is scoped `WHERE service_id = ?`
(fixing the latent global-clear bug), with a real regression test proving
service A's rows survive service B's rebuild; existing MediaIndex tests pass
unchanged via the `service_id=""` default. The `LineSeriesChart` extraction is
non-regressive (PrometheusChartWidget tests stay green; it becomes a thin
wrapper). Cascade-delete is wired into `delete_service` (best-effort try/except)
and initialized in `lifespan`, with an end-to-end integration test covering
**both** concerns with multi-instance preservation. All four gates are green:
backend `pytest` (**322 passed**), `ruff` (**clean**), frontend
`npm run build` (**exit 0**), `npm run lint` (**0 errors**).
Findings:
- **[CRITICAL — archive blocker, NOT a code defect]** **All 30
implementation/verification task checkboxes** remain unchecked in `tasks.md`
(§1.11.9, §2.12.9, §3.13.9, §4.14.3, §5.15.5), and **`apply-progress.md`
does not exist** to reconcile them. The underlying work *is* done and verified
complete against source; the blocker is that the task tracker was never updated
and no apply-progress artifact was produced. Reconciliation = tick the boxes +
write `apply-progress.md` (no code change). See §4.
- **[WARNING — review workload]** Slice 2 (`1fb12b8`) lands **~644 non-test
source lines** (widget adapter + `LineSeriesChart` extraction + 3 FE widgets),
above the 400-line budget and above the slice-2 forecast of "~350400". The
boundary is exactly the qBit-widget feature (no unrelated files, no scope
creep), and the forecast itself rated slice 2 "Medium"; but no `size:exception`
was recorded. Recommend recording the slice-2 actual in the archive summary.
**Non-blocking.** See §6.
- **[INFO]** `LineSeriesChart.test.tsx` is smoke-only (asserts `container.firstChild`
is not null; does not assert recharts `<Line>` SVGs rendered). Same weak-render
pattern noted in the Change A report. The PrometheusChartWidget rendered case
is title-only. Non-blocking coverage note.
- **[INFO]** `QbittorrentSampleStore.append` prunes via
`ts NOT IN (SELECT ts … LIMIT 120)`; in the degenerate case of two samples
sharing an identical `ts`, the keep-set could exceed `MAX_SAMPLES`. At a 5 s
poll the probability is effectively nil; the per-service cap invariant holds in
all realistic operation. Non-blocking.
- **[INFO]** Stale generated `.pi-map.md` files predate these new files
(`service_data.py`, `qbittorrent_store.py`, `clients/qbittorrent.py`,
`Qbittorrent*.tsx`, `LineSeriesChart.tsx`, etc.). Not deliverable source;
regenerate via `project_map_patch`/`project_map_validate`.
---
## 1. Structured status & actionContext findings
The native `gentle-pi.sdd-status` reports `changeName: null` /
`blockedReasons: ["Change selection is ambiguous: mobile-responsive-parity,
service-storage-harness, services-as-hub-ia"]` because the engine auto-detected
three active changes. This verify task was **explicitly assigned**
`service-storage-harness`; the ambiguity is a parent-resolution artifact and does
not block this phase.
- `artifactStore: openspec`; change root
`openspec/changes/service-storage-harness/`.
- Artifacts present: `proposal.md`, `design.md`, `spec.md`, `tasks.md`.
- **`apply-progress.md`: MISSING** (confirmed: directory contains only the four
planning docs). This is the root cause of the §4 archive blocker.
- `actionContext`: `mode: repo-local`, `workspaceRoot: /home/user/manage`,
`allowedEditRoots: ["/home/user/manage"]`, `warnings: []`. Implementation
ownership and all target files are provably inside the authoritative workspace. ✓
## 2. Gate results (actual output, run at `c9404f0`)
| Gate | Command | Result | Evidence |
|------|---------|--------|----------|
| Backend tests | `cd backend && PYTHONPATH=src python3 -m pytest -q` | **PASS** | **322 passed, 2 warnings** in 39.36s. Includes new `test_service_data.py` (9), `test_qbittorrent_client.py` (12), qBit adapter tests (6) in `test_widgets.py`, scoped MediaIndex regression (`test_replace_scoped_by_service_id_preserves_other_services`), and cascade integration (`test_cascade_delete_removes_harness_data_across_concerns`). |
| Backend lint | `cd backend && PYTHONPATH=src python3 -m ruff check src tests` | **PASS** | `All checks passed!` |
| Frontend build | `cd frontend && npm run build` (`tsc -b` + `vite build`) | **PASS** exit 0 | `✓ built in 1.22s`; 2543 modules transformed. Non-fatal `>500 kB` chunk-size warning (pre-existing, orthogonal). |
| Frontend lint | `cd frontend && npm run lint` (`eslint .`) | **PASS** exit 0 | `0 errors, 1 warning`. The warning is `react-hooks/exhaustive-deps` in `WidgetConfigDialog.tsx:370`**pre-existing, untouched by this change** (no slice modified that file). |
| Extraction non-regression | `cd frontend && npx vitest run src/widgets/__tests__/PrometheusChartWidget.test.tsx` | **PASS** | **4 passed** (SS-118: LineSeriesChart extraction did not regress PrometheusChartWidget). |
| New FE widget + chart tests | `npx vitest run src/widgets/__tests__/ src/components/__tests__/LineSeriesChart.test.tsx` | **PASS** | **8 files, 31 tests passed** (3 qBit widgets + PrometheusChart/Gauge/Mean + LineSeriesChart). |
## 3. Spec coverage (SS-101 … SS-128)
| SS | Requirement | Verdict | Evidence |
|----|-------------|---------|----------|
| SS-101 | Harness is lifecycle-only | **PASS** | `service_data.py::ServiceDataHarness` exposes only `register`/`db_path`/`connect`/`run_migrations`/`cascade_delete`. No generic value table, no generic CRUD — each integration keeps bespoke stores (`append/window`, `replace_items/query`). Module docstring states the contract explicitly. |
| SS-102 | Concern registration dataclass | **PASS** | `StorageConcern` frozen dataclass carries `concern_key`, `db_filename`, `migrations: list[str]`, `tables: list[str]`, `service_id_column="service_id"`. `register()` stores by `concern_key`. |
| SS-103 | Idempotent migrations | **PASS** | `run_migrations` splits each migration on `;` and executes per-statement; catches `sqlite3.OperationalError` for "duplicate column name" **and** "no such table", logging at debug and continuing. Test `test_cascade_delete_*` + harness tests re-init without raising. |
| SS-104 | Cascade-delete across concerns | **PASS** | `cascade_delete(service_id)` iterates **every** registered concern, PRAGMA-checks the column exists, `DELETE FROM <table> WHERE <col> = ?`. Both `QBITTORRENT_CONCERN` and `MEDIA_INDEX_CONCERN` are registered in `get_service_data_harness()`, so it covers `qbittorrent_speed_samples` + `media_items`. Skips concern DBs that don't exist (no crash). |
| SS-105 | Sample store schema | **PASS** | `qbittorrent_store.py` migration creates `qbittorrent_speed_samples(service_id, ts, dl_speed, up_speed)` + index `idx_qbit_samples_service_ts ON (service_id, ts)`, in `qbittorrent.db` (per-concern topology via `db_filename`). |
| SS-106 | append/window/prune | **PASS** | `append(service_id, ts, dl, up)` INSERTs then prunes to `MAX_SAMPLES=120` (DELETE `ts NOT IN (top-120 for this service_id)`). `window(service_id, since_ts=None)` SELECTs ordered ASC. Test appends 130 → asserts 120 remain; two services don't cross-contaminate. |
| SS-107 | Store registered as concern | **PASS** | `QBITTORRENT_CONCERN` registered in `get_service_data_harness()` alongside the media concern. |
| SS-108 | Cookie login | **PASS** | `QbittorrentClient._login()` POSTs `/auth/login` (with `Referer` header), expects `"Ok."`, stores SID cookie in the shared `requests.Session`. Credentials resolved from `ServiceRecord.secrets` (Fernet-encrypted at rest) in the adapter. Test `test_login_posts_credentials`. |
| SS-109 | 403 re-login | **PASS** | `_get` retries once after 403: sets `_logged_in=False`, re-`_login()`, re-GETs. Test `test_403_triggers_re_login` asserts `session.get` called twice + `session.post` once. |
| SS-110 | maindata fetch | **PASS** | `maindata()``_get("/sync/maindata")` returns the dict (`server_state` + `torrents`). Timeout/connection/non-2xx propagate as exceptions (`raise_for_status`); the adapter catches them (SS-115). Test `test_maindata_returns_full_payload`. |
| SS-111 | Three widget kinds dispatched | **PASS** | `QbittorrentWidgetSource.fetch` resolves `QbittorrentClient` inline from `ServiceRecord` config+secrets, dispatches on `widget_kind ∈ {totals, active, speed}`. Registered in `SERVICE_ADAPTERS["qbittorrent"]`. |
| SS-112 | totals = item count | **PASS** | `totals``{"total": len(torrents), "by_state": {...}}`. Count of currently-listed torrents, **not** cumulative bytes. Test `test_qbittorrent_totals_counts_all_torrents`. |
| SS-113 | active = DL/UL filter | **PASS** | `active` keeps only `state in {"downloading","uploading"}``{"torrents": [...]}`. Test `test_qbittorrent_active_filters_dl_ul_only` asserts queued/stalled excluded. |
| SS-114 | speed appends + returns series | **PASS** | `speed` reads `dl_info_speed`/`up_info_speed`, `store.append(service.id, ts, dl, up)`, returns `{"series": [{label:"download", points:[...]}, {label:"upload", points:[...]}]}` from `.window()` with `t = ts * 1000` (JS ms). Test `test_qbittorrent_speed_appends_and_returns_series` asserts labels + last download value. |
| SS-115 | Errors degrade gracefully | **PASS** | `fetch` body wrapped in `try/except`: `asyncio.TimeoutError → {"error": "...timed out"}`, generic `Exception → {"error": "...failed: ..."}`. Missing service / missing creds return `{"error":...}` before any network call. Tests cover missing-service, missing-credentials, timeout. Never raises. |
| SS-116 | Three FE widget components + binding | **PASS** | `QbittorrentTotalsWidget`, `QbittorrentActiveTorrentsWidget`, `QbittorrentSpeedWidget` exist under `frontend/src/widgets/`; `registry.ts` binds them to `qbittorrent` with kinds `totals`/`active`/`speed`. |
| SS-117 | Speed widget reuses shared renderer | **PASS** | `QbittorrentSpeedWidget``<LineSeriesChart series={series} height={220} />`. No new charting code, no new charting dependency. |
| SS-118 | LineSeriesChart extraction non-regressive | **PASS** | `LineSeriesChart.tsx` owns `mergeSeries`/`formatTime`/`CHART_COLORS`/`ChartSeries`/recharts JSX; `PrometheusChartWidget.tsx` is a thin wrapper (`useWidgetData``<LineSeriesChart series={series} />`). `PrometheusChartWidget.test.tsx`**4 passed**. |
| SS-119 | service_id column added (idempotent) | **PASS** | `MEDIA_INDEX_CONCERN.migrations = ["ALTER TABLE media_items ADD COLUMN service_id TEXT NOT NULL DEFAULT ''"]`; `init_schema` CREATE TABLE also includes it. `db_filename="media_index.sqlite"` == `DEFAULT_INDEX_PATH.name`; harness `base_dir` == `.cache/media_library_viewer`**file location unchanged**. |
| SS-120 | Existing rows backfilled | **PASS** | Column `DEFAULT ''` backfills all pre-existing rows to `service_id=""` (preserving visibility). |
| SS-121 | replace_items scoped (bug fix) | **PASS** | `replace_items(rows, service_id="")` does `DELETE FROM media_items WHERE service_id = ?` (NOT global). **Regression test `test_replace_scoped_by_service_id_preserves_other_services`** inserts svc-a (2 rows), replaces svc-b (1 row), asserts `item_count==3` (svc-a survives), and scoped queries return only each service's rows. Latent global-clear bug confirmed fixed. |
| SS-122 | query backward-compatible | **PASS** | `query(..., service_id="")`: non-empty appends `WHERE service_id = ?`; empty (default) skips the filter → all rows. Existing `test_media_index.py` / `test_api.py` callers pass no `service_id` and run **unchanged** (322 passed). |
| SS-123 | Worker threads service_id | **PASS** | `run_build(..., service_id="")` receives `--service-id` from argparse and passes `service_id=service_id` into `build_media_index(...)`, which calls `index.replace_items(normalized_rows, service_id=service_id)` (line 488). Newly-built rows stamped with the real service_id. |
| SS-124 | MediaIndex registered as concern | **PASS** | `MEDIA_INDEX_CONCERN` (tables=`["media_items"]`) registered in `get_service_data_harness()`, so `media_items` participates in cascade-delete. |
| SS-125 | delete_service triggers cascade | **PASS** | `settings_store.delete_service` (after `DELETE FROM services WHERE id=?`) does `try: get_service_data_harness().cascade_delete(service_id); except Exception: logger.exception(...)`**best-effort, local import** to avoid circular dependency. A cascade failure does not crash deletion. |
| SS-126 | End-to-end cascade across both concerns | **PASS** | `test_cascade_delete_removes_harness_data_across_concerns`: qBit A+B samples appended, delete A → A gone, B survives; media items A+B built with distinct service_ids, delete A → A rows gone (`total_a_after==0`), B survives (`total_b_after==1`). Both concerns + multi-instance preservation proven. |
| SS-127 | Backend tests + lint green | **PASS** | `pytest`**322 passed**; `ruff check src tests``All checks passed!`. New tests present for harness lifecycle, qBit store, qBit client (login/403/cookie-reuse/maindata), qBit adapter (3 branches + errors), scoped MediaIndex regression, cascade-delete integration. |
| SS-128 | Frontend build + lint green | **PASS** | `npm run build` exit 0; `npm run lint` 0 errors (1 pre-existing warning). New widget tests cover **loading/error/rendered** states (Totals includes an explicit `isLoading:true` skeleton case — improving on Change A's SC-125 gap). PrometheusChartWidget tests green (SS-118). |
**Functional spec coverage: 28/28 fully PASS.**
---
## 4. Task completion status — ⚠ archive blocker (reconcilable)
`tasks.md` checkbox state (via `grep -nE '^\s*- \[' tasks.md`):
- **Slice 1 (§1.11.9): all 9 `[ ]` — UNCHECKED**
- **Slice 2 (§2.12.9): all 9 `[ ]` — UNCHECKED**
- **Slice 3 (§3.13.9): all 9 `[ ]` — UNCHECKED**
- **Slice 4 (§4.14.3): all 3 `[ ]` — UNCHECKED**
- **Integration (§5.15.5): all 5 `[ ]` — UNCHECKED**
Total: **0 checked, 30 unchecked**. **`apply-progress.md` does not exist.**
**This is a CRITICAL completeness issue for the archive gate per the verify
contract.** However — as with Change A — **all 30 items are verifiably DONE
against source**. Selected reconciliation (full mapping available on request):
| Unchecked task | Actual state (verified) |
|----------------|-------------------------|
| 1.1 `ServiceDataHarness` module | **done** (`service_data.py` exists; lifecycle-only) |
| 1.2 `QbittorrentSampleStore` | **done** (`qbittorrent_store.py`; `QBITTORRENT_CONCERN` registered) |
| 1.3 `QbittorrentClient` | **done** (`clients/qbittorrent.py`; login/403/maindata) |
| 1.4/1.5 integration def + registry | **done** (`integrations/qbittorrent.py` + `registry.py` entry) |
| 1.6 harness init in lifespan | **done** (`main.py:56-60` try/except) |
| 1.7/1.8 backend tests | **done** (`test_service_data.py`=9, `test_qbittorrent_client.py`=12) |
| 2.1 adapter | **done** (`QbittorrentWidgetSource` + `SERVICE_ADAPTERS`) |
| 2.3 LineSeriesChart extract | **done** (`components/LineSeriesChart.tsx`; PrometheusChartWidget thin wrapper) |
| 2.42.6/2.7 3 FE widgets + binding | **done** (`QbittorrentTotals/Active/SpeedWidget` + registry.ts) |
| 2.8 FE tests | **done** (3 widget tests + LineSeriesChart test) |
| 3.13.7 MediaIndex migration | **done** (concern + column + scoped replace_items + scoped query + worker/router threading) |
| 3.8 scoped MediaIndex tests | **done** (`test_replace_scoped_by_service_id_preserves_other_services`) |
| 4.1/4.2 cascade wiring + integration test | **done** (`delete_service` cascade + `test_cascade_delete_removes_harness_data_across_concerns`) |
| 5.15.5 full-suite gates | **done** (322 passed / ruff clean / build+lint exit 0 / extraction non-regression) |
The unchecked boxes are **stale** (work performed, tracker not updated), and **no
`apply-progress.md` exists** to serve as the stale-checkbox reconciliation record
the contract permits. Resolution is a **documentation-only** step: tick
§1.11.9, §2.12.9, §3.13.9, §4.14.3, §5.15.5, and author `apply-progress.md`
documenting the four landed slices. **No code change is required.**
> Per the verify contract, an unchecked implementation-task line is an archive
> blocker until reconciled. Because the implementation is verified complete, this
> blocks **archive** but does **not** block `sdd-sync` of the green code.
## 5. TDD compliance & assertion-quality assessment
Strict-TDD was **not** declared active for this change in `openspec/config.yaml` /
parent prompt / (absent) `apply-progress.md`, so the formal TDD-cycle-evidence
check is **not applicable**. Assertion quality was audited adversarially.
**Backend assertions — GENUINELY BEHAVIORAL (good).** Spot-checked:
- `test_media_index.py::test_replace_scoped_by_service_id_preserves_other_services`:
inserts two services' rows, replaces the second, asserts the **count** (3, not
1 — proves the global-clear bug is fixed) and the **id sets** per scoped query
(`{"a1","a2"}`, `{"b1"}`). No tautology; directly locks in SS-121.
- `test_qbittorrent_client.py`: `test_cookie_reuse_does_not_re_login` asserts
`session.post.assert_not_called()` after login (proves no re-auth per request);
`test_403_triggers_re_login` asserts `get` call_count==2 + `post` called once.
- `test_widgets.py` qBit adapters: totals asserts `total` + `by_state` mapping;
active asserts DL/UL kept and queued/stalled excluded; speed asserts series
labels (`["download","upload"]`) + last download value.
- `test_services.py::test_cascade_delete_removes_harness_data_across_concerns`:
asserts qBit A window `== []` after delete while B `len==1`; media A
`total_a_after==0` while B `total_b_after==1`. Real cross-concern, multi-instance.
**Frontend assertions — adequate, two minor notes.**
- qBit widget tests render the **actual value text** (e.g. `"4"`, state badges
`"downloading: 1"`) and the error alert — meaningful. The Totals test includes
an explicit **loading** skeleton case (`isLoading:true``[data-slot="skeleton"]`),
which closes the gap Change A's SC-125 left open. Good.
- `LineSeriesChart.test.tsx` is **smoke-only** (asserts `container.firstChild`
non-null; does not assert the recharts `<Line>` series rendered). Acceptable as
a "renders without crashing" guard but does not prove lines drew. Non-blocking.
No ghost loops, no type-only assertions, no implementation-detail CSS assertions,
no tautologies found.
## 6. Review-workload / PR-boundary findings
Per-slice changed lines (numstat; "source" excludes tests/docs):
| Commit | Slice | Source Δ | Total Δ | Over 400? | Verdict |
|--------|-------|----------|---------|-----------|---------|
| `e7bd0af` | 1 harness+store+client+integration | +400 / -0 | +685 / -1 | at budget (source) | **OK** |
| `1fb12b8` | 2 adapter+LineSeriesChart+3 FE widgets | **+644 / -81** | +831 / -81 | **over** | **WARNING** — additive feature code+tests; boundary is exactly qBit widgets (see below) |
| `c87f398` | 3 MediaIndex migration | +46 / -7 | +93 / -7 | under | **OK** |
| `75c949a` | 4 cascade-delete wiring | +13 / -3 | +99 / -22 | under | **OK** |
The `tasks.md` Review Workload Forecast (`stacked-to-main`, 4 slices,
~9851,205 total) was followed: S1→S2→S3→S4, each independently green. **Slice 2's
+644 source insertions exceed the 400-line budget** and the slice's own
"~350400" forecast (which under-counted the 3 components + LineSeriesChart
extraction + their tests). No `size:exception` was recorded. This is a
**forecast-vs-actual variance on an additive slice**, not scope creep — the
boundary is exactly the qBit-widget feature and no unrelated files were touched.
Recommend recording the slice-2 actual in the archive summary. **Non-blocking.**
Scope was honored: no backend API/type-contract widening beyond the planned
`jellyfin_service_id` query param (defaulted, backward-compatible); MediaIndex
file location unchanged; no coupling to the unrelated `mobile-responsive-parity`
or `services-as-hub-ia` changes (this change builds/tests green independently).
## 7. Adversarial checks
- **Harness initialized at startup?** **Yes.** `main.py::lifespan` calls
`get_service_data_harness()` (which lazy-registers both concerns and runs
migrations) inside a try/except after `ensure_defaults()` (lines 5660). So
`cascade_delete` is operational at runtime, not just in tests.
- **qBit client cookie reuse (no re-auth per request)?** **Correct.** `_get`
only calls `_login()` when `not self._logged_in`; the SID cookie persists in
the `requests.Session`. `test_cookie_reuse_does_not_re_login` locks this in.
- **Speed timestamp ×1000 vs LineSeriesChart expectation?** **Consistent.**
Adapter: `{"t": s["ts"] * 1000, ...}` (store keeps unix seconds; ×1000 → JS ms).
`LineSeriesChart.formatTime(ms)` = `new Date(ms).toLocaleTimeString()`. Same
`{t:ms, v}` contract Prometheus's `normalize_prometheus_matrix` produces, so
both chart consumers are interchangeable.
- **Dead code / unregistered concerns?** **None found.** Both
`QBITTORRENT_CONCERN` and `MEDIA_INDEX_CONCERN` register in
`get_service_data_harness()`. `SERVICE_ADAPTERS` has `qbittorrent`;
`SERVICE_DEFINITIONS` has `qbittorrent`; frontend `SERVICE_REGISTRY` has
`qbittorrent`. `ruff` (catches unused imports) is clean; `tsc`/`eslint` clean.
(`authentik` is backend-only by design — pre-existing, unrelated.)
- **qBit widget kinds declared consistently in BOTH registries?** **Yes.** Backend
`integrations/qbittorrent.py` declares `totals`(30s)/`active`(15s)/`speed`(5s);
frontend `registry.ts` declares the same three kinds with **identical** names,
descriptions, and refresh intervals. Empty config schemas match (`{type:"object",
properties:{}, required:[]}` ↔ empty `model_cls`).
- **MediaIndex backward-compat real?** **Yes.** Existing `test_media_index.py`
calls `replace_items(rows)` / `.query(...)` with no `service_id`; they default
to `""` → delete `WHERE service_id=''` / no filter → all rows. Full suite (322)
green, including `test_api.py::TestMediaIndexApi`.
## 8. Residual risks / non-blocking findings
1. **[CRITICAL-process] 30 unchecked tasks + missing `apply-progress.md`** (§4) — archive blocker; reconciliation is doc-only.
2. **[WARNING] Slice 2 over budget** (§6) — +644 source lines vs 400-line budget; additive feature slice, no scope creep; no `size:exception` recorded.
3. **[INFO] `LineSeriesChart.test.tsx` smoke-only** (§5) — asserts mount, not rendered lines.
4. **[INFO] Prune `NOT IN (… LIMIT 120)` edge case** — duplicate `ts` values could in theory keep >120 rows; negligible at 5 s poll.
5. **[INFO] Stale generated `.pi-map.md`** files predate the new modules; not deliverable source. Regenerate.
6. **[INFO] Slice-2 commit hash drift** — brief cited `8b0e7ea`; actual is `1fb12b8`. Content matches; informational.
7. **[INFO] Uncommitted `MediaTab.tsx`** cosmetic reformat + untracked `.pi-tmp/*` (orthogonal to this change). **No files are staged** (`git diff --cached` empty).
8. **[INFO] Chunk-size build warning** (~1.1 MB JS) — non-fatal, pre-existing, orthogonal.
9. **No browser/visual smoke** performed (out of scope); the qBit speed recharts line is only structurally tested.
## 9. Exact blockers
- **BLOCKER (archive only, doc-reconcilable):** 30 unchecked
implementation/verification tasks (§1.15.5) and absent `apply-progress.md`.
Implementation is verified complete; resolution = tick boxes + write
`apply-progress.md`.
No code-level blockers. All functional requirements SS-101…SS-128 pass. All four
gates green. **Code is ready for `sdd-sync`; archive requires the
checkbox/apply-progress reconciliation.**
## 10. Recommended next phase
**`sdd-sync`** (code PASS). Concurrently/after: author `apply-progress.md`
documenting the four landed slices, and tick §1.11.9, §2.12.9, §3.13.9,
§4.14.3, §5.15.5 in `tasks.md` to clear the archive blocker. Optionally
strengthen `LineSeriesChart.test.tsx` to assert rendered `<Line>` elements, and
regenerate the stale `.pi-map.md`.
---
### Appendix A — Verification commands run (at `c9404f0`)
```
cd backend && PYTHONPATH=src python3 -m pytest -q → 322 passed (2 warnings)
cd backend && PYTHONPATH=src python3 -m ruff check src tests → All checks passed!
cd backend && PYTHONPATH=src python3 -m pytest tests/test_service_data.py tests/test_qbittorrent_client.py -q
→ 40 passed
cd frontend && npm run build → exit 0 (✓ built; >500kB warning pre-existing)
cd frontend && npm run lint → exit 0 (0 errors, 1 pre-existing warning)
cd frontend && npx vitest run src/widgets/__tests__/PrometheusChartWidget.test.tsx
→ 4 passed (SS-118 non-regression)
cd frontend && npx vitest run src/widgets/__tests__/ src/components/__tests__/LineSeriesChart.test.tsx
→ 8 files, 31 tests passed
grep -nE '^\s*- \[' openspec/changes/service-storage-harness/tasks.md → 30 unchecked (all tasks)
ls openspec/changes/service-storage-harness/apply-progress.md → ENOENT (missing)
git diff --cached --name-only → empty (no staged files)
```
### Appendix B — Files substantively changed
**Backend (new)**
- `services/service_data.py``StorageConcern`, `ServiceDataHarness` (lifecycle), `get_service_data_harness()`.
- `services/qbittorrent_store.py``QBITTORRENT_CONCERN`, `QbittorrentSampleStore` (append/window/prune).
- `clients/qbittorrent.py``QbittorrentClient` (cookie login, 403 re-login, maindata).
- `integrations/qbittorrent.py``QbittorrentConfig`/`QbittorrentWidgetConfig`/`DEFINITION` (3 widget kinds).
**Backend (modified)**
- `integrations/registry.py``qbittorrent` registered in `SERVICE_DEFINITIONS`.
- `widgets/sources.py``QbittorrentWidgetSource` (3 branches) + `SERVICE_ADAPTERS["qbittorrent"]`.
- `services/media_index_impl.py``MEDIA_INDEX_CONCERN`, `+service_id` column, scoped `replace_items`/`query`, `build_media_index(service_id=...)`.
- `workers/media_index_worker.py` — threads `--service-id` into `build_media_index`.
- `routers/media.py``query_media(jellyfin_service_id=...)``index.query(service_id=...)`.
- `services/settings_store.py``delete_service` calls harness `cascade_delete` (best-effort).
- `main.py``lifespan` initializes `get_service_data_harness()`.
**Backend tests (new/extended)**
- `tests/test_service_data.py` (new, 9), `tests/test_qbittorrent_client.py` (new, 12).
- `tests/test_media_index.py` (+scoped regression), `tests/test_widgets.py` (+6 qBit adapter tests), `tests/test_services.py` (+cascade integration).
**Frontend (new/modified)**
- `components/LineSeriesChart.tsx` (new, extracted), `widgets/PrometheusChartWidget.tsx` (thin wrapper).
- `widgets/QbittorrentTotalsWidget.tsx`, `QbittorrentActiveTorrentsWidget.tsx`, `QbittorrentSpeedWidget.tsx` (new).
- `integrations/registry.ts``qbittorrent` binding (3 widget kinds).
- Tests: `QbittorrentTotals/Active/SpeedWidget.test.tsx` (new) + `LineSeriesChart.test.tsx` (new).
+1 -1
View File
@@ -5,7 +5,7 @@ context: |
Frontend stack: React 18, TypeScript, Vite, TanStack Query, Tailwind CSS v4 (CSS @theme config in src/index.css), shadcn/ui (Radix primitives), lucide-react, react-router-dom, react-oidc-context.
MIGRATION IN PROGRESS: MUI v9 (@mui/material, @mui/icons-material, @mui/x-data-grid, @emotion/react, @emotion/styled) -> shadcn/ui + Tailwind + lucide-react. The shell (frontend/src/App.tsx) and frontend/src/components/ObservabilityPage.tsx are already migrated and are the style targets. @mui/x-data-grid (pages/Media.tsx, pages/FileBrowser.impl.tsx) migrates to TanStack Table (@tanstack/react-table) with shadcn table styling.
DESIGN TOKENS: src/index.css already defines a full Tailwind v4 @theme token system (light + .dark), Inter font, primary #4f8cff, radius 0.625rem. tailwind.config.cjs is minimal. theme.ts is a no-op shim, safe to delete.
OBSERVABILITY MODEL: Manage is a THIN dashboard. Charts/metrics/logs live in EXTERNAL, decoupled Grafana. In-app surfaces show Alertmanager alerts + Prometheus target health + Grafana deep-links (kiosk iframe). Do NOT re-implement charting in-app. No recharts/d3 is in use.
OBSERVABILITY MODEL: Manage renders Prometheus-backed metrics directly via recharts (line charts, gauges, mean widgets) querying Prometheus /api/v1/query_range. Alertmanager alerts + Prometheus target health are shown in-app. Grafana is no longer integrated.
rules:
proposal:
+128
View File
@@ -0,0 +1,128 @@
# Prometheus Charting
> Domain: `prometheus-charting` · **Canonical specification.** Synced from change `prometheus-direct-charting`.
>
> This is the merged end-state of cutting Grafana out of the metric-visualization path and making
> Prometheus the direct data source. It captures the durable, post-change contracts for in-app
> Prometheus-backed metric rendering, not the per-slice delivery strategy (which remains on record
> in the change's `spec.md` / `tasks.md` under `openspec/changes/prometheus-direct-charting/`).
## Purpose
Define WHAT must be true of Manage's metric-visualization surface after the change: Prometheus is
queried **directly** (no Grafana middleman) to back a multi-series line **chart**, a scalar
**gauge**, and a windowed **mean**, alongside the existing instant-query numeric **metric** widget.
In-app charting via `recharts` is the sanctioned renderer; the `grafana` service type, its widgets,
status endpoint, and UI surfaces are fully removed; orphaned Grafana widget instances degrade
gracefully; and operators migrate by recreating services/widgets (no silent data migration). This
spec is acceptance-focused and verifiable; it deliberately does not prescribe implementation.
## Requirements
### Requirement: SC-101 — Prometheus range query returns the existing series shape
When a `prometheus` widget of kind `chart` is fetched, the backend MUST query `{base_url}/api/v1/query_range` with `query`, `start`, `end`, and `step` derived from the widget config, and return a payload of shape `{ "series": [{ "label": str, "points": [{ "t": int, "v": float|null }] }] }` — the exact shape the frontend chart renderer already consumes.
### Requirement: SC-102 — Series label normalization is shared and Prometheus-native
The metric-label → readable-label normalization MUST live in a single shared helper (not duplicated in a Grafana path) and MUST produce meaningful labels for Prometheus matrix results, including deduplicating repeated labels via a `label (n)` suffix.
### Requirement: SC-103 — Range query errors degrade gracefully
A Prometheus timeout, connection error, or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional.
### Requirement: SC-104 — Step is derived from the window preset
Given a window preset (1h / 6h / 24h / 7d), the backend MUST derive a `step` that yields a reasonable number of points (target ~100300 points). Users do not configure `step` directly.
### Requirement: SC-105 — Chart widget moves from grafana to prometheus
A widget kind named `chart` MUST be bound to the `prometheus` service type in both the backend registry and the frontend `SERVICE_REGISTRY`. The `grafana` service type MUST NOT offer a `chart` kind.
### Requirement: SC-106 — Chart renderer is reused unchanged
The recharts rendering (line chart, multi-series, axes, tooltip, `mergeSeries`, color tokens) MUST be preserved in the rebranded `PrometheusChartWidget`. The rename is structural; the rendering code is not rewritten.
### Requirement: SC-107 — Chart supports multiple series
The `chart` widget MUST render all series returned by the range query, each as its own line with a distinct color. There is no single-series restriction on `chart`.
### Requirement: SC-108 — Chart window is a preset
The `chart` widget config MUST expose the time window as a preset selector (`1h`, `6h`, `24h`, `7d`), not raw `from`/`to`/`step` fields. The preset is stored in widget config and resolved to `start`/`end` server-side.
### Requirement: SC-109 — Gauge renders an instant scalar
A widget kind named `gauge` MUST be bound to the `prometheus` service. Its data fetch MUST run an instant PromQL query (`/api/v1/query`) and return the scalar result for rendering as a gauge.
### Requirement: SC-110 — Gauge supports configurable threshold bands
The `gauge` widget config MUST accept optional threshold values (e.g. `warn_at`, `crit_at`) and the renderer MUST display green / amber / red bands accordingly. When thresholds are omitted, the gauge renders with a single neutral color and no bands.
### Requirement: SC-111 — Gauge is scalar-only
The `gauge` widget MUST render exactly one scalar value. If the instant query returns multiple series, the adapter MUST return `{ "error": str }` (not silently pick one), directing the user to refine the PromQL.
### Requirement: SC-112 — Mean computes client-side over a window
A widget kind named `mean` MUST be bound to the `prometheus` service. Its data fetch MUST run a range query over the configured window preset and return the arithmetic mean of all non-null point values as a single scalar.
### Requirement: SC-113 — Mean uses plain PromQL + window preset
The `mean` widget config MUST accept a plain PromQL expression (no requirement to wrap in `avg_over_time`) plus a window preset. Users do not write range-vector functions.
### Requirement: SC-114 — Mean is scalar-only
The `mean` widget MUST render exactly one scalar value. If the range query returns multiple series, the adapter MUST return `{ "error": str }` (not silently aggregate across series).
### Requirement: SC-115 — No grafana references in backend source
After the change, `grep -ri grafana backend/src --include='*.py'` MUST return no matches (excluding comments/changelog that are explicitly about the removal, if any are retained — but ideally zero).
### Requirement: SC-116 — No grafana references in frontend source
After the change, `grep -ri grafana frontend/src` MUST return no matches, **excluding** (a) test fixtures where "Grafana" appears as a user-authored dashboard *shortcut label* unrelated to the grafana service type (e.g. `Dashboard.test.tsx`), and (b) `LinksTab.tsx` / `service-tabs/index.ts` lines that are themselves being deleted as part of SC-118. (Source finding: `ObservabilityPage.tsx` was refactored into `service-tabs/`.)
### Requirement: SC-117 — Grafana service type is gone from registries
Neither the backend `SERVICE_DEFINITIONS` / `SERVICE_ADAPTERS` nor the frontend `SERVICE_REGISTRY` / `BUILTIN_WIDGETS` MUST contain a `grafana` entry. The `integrations/grafana.py` file MUST be deleted.
### Requirement: SC-118 — Grafana status checks and UI sections are removed
The `get_grafana_status` endpoint and its frontend hook (`useGrafanaStatus`) MUST be removed. The UI surface previously in `ObservabilityPage.tsx` has been refactored into a per-service-type `service-tabs/` architecture; the Grafana removal targets are therefore `service-tabs/LinksTab.tsx` + its test, the `grafana` case in `service-tabs/index.ts`, the grafana entry in `integrations/navEntries.ts`, the `grafana` member of `Dashboard.tsx`'s `OBSERVABILITY_TYPES` set, and any Grafana empty-state copy in `ServicesPage.tsx`. The literal `ObservabilityPage.tsx` no longer exists; SC-118's *intent* (no Grafana UI surface) is what is verified.
### Requirement: SC-119 — Grafana widget instances degrade gracefully
An existing persisted widget row referencing a `grafana` service MUST NOT crash the dashboard. It resolves to the existing "unknown widget" error state and surfaces a clear message; the operator can then delete it.
### Requirement: SC-120 — Grafana tests are removed
All Grafana-specific tests (backend and frontend) MUST be deleted; no test references grafana.
### Requirement: SC-121 — config.yaml matches implementation
`openspec/config.yaml` MUST NOT contain the stale claims "Do NOT re-implement charting in-app" or "No recharts/d3 is in use." It MUST reflect that in-app charting via `recharts` is the sanctioned approach for Prometheus-backed series, and MUST NOT reference Grafana as a chart path.
### Requirement: SC-122 — CHANGELOG documents the migration
`CHANGELOG.md` MUST include an entry instructing operators to delete existing Grafana service instances and recreate them as Prometheus services, noting that configured `grafana/chart` widgets must be recreated as `prometheus/chart` widgets.
### Requirement: SC-123 — Backend tests pass
`pytest` run from `backend/` MUST pass, including new tests covering: Prom range query → `{series}` normalization, gauge scalar-only enforcement, mean client-side aggregation, and the shared label helper.
### Requirement: SC-124 — Frontend typechecks, builds, and lints
`npm run build` (which runs `tsc -b` + `vite build`) and `npm run lint` from `frontend/` MUST pass.
### Requirement: SC-125 — New widget kinds have tests
`PrometheusChartWidget`, `PrometheusGaugeWidget`, and `PrometheusMeanWidget` MUST each have a frontend test covering at least: loading state, error state, and a rendered data case.
### Requirement: SC-126 — No silent data migration
The change MUST NOT attempt to auto-migrate existing `grafana` service rows into `prometheus` rows (URLs differ; true migration is impossible). Migration is operator-driven per the CHANGELOG note.
### Requirement: SC-127 — Non-blocking on the service-storage-harness change
This change MUST NOT depend on the `service-storage-harness` change. It is independently buildable, testable, and deployable. (The reverse dependency holds: the qBit speed widget depends on this change's chart capability.)
+138
View File
@@ -0,0 +1,138 @@
# Service Storage
> Domain: `service-storage` · **Canonical specification.** Synced from change `service-storage-harness`.
>
> This is the merged end-state of standing up a lifecycle-only per-service data layer and migrating
> the qBittorrent integration and the MediaIndex onto it. It captures the durable, post-change
> contracts for the `ServiceDataHarness` lifecycle layer, the qBittorrent store/client/widget stack,
> the MediaIndex migration, and cross-concern cascade-delete — not the per-slice delivery strategy
> (which remains on record in the change's `spec.md` / `tasks.md` under
> `openspec/changes/service-storage-harness/`).
## Purpose
Define WHAT must be true of Manage's per-service data lifecycle and storage layer after the change:
a lifecycle-only `ServiceDataHarness` provisions per-concern databases, runs idempotent
per-integration migrations, and cascades `service_id` deletes across all owned tables — without
offering any generic value table or CRUD. The qBittorrent integration (cookie-auth client +
time-windowed sample store + three-branch widget adapter + three frontend widgets) is the first
concern built on the harness. The MediaIndex migrates onto the harness as the second concern,
gaining a `service_id` column and a scoped `replace_items` that fixes the latent global-clear bug
where rebuilding one Jellyfin instance wiped another's rows. The shared `LineSeriesChart` renderer
(extracted non-regressively from `PrometheusChartWidget`) backs the qBittorrent speed widget.
Deleting a service cascades across both concerns while preserving other services' rows. This spec
is acceptance-focused and verifiable; it deliberately does not prescribe implementation.
## Requirements
### Requirement: SS-101 — Harness is lifecycle-only
A `ServiceDataHarness` class exists in `backend/src/media_library_viewer_api/services/service_data.py` that owns ONLY lifecycle concerns: per-concern DB provisioning, per-integration migrations, `service_id` cascade-delete. It MUST NOT provide generic data operations (no generic value table, no generic CRUD).
### Requirement: SS-102 — Concern registration
An integration/concern registers via a dataclass carrying: DB filename, ordered migration SQL list, owned-tables list, and `service_id` column name. The harness stores registered concerns.
### Requirement: SS-103 — Idempotent migrations
`run_migrations` runs each concern's migration statements and MUST be idempotent — specifically, re-running `ALTER TABLE ... ADD COLUMN` on an already-migrated DB MUST NOT raise (the harness catches "duplicate column name" per-statement).
### Requirement: SS-104 — Cascade-delete across concerns
`cascade_delete(service_id)` iterates every registered concern and, for each owned table, executes `DELETE FROM <table> WHERE <service_id_column> = ?`. It MUST cover every registered concern (qBittorrent samples + media items after this change).
### Requirement: SS-105 — Schema
`QbittorrentSampleStore` owns a `qbittorrent_speed_samples` table with columns `(service_id, ts, dl_speed, up_speed)` and an index on `(service_id, ts)`, in a dedicated `qbittorrent.db` file (per-concern topology).
### Requirement: SS-106 — append/window/prune operations
The store exposes `append(service_id, ts, dl_speed, up_speed)`, `window(service_id, since_ts)` returning ordered rows, and prunes per-append to `MAX_SAMPLES = 120`.
### Requirement: SS-107 — Registered as a harness concern
The qBittorrent sample store is registered with the harness so its table participates in cascade-delete (SS-104).
### Requirement: SS-108 — Cookie login
`QbittorrentClient` authenticates via `POST /api/v2/auth/login` with username/password, stores the resulting cookie, and reuses it for subsequent requests. Credentials are resolved from the `ServiceRecord` secrets (Fernet-encrypted at rest).
### Requirement: SS-109 — 403 re-login
On HTTP 403 the client MUST transparently re-login once and retry the request.
### Requirement: SS-110 — maindata fetch
The client exposes `maindata()` calling `/api/v2/sync/maindata` and returning its dict. Errors (timeout, connection, non-2xx) propagate as exceptions for the adapter to catch.
### Requirement: SS-111 — Three widget kinds dispatched
`QbittorrentWidgetSource.fetch(service, widget_kind, config)` dispatches on `widget_kind` ∈ {`totals`, `active`, `speed`}, resolving the client inline from `ServiceRecord` (same pattern as `PrometheusWidgetSource`).
### Requirement: SS-112 — totals = item count
`totals` returns the count of currently-listed torrents from `maindata()` as `{total: int}`. It is NOT cumulative transfer bytes.
### Requirement: SS-113 — active = downloading/uploading filter
`active` returns the subset of torrents whose `state` is `downloading` or `uploading` as `{torrents: [...]}`.
### Requirement: SS-114 — speed appends sample + returns series
`speed` reads the current dl/up speeds from `maindata()`, appends a sample via `QbittorrentSampleStore.append`, and returns `{series: [{label: "download", points: [...]}, {label: "upload", points: [...]}]}` from `.window()` — the exact shape `LineSeriesChart` consumes (timestamps in JS milliseconds).
### Requirement: SS-115 — Errors degrade gracefully
Adapter errors (auth failure, timeout, connection) return `{error: str}` and MUST NOT raise.
### Requirement: SS-116 — Three widget components
`QbittorrentTotalsWidget`, `QbittorrentActiveTorrentsWidget`, `QbittorrentSpeedWidget` exist under `frontend/src/widgets/` and are bound to the `qbittorrent` service binding in `integrations/registry.ts` with their respective kinds.
### Requirement: SS-117 — Speed widget reuses shared renderer
`QbittorrentSpeedWidget` renders via the shared `LineSeriesChart` component (extracted from `PrometheusChartWidget`). No new charting code or charting dependency.
### Requirement: SS-118 — LineSeriesChart extraction is non-regressive
The extraction of the recharts body into `frontend/src/components/LineSeriesChart.tsx` MUST leave `PrometheusChartWidget`'s behavior and tests green; `PrometheusChartWidget` becomes a thin wrapper.
### Requirement: SS-119 — service_id column added
`media_items` gains a `service_id TEXT NOT NULL DEFAULT ''` column via an idempotent harness migration. The `media_index.db` file location is UNCHANGED.
### Requirement: SS-120 — Existing rows backfill
All pre-existing `media_items` rows receive `service_id = ''` (via the column DEFAULT), preserving their visibility.
### Requirement: SS-121 — replace_items is scoped (bug fix)
`replace_items(service_id=...)` deletes only `WHERE service_id = ?` (not a global `DELETE FROM media_items`). This FIXES the latent global-clear bug where rebuilding for one Jellyfin instance wiped another's rows. A regression test MUST prove service A's rows survive service B's rebuild.
### Requirement: SS-122 — query is backward-compatible
`query(service_id="")` returns all rows (no filter); `query(service_id="X")` scopes to service X. Existing tests that pass no `service_id` MUST continue to pass unchanged.
### Requirement: SS-123 — Worker threads service_id into rows
The media index worker (which already receives `--service-id`) passes it into `replace_items` so newly-built rows are stamped with the real service_id.
### Requirement: SS-124 — Registered as a harness concern
MediaIndex registers `media_items` as a harness-owned table so it participates in cascade-delete.
### Requirement: SS-125 — delete_service triggers harness cascade
`settings_store.delete_service` calls `ServiceDataHarness.cascade_delete(service_id)` after its existing cleanup, in a best-effort try/except (a cascade failure MUST NOT crash the service deletion; it logs and continues).
### Requirement: SS-126 — End-to-end cascade across both concerns
Deleting a service removes both its qBittorrent samples AND its media rows. An integration test MUST prove this across both concerns, and MUST prove rows of OTHER services are preserved.
### Requirement: SS-127 — Backend tests + lint green
`PYTHONPATH=src python3 -m pytest -q` and `PYTHONPATH=src python3 -m ruff check src tests` from `backend/` MUST pass, including new tests for: harness lifecycle, qBit store, qBit client (login/403/maindata), qBit widget adapter (3 branches), MediaIndex scoped replace_items regression, and cascade-delete integration.
### Requirement: SS-128 — Frontend build + lint green
`npm run build` and `npm run lint` from `frontend/` MUST pass (0 errors). New widget tests cover loading/error/rendered states; PrometheusChartWidget tests stay green (SS-118).