perf(qbittorrent): rid incremental sync + shared cache + backoff (stop hanging qBittorrent)
The app was saturating qBittorrent's single-threaded web server and causing
its own Web UI (and the reverse proxy) to hang/504: each of the 3 qBittorrent
widgets fetched /sync/maindata independently, every call was a FULL snapshot
(no rid), and polling was aggressive (5s for speed). For large torrent lists
each snapshot is heavy, so the server queued and Traefik timed out.
QbittorrentClient.maindata now:
- Uses the incremental rid protocol: the first call is a full_update;
subsequent calls send the last rid and get a small diff that is merged into
a cached snapshot (full_update replaces; partial_update merges server_state,
torrents {added/None-removed/..._removed}, categories, tags, trackers).
Payloads shrink dramatically for large libraries.
- Serves a short-TTL (3s) cached snapshot under a lock, so concurrent widget
polls collapse onto a single HTTP fetch instead of N.
- Backs off exponentially (capped 30s) on repeated failure, serving the last
good snapshot when available, so a struggling qBittorrent isn't hammered
further. Returns a shallow race-safe copy of the snapshot per call.
Also slow the speed widget poll from 5s -> 15s (backend widget-kind +
frontend registry) for ~3x fewer calls.
Tests: rid full+partial merge, cache collapses within-TTL calls, backoff
skips the network after failure and serves stale. 393/393 backend + 180/180
frontend tests pass; ruff + tsc + ESLint clean.
This commit is contained in:
Developer
@@ -2,7 +2,7 @@
|
||||
dir: backend/src/media_library_viewer_api/clients
|
||||
|
||||
## role
|
||||
Package of API and protocol client wrappers that standardize communication with external services (media servers, identity providers, torrent clients, remote/local hosts) for the media library viewer backend.
|
||||
Provides external service integration clients (APIs, SSH, local execution) for fetching data from media platforms and directory services.
|
||||
## parent
|
||||
index: backend/src/media_library_viewer_api/.pi-map.index.md
|
||||
map: backend/src/media_library_viewer_api/.pi-map.md
|
||||
|
||||
@@ -4,7 +4,7 @@ dir: backend/src/media_library_viewer_api/clients
|
||||
index: backend/src/media_library_viewer_api/clients/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Package of API and protocol client wrappers that standardize communication with external services (media servers, identity providers, torrent clients, remote/local hosts) for the media library viewer backend.
|
||||
Provides external service integration clients (APIs, SSH, local execution) for fetching data from media platforms and directory services.
|
||||
## files
|
||||
- __init__.py | Swaps the position of two tmux panes within a window or between windows | dep: tmux, sh
|
||||
- authentik.py | API client wrapper for Authentik directory service providing paginated user browsing and search via REST API. | exp: class:AuthentikClient, method:__init__(self, base_url: str, api_token: str, timeout), call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:self.session.headers.update, raise:ValueError, method:get(self, path: str, **params: Any) → Any, call:params.items, call:logger.debug, call:sorted, call:clean_params.keys, call:self.session.get, call:response.raise_for_status, call:logger.warning, call:response.json, raise:requests.HTTPError, method:users(self, search, page, page_size) → dict[str, Any], call:self.get, call:isinstance, call:logger.warning, call:type, call:payload.get, call:int, call:pagination.get, call:logger.info, call:len | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
@@ -12,12 +12,12 @@ Package of API and protocol client wrappers that standardize communication with
|
||||
- jellyfin.py | Wraps the Jellyfin/Emby HTTP API to provide methods for fetching users, libraries, media items, playback sessions, and image URLs as plain Python dictionaries. | exp: class:JellyfinClient, method:__init__(self, base_url: str, api_key: str, timeout), call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:self.session.headers.update, raise:ValueError, method:get(self, path: str, **params: Any) → Any, call:params.items, call:logger.debug, call:sorted, call:clean_params.keys, call:self.session.get, call:response.raise_for_status, call:logger.warning, call:response.json, raise:requests.HTTPError, method:users(self) → list[dict[str, Any]], call:self.get, call:logger.info, call:len, method:resolve_user_id(self, identifier: str | None) → str, call:self.users, call:any, call:str, call:u.get, call:next, call:logger.info, call:logger.warning, raise:RuntimeError, method:libraries(self, user_id: str) → list[dict[str, Any]], call:self.get(f"/Users/{user_id}/Views").get, call:logger.info, call:len, method:items(self, user_id: str, parent_id, start_index, limit, search, include_item_types, recursive, sort_by, sort_order) → dict[str, Any], call:logger.debug, call:self.get, call:str(recursive).lower, method:item_count(self, user_id: str, include_item_types: str, parent_id) → int, call:self.get, call:int, call:response.get, call:logger.debug, method:media_counts(self, user_id: str) → dict[str, int], call:self.item_count, method:library_item_counts(self, user_id: str, libraries: list[dict[str, Any]]) → list[dict[str, Any]], call:lib.get, call:self.item_count, call:results.append, method:sessions(self, active_within_seconds) → list[dict[str, Any]], call:self.get, call:cast, call:isinstance, method:active_sessions(self, active_within_seconds) → list[dict[str, Any]], call:self.sessions, call:session.get, call:logger.info, call:len, method:image_url(self, item_id: str, image_type) → str | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- jellyseerr.py | HTTP API client wrapper for Jellyseerr to fetch user data and enrich Jellyfin user lists. | exp: class:JellyseerrClient, method:__init__(self, base_url: str, api_key: str, timeout), call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:self.session.headers.update, raise:ValueError, method:get(self, path: str, **params: Any) → Any, call:params.items, call:logger.debug, call:sorted, call:clean_params.keys, call:self.session.get, call:response.raise_for_status, call:logger.warning, call:response.json, raise:requests.HTTPError, method:absolute_url(self, path: str | None) → str, call:path.startswith, method:jellyfin_users(self) → list[dict[str, Any]], call:self.get, call:isinstance, call:logger.info, call:len, call:payload.get, method:users(self, page_size) → list[dict[str, Any]], call:max, call:int, call:self.get, call:isinstance, call:payload.get, call:results.extend, call:page_info.get, call:logger.debug, call:len, call:logger.info | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- local.py | Provides a local command execution client that mirrors remote SSH helpers to run POSIX shell commands, list directories, stat paths, and run ffprobe on the API host for built-in local monitoring. | exp: class:CommandResult, class:LocalCommandClient, method:__init__(self, timeout), method:run(self, command: str, timeout) → CommandResult, call:logger.debug, call:subprocess.run, call:CommandResult, call:logger.warning, call:result.stderr.strip, call:result.stdout.strip, method:list_dir(self, path: str) → CommandResult, call:shlex.quote, call:self.run, method:stat_path(self, path: str) → CommandResult, call:shlex.quote, call:self.run, method:ffprobe_json(self, path: str) → dict[str, object], call:shlex.quote, call:self.run, call:json.loads, raise:RuntimeError | dep: json, logging, posixpath, shlex, subprocess, dataclasses
|
||||
- qbittorrent.py | Minimal read-only qBittorrent Web API client that manages authenticated sessions to fetch `/sync/maindata`. | exp: class:QbittorrentClient, method:__init__(self, base_url: str, username: str, password: str, timeout) → None, call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, raise:ValueError, method:_login(self) → None, call:self._session.post, call:resp.raise_for_status, call:resp.text.strip, call:name.strip().upper, call:upper.startswith, call:resp.headers.get, call:set_cookie_hdr.split("=", 1)[0].strip, call:any, call:_is_session_cookie, call:resp.cookies.keys, call:bool, call:logger.info, call:sorted, raise:RuntimeError, method:_get(self, path: str, **params: Any) → dict[str, Any], call:self._login, call:self._session.get, call:logger.debug, call:resp.raise_for_status, call:resp.json, method:maindata(self) → dict[str, Any], call:self._get | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- qbittorrent.py | Minimal read-only qBittorrent Web API client that handles authentication and fetches maindata with caching and backoff. | exp: class:QbittorrentClient, method:__init__(self, base_url: str, username: str, password: str, timeout) → None, call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:threading.Lock, raise:ValueError, method:_login(self) → None, call:self._session.post, call:resp.raise_for_status, call:resp.text.strip, call:name.strip().upper, call:upper.startswith, call:resp.headers.get, call:set_cookie_hdr.split("=", 1)[0].strip, call:any, call:_is_session_cookie, call:resp.cookies.keys, call:bool, call:logger.info, call:sorted, raise:RuntimeError, method:_get(self, path: str, **params: Any) → dict[str, Any], call:self._login, call:self._session.get, call:logger.debug, call:resp.raise_for_status, call:resp.json, method:maindata(self) → dict[str, Any], call:time.time, call:self._snapshot.get, call:self._copy_snapshot, call:self._fetch_maindata_incremental, call:self._apply_update, call:min, call:logger.warning, raise:RuntimeError, method:_fetch_maindata_incremental(self) → dict[str, Any], call:self._get, method:_apply_update(self, update: dict[str, Any]) → None, call:bool, call:update.get, call:snap.clear, call:dict, call:list, call:isinstance, call:snap["server_state"].update, call:changed.items, call:snap["torrents"].pop, call:snap["categories"].update, call:snap["categories"].pop, method:_copy_snapshot(self) → dict[str, Any], call:dict, call:snap.get, call:list | dep: logging, threading, time, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- ssh.py | Provides an SSH client wrapper for remote filesystem inspection and media analysis using paramiko, with POSIX shell command execution and host key management. | exp: class:CommandResult, class:RemoteSSHClient, method:__init__(self, host: str, username: str, port, key_filename, private_key, private_key_passphrase, password, known_hosts_path, timeout), raise:ValueError, method:connect(self) → paramiko.SSHClient, call:paramiko.SSHClient, call:client.load_system_host_keys, call:Path, call:bool, call:has_known_host, call:known_hosts_file.is_file, call:client.load_host_keys, call:client.set_missing_host_key_policy, call:paramiko.RejectPolicy, call:paramiko.AutoAddPolicy, call:self._load_private_key, call:client.connect, call:str(exc).lower, call:known_hosts_file.parent.mkdir, call:client.save_host_keys, raise:RuntimeError, method:close(self) → None, call:self._client.close, method:run(self, command: str, timeout) → CommandResult, call:self.connect, call:shlex.quote, call:logger.debug, call:client.exec_command, call:stdout.channel.recv_exit_status, call:CommandResult, call:stdout.read().decode, call:stderr.read().decode, call:logger.warning, call:result.stderr.strip, call:result.stdout.strip, method:list_dir(self, path: str) → CommandResult, call:shlex.quote, call:self.run, call:logger.info, method:stat_path(self, path: str) → CommandResult, call:shlex.quote, call:self.run, call:logger.info, method:ffprobe_json(self, path: str) → dict[str, Any], call:shlex.quote, call:self.run, call:logger.info, call:json.loads, raise:RuntimeError | dep: json, logging, posixpath, shlex, dataclasses, io, pathlib, typing, paramiko, media_library_viewer_api.services.known_hosts
|
||||
## arch
|
||||
Adapter/wrapper pattern where each module encapsulates a specific external service's API or protocol behind a uniform Python interface, returning plain dictionaries and using shared HTTP timeout configuration.
|
||||
Thin wrapper pattern around HTTP libraries (requests) and protocol clients (paramiko SSH), with uniform dict-based outputs, shared timeout configuration, and caching/backoff strategies.
|
||||
## tags
|
||||
call:logger.info, error, call:logger.debug, call:self.get, client, init, timeout, call:logger.warning
|
||||
error, call:logger.info, call:self., call:logger.debug, call:logger.warning, call:self.get, client, init
|
||||
## symbols
|
||||
- AuthentikClient
|
||||
- JellyfinClient
|
||||
|
||||
@@ -8,6 +8,8 @@ SID cookie in the requests session. The client re-logins transparently on 403.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
@@ -16,6 +18,11 @@ from media_library_viewer_api.clients.http_timeout import DEFAULT_READ_TIMEOUT,
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# qBittorrent's built-in web server is effectively single-threaded; collapse
|
||||
# concurrent widget polls onto one fetch and back off when it struggles.
|
||||
MAINDATA_CACHE_TTL = 3.0 # seconds a snapshot is served without re-hitting qBittorrent
|
||||
MAINDATA_BACKOFF_MAX = 30.0 # cap exponential backoff after repeated failures
|
||||
|
||||
|
||||
class QbittorrentClient:
|
||||
"""Small wrapper around the qBittorrent Web API.
|
||||
@@ -40,6 +47,24 @@ class QbittorrentClient:
|
||||
self.timeout = http_timeout(timeout)
|
||||
self._session = requests.Session()
|
||||
self._logged_in = False
|
||||
# /sync/maindata is the only hot endpoint. Maintain a rid-merged
|
||||
# snapshot (incremental updates -> small payloads), a short-TTL cache
|
||||
# + lock so concurrent widgets share one fetch, and back off when
|
||||
# qBittorrent is struggling rather than piling on (its web server is
|
||||
# single-threaded and otherwise hangs the Web UI for everyone).
|
||||
self._rid: int | None = None
|
||||
self._snapshot: dict[str, Any] = {
|
||||
"server_state": {},
|
||||
"torrents": {},
|
||||
"categories": {},
|
||||
"tags": [],
|
||||
"trackers": [],
|
||||
}
|
||||
self._maindata_lock = threading.Lock()
|
||||
self._maindata_fetched_at: float = 0.0
|
||||
self._maindata_ttl: float = MAINDATA_CACHE_TTL
|
||||
self._backoff_until: float = 0.0
|
||||
self._consecutive_failures = 0
|
||||
|
||||
def _login(self) -> None:
|
||||
"""POST username/password to ``/auth/login``; store the SID cookie.
|
||||
@@ -74,6 +99,7 @@ class QbittorrentClient:
|
||||
)
|
||||
resp.raise_for_status()
|
||||
body = resp.text.strip()
|
||||
|
||||
# qBittorrent signals a successful login with the body "Ok." and/or by
|
||||
# setting a session cookie. The cookie is named "SID" in older versions
|
||||
# and "QBT_SID" / "QBT_SID_<port>" in newer ones. Some setups return 204
|
||||
@@ -119,10 +145,102 @@ class QbittorrentClient:
|
||||
return resp.json()
|
||||
|
||||
def maindata(self) -> dict[str, Any]:
|
||||
"""Fetch ``/sync/maindata``.
|
||||
"""Return the current ``/sync/maindata`` snapshot.
|
||||
|
||||
Returns a dict with ``server_state`` (containing ``dl_info_speed``,
|
||||
``up_info_speed``, etc.) and ``torrents`` (a dict of
|
||||
``{hash: {name, state, progress, size, dlspeed, upspeed, ...}}``).
|
||||
Uses qBittorrent's incremental ``rid`` protocol (first call is a full
|
||||
update, subsequent calls send the last rid and get a small diff that is
|
||||
merged into the cached snapshot), so payloads stay small. A short-TTL
|
||||
cache + lock collapses concurrent widget polls onto a single fetch, and
|
||||
on repeated failures the client backs off instead of hammering
|
||||
qBittorrent's single-threaded web server (serving the last good
|
||||
snapshot when available).
|
||||
|
||||
Returns a dict with ``server_state`` and ``torrents``.
|
||||
"""
|
||||
return self._get("/sync/maindata")
|
||||
now = time.time()
|
||||
with self._maindata_lock:
|
||||
# Serve a fresh-enough cached snapshot without re-hitting qBittorrent.
|
||||
if self._snapshot.get("torrents") and (now - self._maindata_fetched_at) < self._maindata_ttl:
|
||||
return self._copy_snapshot()
|
||||
# While backing off, don't pile on; serve stale or raise.
|
||||
if now < self._backoff_until:
|
||||
if self._snapshot.get("torrents"):
|
||||
return self._copy_snapshot()
|
||||
raise RuntimeError(
|
||||
"qBittorrent maindata unavailable (backing off after repeated failures)"
|
||||
)
|
||||
try:
|
||||
update = self._fetch_maindata_incremental()
|
||||
self._apply_update(update)
|
||||
except Exception as exc:
|
||||
self._consecutive_failures += 1
|
||||
delay = min(2 ** self._consecutive_failures, MAINDATA_BACKOFF_MAX)
|
||||
self._backoff_until = time.time() + delay
|
||||
logger.warning(
|
||||
"qBittorrent maindata fetch failed (#%s); backing off %.0fs: %s",
|
||||
self._consecutive_failures,
|
||||
delay,
|
||||
exc,
|
||||
)
|
||||
if self._snapshot.get("torrents"):
|
||||
return self._copy_snapshot()
|
||||
raise RuntimeError(f"qBittorrent maindata failed: {exc}") from exc
|
||||
self._maindata_fetched_at = time.time()
|
||||
self._consecutive_failures = 0
|
||||
self._backoff_until = 0.0
|
||||
return self._copy_snapshot()
|
||||
|
||||
def _fetch_maindata_incremental(self) -> dict[str, Any]:
|
||||
"""GET /sync/maindata, sending the last rid for an incremental update."""
|
||||
params: dict[str, Any] = {}
|
||||
if self._rid is not None:
|
||||
params["rid"] = self._rid
|
||||
return self._get("/sync/maindata", **params)
|
||||
|
||||
def _apply_update(self, update: dict[str, Any]) -> None:
|
||||
"""Merge a full or partial maindata update into the cached snapshot."""
|
||||
is_full = bool(update.get("full_update")) or self._rid is None
|
||||
self._rid = update.get("rid", self._rid)
|
||||
snap = self._snapshot
|
||||
if is_full:
|
||||
snap.clear()
|
||||
snap["server_state"] = dict(update.get("server_state") or {})
|
||||
snap["torrents"] = dict(update.get("torrents") or {})
|
||||
snap["categories"] = dict(update.get("categories") or {})
|
||||
snap["tags"] = list(update.get("tags") or [])
|
||||
snap["trackers"] = list(update.get("trackers") or [])
|
||||
return
|
||||
# Partial update — merge the diff.
|
||||
server_state = update.get("server_state")
|
||||
if isinstance(server_state, dict):
|
||||
snap["server_state"].update(server_state)
|
||||
changed = update.get("torrents")
|
||||
if isinstance(changed, dict):
|
||||
for hash_, fields in changed.items():
|
||||
if fields is None:
|
||||
snap["torrents"].pop(hash_, None)
|
||||
else:
|
||||
snap["torrents"][hash_] = fields
|
||||
for hash_ in update.get("torrents_removed") or []:
|
||||
snap["torrents"].pop(hash_, None)
|
||||
categories = update.get("categories")
|
||||
if isinstance(categories, dict):
|
||||
snap["categories"].update(categories)
|
||||
for name in update.get("categories_removed") or []:
|
||||
snap["categories"].pop(name, None)
|
||||
if "tags" in update:
|
||||
snap["tags"] = list(update.get("tags") or [])
|
||||
if "trackers" in update:
|
||||
snap["trackers"] = list(update.get("trackers") or [])
|
||||
|
||||
def _copy_snapshot(self) -> dict[str, Any]:
|
||||
"""Return a shallow, race-safe copy of the current snapshot."""
|
||||
snap = self._snapshot
|
||||
return {
|
||||
"rid": self._rid,
|
||||
"server_state": dict(snap.get("server_state") or {}),
|
||||
"torrents": dict(snap.get("torrents") or {}),
|
||||
"categories": dict(snap.get("categories") or {}),
|
||||
"tags": list(snap.get("tags") or []),
|
||||
"trackers": list(snap.get("trackers") or []),
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: backend/src/media_library_viewer_api/integrations
|
||||
|
||||
## role
|
||||
Provides pluggable external service integrations (Alertmanager, Jellyfin, Nextcloud, Prometheus, qBittorrent, Authentik, Backups, SSH) with standardized config schemas, widgets, connection testing, and a central registry for discovery.
|
||||
Provides pluggable external service integrations (e.g., Alertmanager, Jellyfin, Prometheus, qBittorrent) with standardized configuration, connection testing, and widget definitions for a monitoring dashboard.
|
||||
## parent
|
||||
index: backend/src/media_library_viewer_api/.pi-map.index.md
|
||||
map: backend/src/media_library_viewer_api/.pi-map.md
|
||||
|
||||
@@ -4,7 +4,7 @@ dir: backend/src/media_library_viewer_api/integrations
|
||||
index: backend/src/media_library_viewer_api/integrations/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Provides pluggable external service integrations (Alertmanager, Jellyfin, Nextcloud, Prometheus, qBittorrent, Authentik, Backups, SSH) with standardized config schemas, widgets, connection testing, and a central registry for discovery.
|
||||
Provides pluggable external service integrations (e.g., Alertmanager, Jellyfin, Prometheus, qBittorrent) with standardized configuration, connection testing, and widget definitions for a monitoring dashboard.
|
||||
## files
|
||||
- __init__.py | Defines a closed registry module for service integrations.
|
||||
- alertmanager.py | Defines a service integration for Prometheus Alertmanager, providing configuration models, connection testing, alert summarization, and widget definitions for displaying active alerts. | exp: class:AlertmanagerConfig, class:AlertmanagerAlertsWidgetConfig, func:summarize_alerts(alerts: list[dict[str, Any]], severity_filter) → dict[str, Any], call:alert.get, call:labels.get, call:by_severity.get, call:open_alerts.append, call:annotations.get, call:open_alerts.sort, call:len, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:str(config.get("base_url") or "").rstrip, call:config.get, call:int, call:secrets.get, call:requests.get, call:resp.raise_for_status, call:resp.json, call:payload.get("versionInfo", {}).get, call:TestResult, call:translate_connection_error | dep: typing, requests, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store
|
||||
@@ -14,11 +14,11 @@ Provides pluggable external service integrations (Alertmanager, Jellyfin, Nextcl
|
||||
- jellyfin.py | Defines the Jellyfin media server service integration, including connection testing, configuration models, and widget definitions for activity monitoring. | exp: class:JellyfinConfig, class:JellyfinActivityWidgetConfig, class:JellyfinNowPlayingWidgetConfig, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:str, call:config.get, call:secrets.get, call:int, call:JellyfinClient, call:client.users, call:TestResult, call:len, call:translate_connection_error | dep: typing, media_library_viewer_api.clients.jellyfin, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store, media_library_viewer_api.clients.jellyfin.JellyfinClient, media_library_viewer_api.services.settings_store.SettingsStore
|
||||
- nextcloud.py | Defines a Nextcloud service integration with connection testing and configuration for a media library viewer API. | exp: class:NextcloudConfig, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:str(config.get("base_url") or "").rstrip, call:config.get, call:requests.get, call:resp.raise_for_status, call:resp.json, call:payload.get, call:TestResult, call:translate_connection_error | dep: typing, requests, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store
|
||||
- prometheus.py | Defines the Prometheus service integration for a media library viewer API, including connection testing via a Grafana gateway and configuration models for metric, chart, gauge, and mean widgets. | exp: class:PrometheusConfig, class:PrometheusMetricWidgetConfig, class:PrometheusChartWidgetConfig, class:PrometheusGaugeWidgetConfig, class:PrometheusMeanWidgetConfig, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:str(config.get("grafana_url") or "").rstrip, call:config.get, call:secrets.get, call:int, call:TestResult, call:requests.post, call:resp.raise_for_status, call:translate_connection_error | dep: typing, requests, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store
|
||||
- qbittorrent.py | Defines the qBittorrent service integration, including connection config, secret fields, three widget kinds (totals, active, speed), and a connection test function. | exp: class:QbittorrentConfig, class:QbittorrentWidgetConfig, class:QbittorrentSpeedWidgetConfig, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:config.get, call:secrets.get, call:int, call:QbittorrentClient, call:client.maindata, call:data.get("server_state", {}).get, call:TestResult, call:str(exc).lower, call:translate_connection_error | dep: typing, media_library_viewer_api.clients.qbittorrent, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store, media_library_viewer_api.clients.qbittorrent.QbittorrentClient, media_library_viewer_api.services.settings_store.SettingsStore
|
||||
- qbittorrent.py | Defines the qBittorrent service integration, including connection config models, secret fields, widget definitions (totals, active, speed), and a connection test function. | exp: class:QbittorrentConfig, class:QbittorrentWidgetConfig, class:QbittorrentSpeedWidgetConfig, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:config.get, call:secrets.get, call:int, call:QbittorrentClient, call:client.maindata, call:data.get("server_state", {}).get, call:TestResult, call:str(exc).lower, call:translate_connection_error | dep: typing, media_library_viewer_api.clients.qbittorrent, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store, media_library_viewer_api.clients.qbittorrent.QbittorrentClient, media_library_viewer_api.services.settings_store.SettingsStore
|
||||
- registry.py | Maintains a closed registry of service definitions and provides lookup functions to query available services, their types, and widget kinds. | exp: func:list_service_types() → list[str], call:sorted, func:get_service_definition(service_type: str) → ServiceDefinition | None, call:SERVICE_DEFINITIONS.get, func:get_widget_kind(service_type: str, widget_kind: str) → WidgetKind | None, call:get_service_definition, call:definition.widget_kind, func:require_service_definition(service_type: str) → ServiceDefinition, call:get_service_definition, raise:ValueError | dep: media_library_viewer_api.integrations.alertmanager, media_library_viewer_api.integrations.authentik, media_library_viewer_api.integrations.backups, media_library_viewer_api.integrations.base, media_library_viewer_api.integrations.jellyfin, media_library_viewer_api.integrations.nextcloud, media_library_viewer_api.integrations.prometheus, media_library_viewer_api.integrations.qbittorrent, media_library_viewer_api.integrations.ssh_tasks
|
||||
- ssh_tasks.py | Defines a service plugin that runs reusable saved tasks over SSH by managing connection configuration, secrets, and connection testing. | exp: class:SshTasksConfig, class:SshTaskOutputWidgetConfig, func:test_connection(config: dict[str, Any], secrets: dict[str, str], store: SettingsStore) → TestResult, call:str(config.get("host") or "").strip, call:config.get, call:int, call:ServiceRecord, call:build_ssh_client, call:client.connect, call:str(exc).lower, call:TestResult, call:translate_connection_error, call:client.close | dep: typing, media_library_viewer_api.integrations.base, media_library_viewer_api.services.settings_store, media_library_viewer_api.services.task_runner, media_library_viewer_api.widgets.sources
|
||||
## arch
|
||||
Plugin/registry pattern with abstract base classes defining config models, secrets, widgets, and connection tests; each integration is a self-contained module registered in a closed registry for lookup and dynamic loading.
|
||||
Registry-based plugin pattern with abstract base classes defining config schemas, secrets, widgets, and connection tests; each integration is a self-contained module auto-registered in a closed central registry for discovery and lookup.
|
||||
## tags
|
||||
config, connection, widget, media_library_viewer_api, service, error, integrations, test
|
||||
## symbols
|
||||
|
||||
@@ -111,7 +111,7 @@ DEFINITION = ServiceDefinition(
|
||||
description="Live download/upload speed over a short window.",
|
||||
model_cls=QbittorrentSpeedWidgetConfig,
|
||||
default_config={"unit": "bytes_per_sec", "scale": "auto"},
|
||||
refresh_interval_ms=5_000,
|
||||
refresh_interval_ms=15_000,
|
||||
),
|
||||
],
|
||||
test_callable=test_connection,
|
||||
|
||||
@@ -117,6 +117,90 @@ class QbittorrentClientTests(unittest.TestCase):
|
||||
self.assertEqual(result["server_state"]["dl_info_speed"], 12345)
|
||||
self.assertEqual(len(result["torrents"]), 2)
|
||||
|
||||
def test_maindata_uses_rid_and_merges_partial_update(self) -> None:
|
||||
"""First call is a full fetch (no rid); later calls send rid and merge the diff."""
|
||||
self.client._logged_in = True
|
||||
self.client._maindata_ttl = 0 # force a real fetch each call
|
||||
full = {
|
||||
"rid": 10,
|
||||
"full_update": True,
|
||||
"server_state": {"dl_info_speed": 100},
|
||||
"torrents": {"a": {"name": "A", "state": "downloading"}},
|
||||
}
|
||||
partial = {
|
||||
"rid": 11,
|
||||
"full_update": False,
|
||||
"server_state": {"dl_info_speed": 200},
|
||||
"torrents": {"a": {"name": "A", "state": "pausedDL"}},
|
||||
}
|
||||
self.session.get.side_effect = [self._get_response(full), self._get_response(partial)]
|
||||
|
||||
r1 = self.client.maindata()
|
||||
self.assertNotIn("rid", self.session.get.call_args_list[0].kwargs["params"])
|
||||
self.assertEqual(r1["server_state"]["dl_info_speed"], 100)
|
||||
self.assertEqual(r1["torrents"]["a"]["state"], "downloading")
|
||||
|
||||
r2 = self.client.maindata()
|
||||
self.assertEqual(self.session.get.call_args_list[1].kwargs["params"].get("rid"), 10)
|
||||
self.assertEqual(r2["server_state"]["dl_info_speed"], 200) # merged
|
||||
self.assertEqual(r2["torrents"]["a"]["state"], "pausedDL") # merged
|
||||
|
||||
def test_maindata_caches_concurrent_calls_within_ttl(self) -> None:
|
||||
"""Two calls within the TTL collapse to a single HTTP fetch."""
|
||||
self.client._logged_in = True
|
||||
payload = {
|
||||
"rid": 1,
|
||||
"full_update": True,
|
||||
"server_state": {"dl_info_speed": 5},
|
||||
"torrents": {"a": {"state": "downloading"}},
|
||||
}
|
||||
self.session.get.return_value = self._get_response(payload)
|
||||
|
||||
self.client.maindata()
|
||||
r2 = self.client.maindata() # served from cache — no extra HTTP
|
||||
|
||||
self.assertEqual(self.session.get.call_count, 1)
|
||||
self.assertEqual(r2["server_state"]["dl_info_speed"], 5)
|
||||
|
||||
def test_maindata_backoff_after_failure_does_not_pile_on(self) -> None:
|
||||
"""A failed fetch arms backoff; the next call skips the network entirely."""
|
||||
self.client._logged_in = True
|
||||
self.client._maindata_ttl = 0
|
||||
bad = MagicMock()
|
||||
bad.status_code = 503
|
||||
bad.raise_for_status.side_effect = requests.HTTPError("503 Server Error")
|
||||
bad.text = ""
|
||||
self.session.get.return_value = bad
|
||||
|
||||
with self.assertRaises(RuntimeError): # no snapshot yet -> raises + arms backoff
|
||||
self.client.maindata()
|
||||
self.assertEqual(self.session.get.call_count, 1)
|
||||
|
||||
with self.assertRaises(RuntimeError): # within backoff -> no new HTTP
|
||||
self.client.maindata()
|
||||
self.assertEqual(self.session.get.call_count, 1)
|
||||
|
||||
def test_maindata_serves_stale_snapshot_during_backoff(self) -> None:
|
||||
"""After a good fetch, a later failure serves stale data instead of erroring."""
|
||||
self.client._logged_in = True
|
||||
self.client._maindata_ttl = 0
|
||||
good = {
|
||||
"rid": 1,
|
||||
"full_update": True,
|
||||
"server_state": {"dl_info_speed": 7},
|
||||
"torrents": {"a": {"state": "downloading"}},
|
||||
}
|
||||
bad = MagicMock()
|
||||
bad.status_code = 503
|
||||
bad.raise_for_status.side_effect = requests.HTTPError("503")
|
||||
bad.text = ""
|
||||
self.session.get.side_effect = [self._get_response(good), bad]
|
||||
|
||||
r1 = self.client.maindata()
|
||||
self.assertEqual(r1["server_state"]["dl_info_speed"], 7)
|
||||
r2 = self.client.maindata() # fetch fails -> serves stale snapshot, no raise
|
||||
self.assertEqual(r2["server_state"]["dl_info_speed"], 7)
|
||||
|
||||
@patch("media_library_viewer_api.clients.qbittorrent.requests.Session")
|
||||
def test_login_http_error_propagates(self, mock_session_cls: MagicMock) -> None:
|
||||
"""A network error during login propagates as requests exception."""
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: frontend/src/integrations
|
||||
|
||||
## role
|
||||
Central integration layer that maps backend service types to frontend React components, navigation entries, and configuration schemas.
|
||||
Frontend integration layer that maps service types and widgets to their React components, configuration schemas, and navigation entries.
|
||||
## parent
|
||||
index: frontend/src/.pi-map.index.md
|
||||
map: frontend/src/.pi-map.md
|
||||
|
||||
@@ -4,15 +4,15 @@ dir: frontend/src/integrations
|
||||
index: frontend/src/integrations/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Central integration layer that maps backend service types to frontend React components, navigation entries, and configuration schemas.
|
||||
Frontend integration layer that maps service types and widgets to their React components, configuration schemas, and navigation entries.
|
||||
## files
|
||||
- navEntries.ts | Defines a static mapping of service types to navigation entries and provides a filter function to return only entries for currently configured services. | exp: NavEntry, SERVICE_TYPE_NAV_ENTRIES, func:configuredNavEntries(configuredTypes: Set<string>) → NavEntry[], call:SERVICE_TYPE_NAV_ENTRIES.filter, call:configuredTypes.has | dep: lucide-react
|
||||
- registry.test.ts | Tests the service and widget registry, verifying service registrations, widget bindings, configuration schemas, and widget resolution logic. | dep: vitest, ./registry, ../types
|
||||
- registry.ts | Provides a frontend registry that maps service types and built-in widget kinds to their corresponding React components, metadata, and configuration schemas. | exp: WidgetComponentProps, ServiceWidgetBinding, ServiceBinding, SERVICE_REGISTRY, BUILTIN_WIDGETS, ResolvedWidget, func:getServiceBinding(serviceType: string) → ServiceBinding | undefined, func:getBuiltinBinding(kind: string) → ServiceWidgetBinding | undefined, func:resolveWidget(widget: WidgetInstance, services: ServiceInstance[]) → ResolvedWidget | undefined, call:services.find, call:getServiceBinding, call:binding?.widgets.find, call:getBuiltinBinding, func:enrichServiceTypes(types: ServiceTypeInfo[]) → ServiceTypeInfo[] | dep: react, ../widgets/AlertmanagerAlertsWidget, ../widgets/BackupsWidget, ../widgets/MetricChartWidget, ../widgets/MetricGaugeWidget, ../widgets/MetricMeanWidget, ../widgets/JellyfinWidget, ../widgets/JellyfinNowPlayingWidget, ../widgets/PrometheusMetricWidget, ../widgets/QbittorrentActiveTorrentsWidget, ../widgets/QbittorrentSpeedWidget, ../widgets/QbittorrentTotalsWidget, ../widgets/SshTaskWidget, ../widgets/StaticWidget, ../types, various widget components, internal types module
|
||||
- registry.test.ts | Tests the service and widget registry, validating correct registration of service types, widget bindings, configuration schemas, and widget resolution behavior. | dep: vitest, ./registry, ../types
|
||||
- registry.ts | Provides a frontend registry that maps service types and built-in widgets to their React components, config schemas, and metadata, with a resolver function for widget instances. | exp: WidgetComponentProps, ServiceWidgetBinding, ServiceBinding, SERVICE_REGISTRY, BUILTIN_WIDGETS, ResolvedWidget, func:getServiceBinding(serviceType: string) → ServiceBinding | undefined, func:getBuiltinBinding(kind: string) → ServiceWidgetBinding | undefined, func:resolveWidget(widget: WidgetInstance, services: ServiceInstance[]) → ResolvedWidget | undefined, call:services.find, call:getServiceBinding, call:binding?.widgets.find, call:getBuiltinBinding, func:enrichServiceTypes(types: ServiceTypeInfo[]) → ServiceTypeInfo[] | dep: react, ../widgets/AlertmanagerAlertsWidget, ../widgets/BackupsWidget, ../widgets/MetricChartWidget, ../widgets/MetricGaugeWidget, ../widgets/MetricMeanWidget, ../widgets/JellyfinWidget, ../widgets/JellyfinNowPlayingWidget, ../widgets/PrometheusMetricWidget, ../widgets/QbittorrentActiveTorrentsWidget, ../widgets/QbittorrentSpeedWidget, ../widgets/QbittorrentTotalsWidget, ../widgets/SshTaskWidget, ../widgets/StaticWidget, ../types, AlertmanagerAlertsWidget, BackupsWidget, MetricChartWidget, MetricGaugeWidget, MetricMeanWidget, JellyfinWidget, JellyfinNowPlayingWidget, PrometheusMetricWidget, QbittorrentActiveTorrentsWidget, QbittorrentSpeedWidget, QbittorrentTotalsWidget, SshTaskWidget, StaticWidget
|
||||
## arch
|
||||
Registry pattern with static navigation mappings, component resolution logic, and test-validated service/widget bindings.
|
||||
Registry pattern with static mappings and a resolver function, centralizing widget/component resolution and service configuration metadata for the UI.
|
||||
## tags
|
||||
service, widgets, widget, binding, nav, entries, registry, types
|
||||
service, widgets, widget, binding, nav, entries, types, registry
|
||||
## symbols
|
||||
- configuredNavEntries
|
||||
- getServiceBinding
|
||||
|
||||
@@ -119,7 +119,12 @@ export const SERVICE_REGISTRY: Record<string, ServiceBinding> = {
|
||||
name: "Chart",
|
||||
description: "Multi-series line chart from a PromQL range query.",
|
||||
refreshIntervalMs: 60_000,
|
||||
defaultConfig: { promql: "", window: "1h", unit: "none", scale: "auto" },
|
||||
defaultConfig: {
|
||||
promql: "",
|
||||
window: "1h",
|
||||
unit: "none",
|
||||
scale: "auto",
|
||||
},
|
||||
configSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
@@ -213,7 +218,7 @@ export const SERVICE_REGISTRY: Record<string, ServiceBinding> = {
|
||||
kind: "speed",
|
||||
name: "Speed chart",
|
||||
description: "Live download/upload speed over a short window.",
|
||||
refreshIntervalMs: 5_000,
|
||||
refreshIntervalMs: 15_000,
|
||||
defaultConfig: { unit: "bytes_per_sec", scale: "auto" },
|
||||
configSchema: {
|
||||
type: "object",
|
||||
|
||||
Reference in New Issue
Block a user