perf(qbittorrent): rid incremental sync + shared cache + backoff (stop hanging qBittorrent)
The app was saturating qBittorrent's single-threaded web server and causing
its own Web UI (and the reverse proxy) to hang/504: each of the 3 qBittorrent
widgets fetched /sync/maindata independently, every call was a FULL snapshot
(no rid), and polling was aggressive (5s for speed). For large torrent lists
each snapshot is heavy, so the server queued and Traefik timed out.
QbittorrentClient.maindata now:
- Uses the incremental rid protocol: the first call is a full_update;
subsequent calls send the last rid and get a small diff that is merged into
a cached snapshot (full_update replaces; partial_update merges server_state,
torrents {added/None-removed/..._removed}, categories, tags, trackers).
Payloads shrink dramatically for large libraries.
- Serves a short-TTL (3s) cached snapshot under a lock, so concurrent widget
polls collapse onto a single HTTP fetch instead of N.
- Backs off exponentially (capped 30s) on repeated failure, serving the last
good snapshot when available, so a struggling qBittorrent isn't hammered
further. Returns a shallow race-safe copy of the snapshot per call.
Also slow the speed widget poll from 5s -> 15s (backend widget-kind +
frontend registry) for ~3x fewer calls.
Tests: rid full+partial merge, cache collapses within-TTL calls, backoff
skips the network after failure and serves stale. 393/393 backend + 180/180
frontend tests pass; ruff + tsc + ESLint clean.
This commit is contained in:
@@ -4,7 +4,7 @@ dir: backend/src/media_library_viewer_api/clients
|
||||
index: backend/src/media_library_viewer_api/clients/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Package of API and protocol client wrappers that standardize communication with external services (media servers, identity providers, torrent clients, remote/local hosts) for the media library viewer backend.
|
||||
Provides external service integration clients (APIs, SSH, local execution) for fetching data from media platforms and directory services.
|
||||
## files
|
||||
- __init__.py | Swaps the position of two tmux panes within a window or between windows | dep: tmux, sh
|
||||
- authentik.py | API client wrapper for Authentik directory service providing paginated user browsing and search via REST API. | exp: class:AuthentikClient, method:__init__(self, base_url: str, api_token: str, timeout), call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:self.session.headers.update, raise:ValueError, method:get(self, path: str, **params: Any) → Any, call:params.items, call:logger.debug, call:sorted, call:clean_params.keys, call:self.session.get, call:response.raise_for_status, call:logger.warning, call:response.json, raise:requests.HTTPError, method:users(self, search, page, page_size) → dict[str, Any], call:self.get, call:isinstance, call:logger.warning, call:type, call:payload.get, call:int, call:pagination.get, call:logger.info, call:len | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
@@ -12,12 +12,12 @@ Package of API and protocol client wrappers that standardize communication with
|
||||
- jellyfin.py | Wraps the Jellyfin/Emby HTTP API to provide methods for fetching users, libraries, media items, playback sessions, and image URLs as plain Python dictionaries. | exp: class:JellyfinClient, method:__init__(self, base_url: str, api_key: str, timeout), call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:self.session.headers.update, raise:ValueError, method:get(self, path: str, **params: Any) → Any, call:params.items, call:logger.debug, call:sorted, call:clean_params.keys, call:self.session.get, call:response.raise_for_status, call:logger.warning, call:response.json, raise:requests.HTTPError, method:users(self) → list[dict[str, Any]], call:self.get, call:logger.info, call:len, method:resolve_user_id(self, identifier: str | None) → str, call:self.users, call:any, call:str, call:u.get, call:next, call:logger.info, call:logger.warning, raise:RuntimeError, method:libraries(self, user_id: str) → list[dict[str, Any]], call:self.get(f"/Users/{user_id}/Views").get, call:logger.info, call:len, method:items(self, user_id: str, parent_id, start_index, limit, search, include_item_types, recursive, sort_by, sort_order) → dict[str, Any], call:logger.debug, call:self.get, call:str(recursive).lower, method:item_count(self, user_id: str, include_item_types: str, parent_id) → int, call:self.get, call:int, call:response.get, call:logger.debug, method:media_counts(self, user_id: str) → dict[str, int], call:self.item_count, method:library_item_counts(self, user_id: str, libraries: list[dict[str, Any]]) → list[dict[str, Any]], call:lib.get, call:self.item_count, call:results.append, method:sessions(self, active_within_seconds) → list[dict[str, Any]], call:self.get, call:cast, call:isinstance, method:active_sessions(self, active_within_seconds) → list[dict[str, Any]], call:self.sessions, call:session.get, call:logger.info, call:len, method:image_url(self, item_id: str, image_type) → str | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- jellyseerr.py | HTTP API client wrapper for Jellyseerr to fetch user data and enrich Jellyfin user lists. | exp: class:JellyseerrClient, method:__init__(self, base_url: str, api_key: str, timeout), call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:self.session.headers.update, raise:ValueError, method:get(self, path: str, **params: Any) → Any, call:params.items, call:logger.debug, call:sorted, call:clean_params.keys, call:self.session.get, call:response.raise_for_status, call:logger.warning, call:response.json, raise:requests.HTTPError, method:absolute_url(self, path: str | None) → str, call:path.startswith, method:jellyfin_users(self) → list[dict[str, Any]], call:self.get, call:isinstance, call:logger.info, call:len, call:payload.get, method:users(self, page_size) → list[dict[str, Any]], call:max, call:int, call:self.get, call:isinstance, call:payload.get, call:results.extend, call:page_info.get, call:logger.debug, call:len, call:logger.info | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- local.py | Provides a local command execution client that mirrors remote SSH helpers to run POSIX shell commands, list directories, stat paths, and run ffprobe on the API host for built-in local monitoring. | exp: class:CommandResult, class:LocalCommandClient, method:__init__(self, timeout), method:run(self, command: str, timeout) → CommandResult, call:logger.debug, call:subprocess.run, call:CommandResult, call:logger.warning, call:result.stderr.strip, call:result.stdout.strip, method:list_dir(self, path: str) → CommandResult, call:shlex.quote, call:self.run, method:stat_path(self, path: str) → CommandResult, call:shlex.quote, call:self.run, method:ffprobe_json(self, path: str) → dict[str, object], call:shlex.quote, call:self.run, call:json.loads, raise:RuntimeError | dep: json, logging, posixpath, shlex, subprocess, dataclasses
|
||||
- qbittorrent.py | Minimal read-only qBittorrent Web API client that manages authenticated sessions to fetch `/sync/maindata`. | exp: class:QbittorrentClient, method:__init__(self, base_url: str, username: str, password: str, timeout) → None, call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, raise:ValueError, method:_login(self) → None, call:self._session.post, call:resp.raise_for_status, call:resp.text.strip, call:name.strip().upper, call:upper.startswith, call:resp.headers.get, call:set_cookie_hdr.split("=", 1)[0].strip, call:any, call:_is_session_cookie, call:resp.cookies.keys, call:bool, call:logger.info, call:sorted, raise:RuntimeError, method:_get(self, path: str, **params: Any) → dict[str, Any], call:self._login, call:self._session.get, call:logger.debug, call:resp.raise_for_status, call:resp.json, method:maindata(self) → dict[str, Any], call:self._get | dep: logging, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- qbittorrent.py | Minimal read-only qBittorrent Web API client that handles authentication and fetches maindata with caching and backoff. | exp: class:QbittorrentClient, method:__init__(self, base_url: str, username: str, password: str, timeout) → None, call:base_url.rstrip, call:self.base_url.endswith, call:http_timeout, call:requests.Session, call:threading.Lock, raise:ValueError, method:_login(self) → None, call:self._session.post, call:resp.raise_for_status, call:resp.text.strip, call:name.strip().upper, call:upper.startswith, call:resp.headers.get, call:set_cookie_hdr.split("=", 1)[0].strip, call:any, call:_is_session_cookie, call:resp.cookies.keys, call:bool, call:logger.info, call:sorted, raise:RuntimeError, method:_get(self, path: str, **params: Any) → dict[str, Any], call:self._login, call:self._session.get, call:logger.debug, call:resp.raise_for_status, call:resp.json, method:maindata(self) → dict[str, Any], call:time.time, call:self._snapshot.get, call:self._copy_snapshot, call:self._fetch_maindata_incremental, call:self._apply_update, call:min, call:logger.warning, raise:RuntimeError, method:_fetch_maindata_incremental(self) → dict[str, Any], call:self._get, method:_apply_update(self, update: dict[str, Any]) → None, call:bool, call:update.get, call:snap.clear, call:dict, call:list, call:isinstance, call:snap["server_state"].update, call:changed.items, call:snap["torrents"].pop, call:snap["categories"].update, call:snap["categories"].pop, method:_copy_snapshot(self) → dict[str, Any], call:dict, call:snap.get, call:list | dep: logging, threading, time, typing, requests, media_library_viewer_api.clients.http_timeout
|
||||
- ssh.py | Provides an SSH client wrapper for remote filesystem inspection and media analysis using paramiko, with POSIX shell command execution and host key management. | exp: class:CommandResult, class:RemoteSSHClient, method:__init__(self, host: str, username: str, port, key_filename, private_key, private_key_passphrase, password, known_hosts_path, timeout), raise:ValueError, method:connect(self) → paramiko.SSHClient, call:paramiko.SSHClient, call:client.load_system_host_keys, call:Path, call:bool, call:has_known_host, call:known_hosts_file.is_file, call:client.load_host_keys, call:client.set_missing_host_key_policy, call:paramiko.RejectPolicy, call:paramiko.AutoAddPolicy, call:self._load_private_key, call:client.connect, call:str(exc).lower, call:known_hosts_file.parent.mkdir, call:client.save_host_keys, raise:RuntimeError, method:close(self) → None, call:self._client.close, method:run(self, command: str, timeout) → CommandResult, call:self.connect, call:shlex.quote, call:logger.debug, call:client.exec_command, call:stdout.channel.recv_exit_status, call:CommandResult, call:stdout.read().decode, call:stderr.read().decode, call:logger.warning, call:result.stderr.strip, call:result.stdout.strip, method:list_dir(self, path: str) → CommandResult, call:shlex.quote, call:self.run, call:logger.info, method:stat_path(self, path: str) → CommandResult, call:shlex.quote, call:self.run, call:logger.info, method:ffprobe_json(self, path: str) → dict[str, Any], call:shlex.quote, call:self.run, call:logger.info, call:json.loads, raise:RuntimeError | dep: json, logging, posixpath, shlex, dataclasses, io, pathlib, typing, paramiko, media_library_viewer_api.services.known_hosts
|
||||
## arch
|
||||
Adapter/wrapper pattern where each module encapsulates a specific external service's API or protocol behind a uniform Python interface, returning plain dictionaries and using shared HTTP timeout configuration.
|
||||
Thin wrapper pattern around HTTP libraries (requests) and protocol clients (paramiko SSH), with uniform dict-based outputs, shared timeout configuration, and caching/backoff strategies.
|
||||
## tags
|
||||
call:logger.info, error, call:logger.debug, call:self.get, client, init, timeout, call:logger.warning
|
||||
error, call:logger.info, call:self., call:logger.debug, call:logger.warning, call:self.get, client, init
|
||||
## symbols
|
||||
- AuthentikClient
|
||||
- JellyfinClient
|
||||
|
||||
Reference in New Issue
Block a user