Compare commits

...

4 Commits

Author SHA1 Message Date
Developer 24427b4869 chore(config): remove dead GRAFANA_URL and wire VITE_GRAFANA_URL
- Remove GRAFANA_URL from backend environment (backend never consumed it).
- Add VITE_GRAFANA_URL to frontend build-args (prod), dev environment, and
  frontend/Dockerfile ARG/ENV so Grafana deep-links resolve correctly.
- Add ALERTMANAGER_WEBHOOK_URL to backend environment so the documented
  alert-forwarding feature is reachable from compose.
- Document VITE_GRAFANA_URL in .env.example.
2026-06-19 20:07:47 +00:00
Developer bb8b040657 docs(monitoring): record legacy poller decommission (slice 3)
Update docs to reflect that Manage no longer scrapes its own system
metrics (slices 1-2). AGENTS.md, REQUIREMENTS.md (decision log +
observability section), monitoring-logging-design.md, MIGRATION_PLAN.md.

Gate: docs only; backend pytest (173) + frontend build/lint/test (22/63)
remain green from slices 1-2.
2026-06-17 20:55:24 +00:00
Developer a8eb751322 refactor(monitoring): remove orphaned frontend DiskSpaceCard (slice 2)
After slice 1 removed /api/monitoring/disk, the frontend DiskSpaceCard
component (and its DiskSpace type) had zero importers — it fed nothing.
Delete them.

Removed (frontend):
- components/DiskSpaceCard.tsx
- components/__tests__/DiskSpaceCard.test.tsx
- types/index.ts: DiskSpace interface

Gate: build + lint + vitest (22 files / 63 tests) green.
2026-06-17 20:51:54 +00:00
Developer 08a3b616f6 refactor(monitoring): decommission legacy SSH-scraping poller (slice 1)
The 2026-06-16/17 observability update externalised metrics to
Prometheus + node_exporter + Grafana, but the legacy Manage-side
SSH-scraping monitor was never removed. It duplicated the new stack,
ran SSH df on every machine every 300s, and fed nothing (its UI was
deleted in e2ad731). This slice decommissions the duplication.

Removed (backend):
- services/monitoring_poller.py (MonitoringPoller) — entire file
- services/monitoring_actions.py (disk_space, run_machine_operation,
  poll_machine_snapshot, build_machine_client) — entire file;
  run_machine_operation had only 2 callers (the poller + /disk), both gone
- tests/test_monitoring_actions.py
- endpoints: POST /api/monitoring/poller, GET /machines/{id}/actions,
  GET /disk (and the now-dead _resolve_machine helper)
- lifespan wiring (main.py), dependency wrapper (dependencies.py),
  poller.start()/kick() from machine save (routers/settings.py)
- SettingsStore: monitoring_machine_actions table CREATE + 2 indexes +
  record/list/prune_machine_actions methods; DROP TABLE IF EXISTS on
  startup cleans existing DBs (user-approved)
- config knobs: monitoring_poll_interval_seconds,
  monitoring_poll_initial_delay_seconds, monitoring_action_retention_days
- test_api.py: TestMonitoring._ensure_machine + test_disk

Kept (fits the new model): /machines, /prometheus-targets, /alerts,
/alertmanager-status, /alertmanager-webhook; the disk_usage JOB template
(manual on-demand, not monitoring); node_exporter_* machine fields
(they point Prometheus at the right host).

Gate: backend pytest 173 passed; ruff clean.
2026-06-17 20:48:56 +00:00
22 changed files with 151 additions and 811 deletions
+1
View File
@@ -41,6 +41,7 @@ VITE_OIDC_SCOPE=openid profile email
VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback
VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/
VITE_DEV_API_PROXY_TARGET=http://backend:8000
VITE_GRAFANA_URL=https://grafana.example.com
# SMTP
SMTP_HOST=smtp.example.com
+4 -1
View File
@@ -1,12 +1,14 @@
# AGENTS.md
## Layout
- Current app is `backend/` (FastAPI) plus `frontend/` (Vite React); ignore Streamlit-era commands in `CONTRIBUTING.md`.
- Backend entrypoint: `backend/src/media_library_viewer_api/main.py` (`media_library_viewer_api.main:app`).
- Frontend entrypoint: `frontend/src/main.tsx`.
- Backend uses a `src/` layout; tests live in `backend/tests/`.
## Commands
- Backend setup: `cd backend && python -m venv .venv && source .venv/bin/activate && pip install -e '.[dev]'`
- Backend run: `uvicorn media_library_viewer_api.main:app --reload --port 8000`; if not installed, use `PYTHONPATH=src uvicorn media_library_viewer_api.main:app --reload --port 8000`.
- Backend tests: run `pytest` from `backend/`; focused checks can use `pytest tests/test_api.py` or `pytest -k <expr>`; if the package is not installed, use `PYTHONPATH=src pytest`.
@@ -17,12 +19,13 @@
- Production stack: `docker compose up --build`
## Repo-Specific Gotchas
- Root compose files rely on environment-variable interpolation, not `env_file`; export required values before running them.
- Production compose needs the host/cert and OIDC variables from `docker-compose.yml` (`BACKEND_APP_HOST`, `FRONTEND_APP_HOST`, `CERT_RESOLVER`, and the frontend OIDC vars).
- Dev compose runs with auth off and does not need SSH key material unless you add remote SSH machines.
- `backend_cache` persists the media index and the managed `known_hosts` file.
- SSH host-key checking is strict, but the first successful connect records the host key into backend-managed `known_hosts`.
- Backend startup validates auth settings, rebuilds managed `known_hosts`, and starts the mail queue and monitoring poller.
- Backend startup validates auth settings, rebuilds managed `known_hosts`, and starts the mail queue and backup alert poller.
- Machine-level settings now own Jellyfin/Jellyseerr/SSH config; the backend seeds a local machine automatically.
- Remote job templates live in `backend/src/media_library_viewer_api/jobs.py`; keep shell quoting intact.
- Backend Ruff config is in `backend/pyproject.toml` and uses line length 120 with Python 3.11.
@@ -52,11 +52,6 @@ class Settings(BaseSettings):
ssh_password: str = ""
ssh_known_hosts_path: str = ""
# Monitoring poller
monitoring_poll_interval_seconds: int = 300
monitoring_poll_initial_delay_seconds: int = 20
monitoring_action_retention_days: int = 30
# Observability
prometheus_enabled: bool = True
prometheus_file_sd_dir: str = "/app/backend/.cache/prometheus-file-sd"
@@ -21,12 +21,6 @@ from media_library_viewer_api.clients.ssh import RemoteSSHClient
from media_library_viewer_api.config import get_settings
from media_library_viewer_api.services.mail_queue import MailQueue
from media_library_viewer_api.services.mail_queue import get_mail_queue as _get_mail_queue
from media_library_viewer_api.services.monitoring_poller import (
MonitoringPoller,
)
from media_library_viewer_api.services.monitoring_poller import (
get_monitoring_poller as _get_monitoring_poller,
)
from media_library_viewer_api.services.settings_store import SettingsStore
from media_library_viewer_api.services.settings_store import get_settings_store as _get_settings_store
@@ -251,11 +245,6 @@ def get_mail_queue() -> MailQueue:
return _get_mail_queue()
def get_monitoring_poller() -> MonitoringPoller:
"""Return the singleton background monitoring poller."""
return _get_monitoring_poller()
def get_settings_store() -> SettingsStore:
"""Return the singleton persistent settings store."""
return _get_settings_store()
+1 -4
View File
@@ -13,7 +13,7 @@ from fastapi.responses import Response as FastAPIResponse
from media_library_viewer_api.auth import require_jwt_auth, validate_auth_settings
from media_library_viewer_api.config import get_settings
from media_library_viewer_api.dependencies import get_mail_queue, get_monitoring_poller, get_settings_store
from media_library_viewer_api.dependencies import get_mail_queue, get_settings_store
from media_library_viewer_api.logging_utils import configure_logging, describe_settings, sanitize_log_extra
from media_library_viewer_api.observability import (
get_request_id,
@@ -46,13 +46,10 @@ async def lifespan(app: FastAPI):
except Exception:
logger.exception("Failed to write Prometheus file-SD targets during startup")
mail_queue = get_mail_queue()
monitoring_poller = get_monitoring_poller()
backup_poller = get_backup_poller()
mail_queue.start()
monitoring_poller.start()
backup_poller.start()
yield
monitoring_poller.stop()
backup_poller.stop()
mail_queue.stop()
logger.info("Backend shutdown complete")
@@ -1,18 +1,14 @@
"""Monitoring router — disk checks, action history, and observability stack status."""
"""Monitoring router — observability stack status (Alertmanager + Prometheus)."""
from __future__ import annotations
import logging
from typing import Any
from fastapi import APIRouter, Body, Depends, HTTPException, Query
from fastapi import APIRouter, Body, Depends
from media_library_viewer_api.config import get_settings
from media_library_viewer_api.dependencies import get_settings_store
from media_library_viewer_api.services.monitoring_actions import (
disk_space,
run_machine_operation,
)
from media_library_viewer_api.services.settings_store import SettingsStore
from media_library_viewer_api.services.targets import build_node_exporter_targets
@@ -68,80 +64,12 @@ def _summary_from_alerts(alerts: list[dict[str, Any]]) -> dict[str, Any]:
router = APIRouter(prefix="/api/monitoring", tags=["monitoring"])
def _resolve_machine(store: SettingsStore, machine_id: str | None) -> dict[str, Any]:
"""Return the requested machine or the first enabled machine.
Monitoring is treated as a machine-by-machine view. If a machine is
explicitly requested but disabled, we surface that as a user-facing error so
the Settings tab can be used to re-enable it.
"""
machines = store.list_machines()
if machine_id:
machine = next((item for item in machines if item["id"] == machine_id), None)
if not machine:
raise HTTPException(status_code=404, detail="Monitoring machine not found")
if not machine.get("enabled"):
raise HTTPException(status_code=409, detail=f"Monitoring machine '{machine['name']}' is disabled")
return machine
for machine in machines:
if machine.get("enabled"):
return machine
raise HTTPException(status_code=404, detail="No enabled monitoring machines configured")
@router.get("/machines")
def get_machines(store: SettingsStore = Depends(get_settings_store)) -> list[dict[str, Any]]:
"""Return enabled monitoring machines for the UI."""
return [m for m in store.list_machines() if m.get("enabled")]
@router.get("/poller")
def get_poller_status() -> dict[str, Any]:
"""Return the backend poller status and configuration."""
from media_library_viewer_api.dependencies import get_monitoring_poller
poller = get_monitoring_poller().snapshot()
logger.info(
"Monitoring poller status requested running=%s poll_count=%s",
poller.get("worker_running"),
poller.get("poll_count"),
)
return poller
@router.get("/machines/{machine_id}/actions")
def get_machine_actions(
machine_id: str,
limit: int = 20,
action: str | None = None,
status: str | None = None,
store: SettingsStore = Depends(get_settings_store),
) -> dict[str, Any]:
"""Return recent action history for a single machine."""
machine = _resolve_machine(store, machine_id)
actions = store.list_machine_actions(machine["id"], limit=limit, action=action, status=status)
return {"items": actions, "total": len(actions)}
@router.get("/disk")
def get_disk_space(
machine_id: str | None = Query(default=None),
store: SettingsStore = Depends(get_settings_store),
) -> dict[str, Any]:
"""Return disk space for the configured path of a given machine."""
machine = _resolve_machine(store, machine_id)
app_settings = get_settings()
path = str(machine.get("media_root") or app_settings.media_root or "/")
logger.info("Monitoring disk requested machine_id=%s path=%s", machine["id"], path)
return run_machine_operation(
machine,
store,
f"disk lookup for {path}",
lambda client: disk_space(client, path),
)
@router.get("/prometheus-targets")
def get_prometheus_targets(store: SettingsStore = Depends(get_settings_store)) -> list[dict[str, Any]]:
"""Return Prometheus file-SD targets for remote Node Exporters.
@@ -12,7 +12,7 @@ from pydantic import BaseModel, Field
from media_library_viewer_api.clients.ssh import RemoteSSHClient
from media_library_viewer_api.config import get_settings
from media_library_viewer_api.dependencies import get_monitoring_poller, get_settings_store
from media_library_viewer_api.dependencies import get_settings_store
from media_library_viewer_api.services.db_maintenance import remove_sqlite_database
from media_library_viewer_api.services.known_hosts import has_known_host
from media_library_viewer_api.services.media_index import MediaIndex
@@ -195,13 +195,8 @@ def post_machine(
) -> dict[str, Any]:
saved = store.upsert_machine(machine.model_dump(exclude_none=True), machine.id)
_write_prometheus_targets(store)
poller = get_monitoring_poller()
try:
saved_machine = MonitoringMachineInput.model_validate(saved)
_validate_saved_machine_ssh(saved_machine, store)
finally:
poller.start()
poller.kick()
saved_machine = MonitoringMachineInput.model_validate(saved)
_validate_saved_machine_ssh(saved_machine, store)
return saved
@@ -215,13 +210,8 @@ def put_machine(
raise HTTPException(status_code=404, detail="Machine not found")
saved = store.upsert_machine(machine.model_dump(exclude_none=True), machine_id)
_write_prometheus_targets(store)
poller = get_monitoring_poller()
try:
saved_machine = MonitoringMachineInput.model_validate(saved)
_validate_saved_machine_ssh(saved_machine, store)
finally:
poller.start()
poller.kick()
saved_machine = MonitoringMachineInput.model_validate(saved)
_validate_saved_machine_ssh(saved_machine, store)
return saved
@@ -1,208 +0,0 @@
"""Shared monitoring action helpers.
The router and the background poller both use these helpers so machine
operations are recorded consistently whether they were triggered by a user
request or by the backend's scheduled polling loop.
"""
from __future__ import annotations
import json
import logging
import shlex
import time
import uuid
from typing import Any, Callable
from fastapi import HTTPException
from media_library_viewer_api.clients.local import LocalCommandClient
from media_library_viewer_api.clients.ssh import RemoteSSHClient
from media_library_viewer_api.config import get_settings
from media_library_viewer_api.observability import record_ssh_command
from media_library_viewer_api.services.settings_store import SettingsStore
logger = logging.getLogger(__name__)
def build_machine_client(machine: dict[str, Any], store: SettingsStore):
"""Build the appropriate command client for a machine definition."""
mode = str(machine.get("mode") or "local").strip().lower()
if mode == "local":
return LocalCommandClient()
key_directory = str(machine.get("key_directory") or "").strip()
key_name = str(machine.get("key_name") or "").strip()
key_path = f"{key_directory}/{key_name}" if key_directory and key_name else None
private_key = str(machine.get("ssh_private_key") or "")
passphrase = str(machine.get("ssh_private_key_passphrase") or "")
ssh_key_id = str(machine.get("ssh_key_id") or "").strip()
if ssh_key_id:
ssh_key = store.get_ssh_key(ssh_key_id)
if ssh_key:
private_key = str(ssh_key.get("private_key") or private_key)
passphrase = str(ssh_key.get("passphrase") or passphrase)
settings = get_settings()
return RemoteSSHClient(
host=str(machine.get("host") or ""),
username=str(machine.get("username") or ""),
port=int(machine.get("port") or 22),
key_filename=key_path,
private_key=private_key or None,
private_key_passphrase=passphrase or None,
password=str(machine.get("password") or "") or None,
known_hosts_path=str(settings.ssh_known_hosts_file),
)
def disk_space(client: Any, path: str = "/") -> dict[str, Any]:
"""Return df information for the filesystem containing ``path``.
Works against any client with a ``run`` method (local shell or SSH).
"""
command = (
"df -P -B1 -- " + shlex.quote(path or "/") + " | awk 'NR==2 {printf "
'"{\\"filesystem\\":\\"%s\\",\\"size\\":%s,"'
'"\\"used\\":%s,\\"available\\":%s,"'
'"\\"used_pct\\":\\"%s\\",\\"mount\\":\\"%s\\"}", "'
"$1,$2,$3,$4,$5,$6}'"
)
logger.debug("Reading disk space for path=%s", path)
result = client.run(command, timeout=20)
if result.exit_status != 0 or not result.stdout.strip():
logger.warning("Failed to read disk space for %s: %s", path, result.stderr or result.stdout)
raise RuntimeError(result.stderr or result.stdout or "failed to read disk space")
data = json.loads(result.stdout)
logger.info("Disk space path=%s mount=%s used_pct=%s", path, data.get("mount"), data.get("used_pct"))
return data
def summarize_operation_result(action: str, result: Any) -> str:
"""Turn an operation result into a compact human-readable summary."""
if result is None:
return action
if isinstance(result, str):
text = result.strip().splitlines()[0] if result.strip() else action
return text[:200]
if isinstance(result, list):
return f"{action}: {len(result)} item(s)"
if isinstance(result, dict):
if action.startswith("disk lookup"):
used_pct = result.get("used_pct")
mount = result.get("mount") or result.get("filesystem")
return f"disk {mount or ''} used {used_pct or '?'}".strip()
if "message" in result and isinstance(result["message"], str):
return result["message"][:200]
return json_compact(result)
return action
def json_compact(value: Any) -> str:
try:
text = json.dumps(value, sort_keys=True, default=str)
return text[:200]
except Exception:
return str(value)[:200]
def run_machine_operation(
machine: dict[str, Any],
store: SettingsStore,
action: str,
callback: Callable[[Any], Any],
*,
summarize: Callable[[Any], str] | None = None,
request_id: str = "",
raise_http: bool = True,
client: Any | None = None,
) -> Any:
"""Run a machine operation, record history, and optionally raise on failure."""
started = time.perf_counter()
if client is None:
client = build_machine_client(machine, store)
try:
result = callback(client)
duration_ms = int((time.perf_counter() - started) * 1000)
record_ssh_command(
machine_id=machine.get("id") or "unknown",
action=action,
status="ok",
duration_seconds=duration_ms / 1000.0,
)
store.record_machine_action(
machine,
action,
"ok",
duration_ms=duration_ms,
request_id=request_id,
message=(summarize(result) if summarize else summarize_operation_result(action, result)),
)
return result
except HTTPException:
raise
except Exception as exc: # pragma: no cover - transport/network fallback
duration_ms = int((time.perf_counter() - started) * 1000)
record_ssh_command(
machine_id=machine.get("id") or "unknown",
action=action,
status="error",
duration_seconds=duration_ms / 1000.0,
)
logger.exception(
"Monitoring %s failed machine_id=%s machine_name=%s",
action,
machine["id"],
machine["name"],
)
error_text = str(exc)
store.record_machine_action(
machine,
action,
"error",
duration_ms=duration_ms,
request_id=request_id,
error=error_text,
)
if not raise_http:
return None
status_code = 503 if machine.get("mode") == "local" else 502
raise HTTPException(
status_code=status_code,
detail=f"{machine['name']}: {action} failed: {exc}",
) from exc
def poll_machine_snapshot(
machine: dict[str, Any],
store: SettingsStore,
*,
metrics_limit: int = 70_000,
request_id: str | None = None,
) -> dict[str, Any]:
"""Collect a backend-scheduled snapshot for a machine.
The legacy POSIX collector has been removed; this now records a lightweight
disk-space lookup on the same schedule so action history stays useful.
"""
request_id = request_id or f"poll:{machine.get('id') or uuid.uuid4().hex}"
results: dict[str, Any] = {"request_id": request_id, "machine_id": machine.get("id"), "actions": []}
client = build_machine_client(machine, store)
settings = get_settings()
path = str(machine.get("media_root") or settings.media_root or "/")
disk = run_machine_operation(
machine,
store,
f"disk lookup for {path}",
lambda client: disk_space(client, path),
request_id=request_id,
raise_http=False,
client=client,
)
results["disk_mount"] = (disk or {}).get("mount") if isinstance(disk, dict) else None
results["actions"].append("disk lookup")
return results
@@ -1,197 +0,0 @@
"""Background poller for monitoring machine snapshots.
The poller runs entirely inside the backend. It periodically reads the defined
machines, collects a small snapshot from each enabled machine over SSH or local
shell execution, and stores the resulting history rows in the settings DB.
This keeps the Monitoring page populated without any daemon or agent running on
the remote machines.
"""
from __future__ import annotations
import logging
import threading
import time
from dataclasses import dataclass
from typing import Any
from media_library_viewer_api.config import get_settings
from media_library_viewer_api.services.monitoring_actions import poll_machine_snapshot
from media_library_viewer_api.services.settings_store import SettingsStore, get_settings_store
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class PollerConfig:
interval_seconds: int = 300
initial_delay_seconds: int = 20
metrics_limit: int = 70_000
retention_days: int = 30
class MonitoringPoller:
"""Single-worker background poller for monitoring snapshots."""
def __init__(self) -> None:
self._thread: threading.Thread | None = None
self._stop_event = threading.Event()
self._lock = threading.Lock()
self._last_run_at: float | None = None
self._last_success_at: float | None = None
self._last_error: str = ""
self._last_cycle_ms: int | None = None
self._poll_count = 0
self._error_count = 0
def _config(self) -> PollerConfig:
settings = get_settings()
return PollerConfig(
interval_seconds=max(30, int(getattr(settings, "monitoring_poll_interval_seconds", 300) or 300)),
initial_delay_seconds=max(0, int(getattr(settings, "monitoring_poll_initial_delay_seconds", 20) or 20)),
metrics_limit=70_000,
retention_days=max(1, int(getattr(settings, "monitoring_action_retention_days", 30) or 30)),
)
def start(self) -> None:
"""Start the background worker if it is not already running."""
with self._lock:
if self._thread and self._thread.is_alive():
return
self._stop_event.clear()
self._thread = threading.Thread(target=self._run, name="monitoring-poller", daemon=True)
self._thread.start()
logger.info("Monitoring poller started")
def kick(self) -> None:
"""Run one immediate snapshot cycle in the background."""
store = get_settings_store()
config = self._config()
threading.Thread(
target=self._run_cycle,
args=(store, config),
name="monitoring-poller-kick",
daemon=True,
).start()
logger.info("Monitoring poller kick requested")
def stop(self, timeout: float = 5.0) -> None:
"""Stop the worker thread and wait briefly for shutdown."""
with self._lock:
thread = self._thread
if not thread:
return
self._stop_event.set()
thread.join(timeout=timeout)
if thread.is_alive():
logger.warning("Monitoring poller did not stop within %.1fs", timeout)
else:
logger.info("Monitoring poller stopped")
with self._lock:
if self._thread is thread:
self._thread = None
def status(self) -> dict[str, Any]:
"""Return a small status snapshot for diagnostics and tests."""
with self._lock:
return {
"worker_running": bool(self._thread and self._thread.is_alive()),
"stop_requested": self._stop_event.is_set(),
"last_run_at": self._last_run_at,
"last_success_at": self._last_success_at,
"last_error": self._last_error,
"last_cycle_ms": self._last_cycle_ms,
"poll_count": self._poll_count,
"error_count": self._error_count,
}
def snapshot(self) -> dict[str, Any]:
"""Return status plus the active polling configuration."""
data = self.status()
config = self._config()
data.update(
{
"interval_seconds": config.interval_seconds,
"initial_delay_seconds": config.initial_delay_seconds,
"retention_days": config.retention_days,
}
)
return data
def _run_cycle(self, store: SettingsStore, config: PollerConfig) -> None:
start = time.perf_counter()
machines = store.list_machines()
enabled = [machine for machine in machines if machine.get("enabled")]
logger.info("Monitoring poll cycle starting enabled_machines=%s", len(enabled))
cycle_errors = 0
for machine in enabled:
if self._stop_event.is_set():
break
try:
snapshot = poll_machine_snapshot(
machine,
store,
metrics_limit=config.metrics_limit,
request_id=f"poll:{machine['id']}:{int(time.time())}",
)
logger.info(
"Monitoring poll snapshot machine_id=%s request_id=%s disk_mount=%s actions=%s",
machine["id"],
snapshot.get("request_id"),
snapshot.get("disk_mount"),
snapshot.get("actions"),
)
except Exception:
cycle_errors += 1
logger.exception(
"Monitoring poll snapshot failed machine_id=%s machine_name=%s", machine["id"], machine["name"]
)
retention_seconds = config.retention_days * 24 * 60 * 60
cutoff_ts = int(time.time()) - retention_seconds
removed = store.prune_machine_actions(cutoff_ts)
if removed:
logger.info("Pruned %s old monitoring action rows older than %s", removed, cutoff_ts)
duration_ms = int((time.perf_counter() - start) * 1000)
with self._lock:
self._last_run_at = time.time()
self._last_cycle_ms = duration_ms
self._poll_count += 1
if cycle_errors:
self._error_count += cycle_errors
self._last_error = f"{cycle_errors} machine(s) failed"
else:
self._last_success_at = self._last_run_at
self._last_error = ""
logger.info(
"Monitoring poll cycle complete enabled_machines=%s errors=%s duration_ms=%s removed_rows=%s",
len(enabled),
cycle_errors,
duration_ms,
removed,
)
def _run(self) -> None:
config = self._config()
if config.initial_delay_seconds:
logger.info("Monitoring poller initial delay=%ss", config.initial_delay_seconds)
if self._stop_event.wait(config.initial_delay_seconds):
return
store = get_settings_store()
while not self._stop_event.is_set():
try:
self._run_cycle(store, config)
except Exception:
with self._lock:
self._last_error = "poller cycle failed"
self._error_count += 1
logger.exception("Monitoring poller cycle failed")
if self._stop_event.wait(config.interval_seconds):
break
_MONITORING_POLLER = MonitoringPoller()
def get_monitoring_poller() -> MonitoringPoller:
return _MONITORING_POLLER
@@ -85,25 +85,10 @@ class SettingsStore:
"""
)
conn.execute("CREATE INDEX IF NOT EXISTS idx_monitoring_machines_mode ON monitoring_machines(mode)")
conn.execute(
"""
CREATE TABLE IF NOT EXISTS monitoring_machine_actions (
id TEXT PRIMARY KEY,
machine_id TEXT NOT NULL,
machine_name TEXT NOT NULL,
mode TEXT NOT NULL,
action TEXT NOT NULL,
status TEXT NOT NULL,
created_at INTEGER NOT NULL,
duration_ms INTEGER NOT NULL,
request_id TEXT NOT NULL,
message TEXT NOT NULL,
error TEXT NOT NULL,
stdout_tail TEXT NOT NULL,
stderr_tail TEXT NOT NULL
)
"""
)
# The legacy SSH-scraping monitor (MonitoringPoller) was decommissioned;
# metrics now live in Prometheus/node_exporter/Grafana. Drop the orphan
# table on startup so existing databases get a clean slate.
conn.execute("DROP TABLE IF EXISTS monitoring_machine_actions")
conn.execute(
"""
CREATE TABLE IF NOT EXISTS ssh_keys (
@@ -178,18 +163,6 @@ class SettingsStore:
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_dashboard_shortcuts_type ON dashboard_shortcuts(shortcut_type)"
)
conn.execute(
"""
CREATE INDEX IF NOT EXISTS idx_monitoring_machine_actions_machine_time
ON monitoring_machine_actions(machine_id, created_at DESC)
"""
)
conn.execute(
"""
CREATE INDEX IF NOT EXISTS idx_monitoring_machine_actions_action_status
ON monitoring_machine_actions(action, status)
"""
)
conn.execute("""
CREATE TABLE IF NOT EXISTS backup_jobs (
id TEXT PRIMARY KEY,
@@ -563,88 +536,6 @@ class SettingsStore:
with self.connect() as conn:
conn.execute("DELETE FROM monitoring_machines WHERE id = ?", (machine_id,))
def record_machine_action(
self,
machine: dict[str, Any],
action: str,
status: str,
*,
duration_ms: int,
request_id: str = "",
message: str = "",
error: str = "",
stdout_tail: str = "",
stderr_tail: str = "",
) -> None:
"""Store a compact action history row for a machine operation."""
self.init_schema()
now = int(time.time())
with self.connect() as conn:
conn.execute(
"""
INSERT INTO monitoring_machine_actions
(
id, machine_id, machine_name, mode, action, status,
created_at, duration_ms, request_id, message, error,
stdout_tail, stderr_tail
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
uuid.uuid4().hex,
str(machine.get("id") or ""),
str(machine.get("name") or ""),
str(machine.get("mode") or "local"),
action,
status,
now,
duration_ms,
request_id,
message,
error,
stdout_tail,
stderr_tail,
),
)
def list_machine_actions(
self,
machine_id: str,
*,
limit: int = 20,
action: str | None = None,
status: str | None = None,
) -> list[dict[str, Any]]:
self.init_schema()
clauses = ["machine_id = ?"]
params: list[Any] = [machine_id]
if action:
clauses.append("action = ?")
params.append(action)
if status:
clauses.append("status = ?")
params.append(status)
sql = (
"SELECT machine_id, machine_name, mode, action, status, "
"created_at, duration_ms, request_id, message, error, stdout_tail, stderr_tail "
f"FROM monitoring_machine_actions WHERE {' AND '.join(clauses)} "
"ORDER BY created_at DESC LIMIT ?"
)
params.append(max(1, min(int(limit), 200)))
with self.connect() as conn:
rows = conn.execute(sql, params).fetchall()
return [dict(row) for row in rows]
def prune_machine_actions(self, older_than_ts: int) -> int:
"""Delete action history rows older than the given timestamp."""
self.init_schema()
with self.connect() as conn:
cur = conn.execute(
"DELETE FROM monitoring_machine_actions WHERE created_at < ?",
(int(older_than_ts),),
)
return int(cur.rowcount or 0)
@staticmethod
def _private_key_summary(private_key: str) -> dict[str, str]:
if not private_key:
-28
View File
@@ -598,34 +598,6 @@ class TestJobs:
class TestMonitoring:
def _ensure_machine(self):
store = app.dependency_overrides[get_settings_store]()
if not store.list_machines():
store.upsert_machine(
{
"name": "Test Machine",
"mode": "ssh",
"enabled": True,
"services": ["monitoring", "files", "jellyfin"],
"host": "test-host",
"username": "test-user",
}
)
def test_disk(self, test_client, mock_ssh):
self._ensure_machine()
mock_ssh.run.return_value = CommandResult(
command="df ...",
exit_status=0,
stdout='{"filesystem":"/dev/sda1","size":1000000000,"used":500000000,"available":500000000,"used_pct":"50%","mount":"/"}',
stderr="",
)
with patch("media_library_viewer_api.services.monitoring_actions.build_machine_client", return_value=mock_ssh):
response = test_client.get("/api/monitoring/disk")
assert response.status_code == 200
data = response.json()
assert data["used_pct"] == "50%"
def test_prometheus_targets_empty(self, test_client):
response = test_client.get("/api/monitoring/prometheus-targets")
assert response.status_code == 200
-36
View File
@@ -1,36 +0,0 @@
from unittest.mock import MagicMock, patch
from media_library_viewer_api.services.monitoring_actions import poll_machine_snapshot
def test_poll_machine_snapshot_records_disk_lookup():
store = MagicMock()
machine = {
"id": "local",
"name": "This machine",
"mode": "local",
"media_root": "/srv/media",
}
with (
patch(
"media_library_viewer_api.services.monitoring_actions.build_machine_client",
return_value=object(),
) as build_client,
patch(
"media_library_viewer_api.services.monitoring_actions.disk_space",
return_value={"mount": "/srv/media", "used_pct": "12.5%"},
) as disk_fn,
):
result = poll_machine_snapshot(machine, store, metrics_limit=123, request_id="poll:test")
assert result["request_id"] == "poll:test"
assert result["disk_mount"] == "/srv/media"
assert result["actions"] == ["disk lookup"]
build_client.assert_called_once_with(machine, store)
disk_fn.assert_called_once_with(build_client.return_value, "/srv/media")
assert store.record_machine_action.call_count == 1
recorded_action = store.record_machine_action.call_args
assert recorded_action.args[1] == "disk lookup for /srv/media"
assert recorded_action.kwargs["request_id"] == "poll:test"
assert recorded_action.args[2] == "ok"
+2 -1
View File
@@ -16,7 +16,7 @@ services:
SSH_KNOWN_HOSTS_PATH: /app/backend/.cache/known_hosts
PROMETHEUS_FILE_SD_DIR: /app/backend/.cache/prometheus-file-sd
ALERTMANAGER_URL: ${ALERTMANAGER_URL:-http://alertmanager:9093}
GRAFANA_URL: ${GRAFANA_URL:-http://grafana:3000}
ALERTMANAGER_WEBHOOK_URL: ${ALERTMANAGER_WEBHOOK_URL:-}
ports:
- "8000:8000"
volumes:
@@ -37,6 +37,7 @@ services:
VITE_API_URL: "/api"
VITE_OIDC_ENABLED: "false"
VITE_DEV_API_PROXY_TARGET: "http://backend:8000"
VITE_GRAFANA_URL: "http://localhost:3000"
ports:
- "5173:5173"
volumes:
+2 -1
View File
@@ -27,7 +27,7 @@ services:
SSH_KNOWN_HOSTS_PATH: /app/backend/.cache/known_hosts
PROMETHEUS_FILE_SD_DIR: ${PROMETHEUS_FILE_SD_DIR:-/app/backend/.cache/prometheus-file-sd}
ALERTMANAGER_URL: ${ALERTMANAGER_URL:-http://alertmanager:9093}
GRAFANA_URL: ${GRAFANA_URL:-http://grafana:3000}
ALERTMANAGER_WEBHOOK_URL: ${ALERTMANAGER_WEBHOOK_URL:-}
volumes:
- ${BACKEND_CACHE_DIR:-./backend-cache}:/app/backend/.cache
restart: unless-stopped
@@ -69,6 +69,7 @@ services:
VITE_OIDC_REDIRECT_URI: ${VITE_OIDC_REDIRECT_URI:?set VITE_OIDC_REDIRECT_URI}
VITE_OIDC_POST_LOGOUT_REDIRECT_URI: ${VITE_OIDC_POST_LOGOUT_REDIRECT_URI:?set VITE_OIDC_POST_LOGOUT_REDIRECT_URI}
VITE_DEV_API_PROXY_TARGET: ${VITE_DEV_API_PROXY_TARGET:-http://backend:8000}
VITE_GRAFANA_URL: ${VITE_GRAFANA_URL:-https://grafana.example.com}
VITE_APP_VERSION: ${APP_VERSION:-0.1.0}
VITE_APP_BUILD_INFO: ${APP_BUILD_INFO:-dev}
depends_on:
+3 -3
View File
@@ -105,9 +105,9 @@ repo/
| `/api/dashboard/counts` | GET | `jellyfin.media_counts()` | Movie/series/episode totals |
| `/api/dashboard/libraries` | GET | `jellyfin.library_item_counts()` | Per-library breakdown |
| `/api/dashboard/now-playing` | GET | `jellyfin.active_sessions()` | Active sessions + transcode info |
| `/api/monitoring/status` | GET | `resources.resource_collector_status()` | Collector running? |
| `/api/monitoring/metrics` | GET | `resources.read_resource_metrics()` | Last-hour JSONL samples |
| `/api/monitoring/disk` | GET | `resources.disk_space()` | df for media root |
| `/api/monitoring/status` | GET | `resources.resource_collector_status()` | Collector running? *(legacy/removed)* |
| `/api/monitoring/metrics` | GET | `resources.read_resource_metrics()` | Last-hour JSONL samples *(legacy/removed)* |
| `/api/monitoring/disk` | GET | `resources.disk_space()` | df for media root *(removed 2026-06-17; metrics now in Prometheus/Grafana)* |
| `/api/monitoring/start` | POST | `resources.start_resource_collector()` | Start collector |
| `/api/monitoring/stop` | POST | `resources.stop_resource_collector()` | Stop collector |
| `/api/monitoring/restart` | POST | `resources.restart_resource_collector()` | Restart collector |
+7
View File
@@ -37,6 +37,12 @@ fully removed (web-ui-rework; see decision log 2026-06-17).
not rendered graphs.
- The legacy in-app D3 monitoring charts and the POSIX remote resource collector are
superseded by this Grafana-based model (see decision log 2026-06-13 and 2026-06-17).
- **Manage no longer scrapes its own system metrics** (decision 2026-06-17). The backend
`MonitoringPoller` (which SSH-ran `df` on every machine every 5 minutes into a local
SQLite `monitoring_machine_actions` table), the `/api/monitoring/disk`, `/poller`, and
`/machines/{id}/actions` endpoints, and the frontend `DiskSpaceCard` have been removed.
Disk/CPU/memory visibility is owned by Prometheus + node_exporter + Grafana. The
`disk_usage` **job template** in Actions remains as a manual on-demand SSH check.
### Tables
@@ -252,6 +258,7 @@ fully removed (web-ui-rework; see decision log 2026-06-17).
## Decision Log
- 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template.
- 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring``/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`.
- 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`.
- 2026-06-13 (Phase 1): Added Prometheus, Loki, Grafana Alloy, Grafana, Alertmanager, and Node Exporter services to `docker-compose.yml` and `docker-compose.dev.yml`. Provisioned Grafana datasources and an initial `Manage Overview` dashboard as code. Configured Alloy to tail Docker logs and ship to Loki. Added Grafana generic OAuth configuration via `monitoring/grafana/grafana.ini` and a dedicated Traefik host rule. Added Alertmanager email routing with env-var interpolation. Added `/grafana` proxy to the Vite dev server for iframe embedding.
+16 -5
View File
@@ -60,10 +60,16 @@ The existing POSIX remote collector will be removed, and the Python backup alert
### Metrics
- `backend/src/media_library_viewer_api/clients/resources.py` deploys a POSIX shell collector to `/tmp` on each remote machine.
- The collector samples `/proc/stat`, `/proc/meminfo`, `/proc/net/dev`, and `/sys/block/*/stat` every 10s and writes JSONL to `/tmp/media_library_viewer_metrics.jsonl`.
- `MonitoringPoller` (`monitoring_poller.py`) runs every 5 minutes, reads the remote JSONL, and stores snapshots in SQLite (`monitoring_machine_actions`).
- Retention defaults to 30 days with periodic pruning.
> **Historical note (2026-06-17):** The legacy Manage-side `MonitoringPoller` that
> SSH-scraped `/proc` + `df` into a local SQLite table (`monitoring_machine_actions`)
> has been **decommissioned**. System metrics now live entirely in the external
> observability stack: `node_exporter` on each machine is scraped by **Prometheus**
> and visualised in **Grafana** (see the standalone `docker-compose.observability.yml`
> stack). Manage is a thin dashboard: it surfaces Alertmanager alerts + Prometheus
> target health + Grafana deep-links, and does not collect or store its own metrics.
- `main.py` has a `log_requests` middleware that emits method, path, client IP, status code, and elapsed time.
- Frontend uses standard `console.log` / browser dev tools; no server-side log aggregation.
### Alerting
@@ -210,7 +216,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
- Manage API overview (request rate, latency, errors).
- Manage operations (SSH commands, media index builds, mail queue).
- Backup runs and alert history.
- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and ` kiosk` mode.
- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and `kiosk` mode.
### Manage React UI
@@ -309,6 +315,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
- [x] Wire Grafana OAuth to Authentik.
**Phase 1 files**:
- `monitoring/prometheus/prometheus.yml`
- `monitoring/prometheus/rules/backup_alerts.yml`
- `monitoring/loki/loki.yml`
@@ -336,6 +343,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
- [x] Remove POSIX collector fallback. The legacy collector code in `backend/src/media_library_viewer_api/clients/resources.py` has been deleted, the collector control endpoints were removed from `routers/monitoring.py`, and `disk_space` was relocated to `services/monitoring_actions.py` as a lightweight SSH/local helper. Metrics are now sourced exclusively from Prometheus/Node Exporter.
**Phase 2 files**:
- `backend/src/media_library_viewer_api/jobs.py` (Node Exporter job templates).
- `backend/src/media_library_viewer_api/routers/settings.py` (machine input fields + target regeneration).
- `backend/src/media_library_viewer_api/services/settings_store.py` (machine persistence fields).
@@ -364,6 +372,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
- [x] Added tests for the Alertmanager endpoints and the backup success gauge.
**Phase 3 files**:
- `backend/src/media_library_viewer_api/routers/monitoring.py` (`/alerts` and `/alertmanager-status` endpoints).
- `backend/src/media_library_viewer_api/observability.py` (`BACKUP_RUNS_LAST_SUCCESS` gauge + updated `record_backup_run`).
- `backend/src/media_library_viewer_api/routers/backups.py` (pass `success=True` to `record_backup_run` on successful reports).
@@ -386,6 +395,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
- [x] Wire the new `/observability` route into `App.tsx` and the sidebar navigation.
**Phase 4 files**:
- `frontend/src/components/ObservabilityPage.tsx` (page component).
- `frontend/src/hooks/useObservability.ts` (React Query hooks).
- `frontend/src/api/client.ts` (API client functions).
@@ -406,6 +416,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
- [ ] Optional: add OpenTelemetry Collector as a translation layer for traces later.
**Phase 5 files**:
- `docker-compose.yml` and `docker-compose.dev.yml` (health checks, resource limits, `depends_on` conditions).
- `monitoring/prometheus/prometheus.yml` (additional scrape jobs for observability services).
- `monitoring/prometheus/rules/backup_alerts.yml` (renamed scope to include observability health alerts).
+3
View File
@@ -15,6 +15,7 @@ ARG VITE_OIDC_SCOPE=openid profile email
ARG VITE_OIDC_REDIRECT_URI=
ARG VITE_OIDC_POST_LOGOUT_REDIRECT_URI=
ARG VITE_DEV_API_PROXY_TARGET=http://backend:8000
ARG VITE_GRAFANA_URL=https://grafana.example.com
ARG VITE_APP_VERSION=0.1.0
ARG VITE_APP_BUILD_INFO=dev
@@ -26,6 +27,7 @@ ENV VITE_API_URL=${VITE_API_URL} \
VITE_OIDC_REDIRECT_URI=${VITE_OIDC_REDIRECT_URI} \
VITE_OIDC_POST_LOGOUT_REDIRECT_URI=${VITE_OIDC_POST_LOGOUT_REDIRECT_URI} \
VITE_DEV_API_PROXY_TARGET=${VITE_DEV_API_PROXY_TARGET} \
VITE_GRAFANA_URL=${VITE_GRAFANA_URL} \
VITE_APP_VERSION=${VITE_APP_VERSION} \
VITE_APP_BUILD_INFO=${VITE_APP_BUILD_INFO}
@@ -50,6 +52,7 @@ COPY frontend/ ./
ENV VITE_API_URL=/api \
VITE_OIDC_ENABLED=false \
VITE_DEV_API_PROXY_TARGET=http://backend:8000 \
VITE_GRAFANA_URL=http://localhost:3000 \
VITE_APP_VERSION=0.1.0 \
VITE_APP_BUILD_INFO=dev
-78
View File
@@ -1,78 +0,0 @@
import { Card, CardContent } from "@/components/ui/card";
import { Progress } from "@/components/ui/progress";
interface Props {
used: number;
available: number;
size: number;
usedPct: string;
}
function formatBytes(bytes: number): string {
if (!bytes || bytes === 0) return "0 B";
const units = ["B", "KB", "MB", "GB", "TB"];
let value = bytes;
let unitIdx = 0;
while (value >= 1000 && unitIdx < units.length - 1) {
value /= 1000;
unitIdx++;
}
return `${value.toFixed(1)} ${units[unitIdx]}`;
}
/**
* Progress-bar class (full static strings so Tailwind's scanner emits them).
* `chart-2`=success, `chart-3`=warning, `destructive`=error, per design §2.3.
*/
function progressBarClass(pct: number): string {
if (pct < 70) {
return "h-3 [&_[data-slot=progress-indicator]]:bg-chart-2";
}
if (pct < 90) {
return "h-3 [&_[data-slot=progress-indicator]]:bg-chart-3";
}
return "h-3 [&_[data-slot=progress-indicator]]:bg-destructive";
}
/**
* Dashboard card that summarizes the configured media disk.
*
* Keeps the progress bar inside the card so the capacity signal, raw byte
* values, and free-space breakdown stay visually grouped. The used / free /
* total / percent breakdown is preserved verbatim from the MUI version.
*/
export function DiskSpaceCard({ used, available, size, usedPct }: Props) {
const pct = Math.max(0, Math.min(100, Number.parseFloat(usedPct) || 0));
const cells = [
{ label: "Used", value: formatBytes(used) },
{ label: "Free", value: formatBytes(available) },
{ label: "Total", value: formatBytes(size) },
];
return (
<Card className="h-full">
<CardContent className="flex flex-col gap-4">
<div className="flex flex-col gap-1">
<span className="text-sm uppercase tracking-wide text-muted-foreground">
Disk space
</span>
<span className="text-lg font-semibold">{usedPct} used</span>
</div>
<Progress value={pct} className={progressBarClass(pct)} />
<div className="grid grid-cols-1 gap-4 sm:grid-cols-3">
{cells.map((cell) => (
<div
key={cell.label}
className="flex h-full flex-col items-center justify-center gap-1 rounded-lg bg-muted/50 p-3 text-center"
>
<span className="text-xs text-muted-foreground">
{cell.label}
</span>
<span className="text-sm font-semibold">{cell.value}</span>
</div>
))}
</div>
</CardContent>
</Card>
);
}
@@ -1,22 +0,0 @@
import { describe, it, expect } from "vitest";
import { render, screen } from "@testing-library/react";
import { DiskSpaceCard } from "../DiskSpaceCard";
describe("DiskSpaceCard", () => {
it("preserves the used / free / total breakdown and percent headline", () => {
render(
<DiskSpaceCard
used={500000000000}
available={500000000000}
size={1000000000000}
usedPct="50"
/>,
);
expect(screen.getByText(/50 used/i)).toBeInTheDocument();
expect(screen.getByText("Used")).toBeInTheDocument();
expect(screen.getByText("Free")).toBeInTheDocument();
expect(screen.getByText("Total")).toBeInTheDocument();
// Disk space label
expect(screen.getByText(/disk space/i)).toBeInTheDocument();
});
});
-9
View File
@@ -276,15 +276,6 @@ export interface AppVersionInfo {
backend_label: string;
}
export interface DiskSpace {
filesystem: string;
size: number;
used: number;
available: number;
used_pct: string;
mount: string;
}
export interface MediaIndexStatus {
exists: boolean;
item_count: number;
@@ -0,0 +1,101 @@
# Plan — decommission-monitoring-poller
> Status: **DRAFT — awaiting user approval before implementation.**
> Scope: a focused backend+frontend decommission, not a full SDD change. Plan-then-implement (user-approved 2026-06-17).
> Root cause this addresses: the 2026-06-16/17 observability update externalised metrics to Prometheus+Grafana+Loki+Alertmanager, but the *legacy Manage-side SSH-scraping monitor* (the `MonitoringPoller`, `/monitoring/disk`, `/monitoring/machines/{id}/actions`, and the `monitoring_machine_actions` SQLite table) was never removed. It duplicates the new stack, drains SSH budget every 300s, and feeds nothing (its UI was deleted in `e2ad731`).
## 1. Problem
Manage runs a background thread (`MonitoringPoller`) that, every 300s, SSHes into every configured machine, runs `df`, and stores the result in its own SQLite table (`monitoring_machine_actions`, 30-day retention). After the observability update, **Prometheus already scrapes node_exporter on these machines and Grafana already owns the dashboards**. The poller is pure duplication: more SSH sessions, more stale state, a second source of truth for "disk usage," and a SQLite table that nothing reads.
The alerting side (Alertmanager proxy + `/alerts` + `/alertmanager-status` + `/alertmanager-webhook` + `/prometheus-targets` + `/machines`) already fits the new model and is untouched by this change.
## 2. Goals / non-goals
**Goals**
- Stop the duplicated SSH-scraping of system metrics.
- Remove the dead `/disk`, `/poller`, `/machines/{id}/actions` surface and the SQLite history that fed it.
- Remove the now-orphaned frontend `DiskSpaceCard` + `DiskSpace` type.
- Leave Manage a clean thin-dashboard: Alertmanager alerts + Prometheus target health + Grafana deep-links.
**Non-goals**
- Do NOT touch the Alertmanager proxy, `/prometheus-targets`, `/machines`, or `/alertmanager-webhook` — they fit the model.
- Do NOT remove the `disk_usage` **job template** in `jobs.py` (user-approved: it is a manual on-demand Actions job, not monitoring).
- Do NOT remove `node_exporter_*` fields on `MonitoringMachine` — they configure where Prometheus scrapes; that is correct and stays.
- Do NOT introduce a Prometheus query proxy / PromQL reader in this change (that was the alternative the user did not pick).
- Do NOT add new features. This is a removal.
## 3. Exact removal map (verified against source)
### Backend — delete entirely
- `backend/src/media_library_viewer_api/services/monitoring_poller.py` (the `MonitoringPoller` class, `PollerConfig`, `_MONITORING_POLLER`, `get_monitoring_poller`).
- **Verified sole callers:** `main.py` lifespan, `dependencies.py` wrapper, `routers/monitoring.py:/poller`, `routers/settings.py` (machine save → `poller.start()/kick()`).
- `backend/src/media_library_viewer_api/services/monitoring_actions.py` (the whole file: `build_machine_client`, `disk_space`, `summarize_operation_result`, `json_compact`, `run_machine_operation`, `poll_machine_snapshot`).
- **Verified:** `run_machine_operation` has exactly 2 callers (`poll_machine_snapshot` here, and `/monitoring/disk`) — both going. `tasks.py` does NOT use it. Nothing else imports the module.
- `backend/tests/test_monitoring_actions.py` (36 lines, tests `poll_machine_snapshot`).
### Backend — edit in place
- `backend/src/media_library_viewer_api/main.py` lifespan (lines ~4856): remove `monitoring_poller = get_monitoring_poller()`, `monitoring_poller.start()`, `monitoring_poller.stop()`, and the `get_monitoring_poller` import on line 16. Keep `backup_poller` and `mail_queue` intact.
- `backend/src/media_library_viewer_api/dependencies.py`: remove the `MonitoringPoller` import block (lines 2429) and the `get_monitoring_poller` wrapper (lines 254256).
- `backend/src/media_library_viewer_api/routers/monitoring.py`: remove imports of `disk_space`, `run_machine_operation`, `poll_machine_snapshot`; remove the three endpoints `/poller` (99), `/machines/{machine_id}/actions` (113), `/disk` (127). Keep `/machines`, `/prometheus-targets`, `/alerts`, `/alertmanager-status`, `/alertmanager-webhook`. Also drop the now-unused `_resolve_machine` helper if it becomes unreferenced after `/disk` and `/actions` removal (verify during impl — `/machines` does not use it).
- `backend/src/media_library_viewer_api/routers/settings.py` (lines 198204 and 218224): remove the `get_monitoring_poller()` + `poller.start()` + `poller.kick()` calls from `post_machine` and `put_machine`. Keep `write_prometheus_targets(store)` (that is the new-model target generation).
- `backend/src/media_library_viewer_api/services/settings_store.py`:
- Remove `CREATE TABLE IF NOT EXISTS monitoring_machine_actions` (lines ~90) and its two indexes (`idx_monitoring_machine_actions_machine_time`, `idx_monitoring_machine_actions_action_status`, lines ~183190) from `init_schema`.
- Remove methods `record_machine_action` (566), `list_machine_actions` (610), `prune_machine_actions` (638).
- Note: existing databases will keep the orphaned `monitoring_machine_actions` table harmlessly (no migration framework here — `init_schema` is `CREATE TABLE IF NOT EXISTS` + ad-hoc `ALTER`). A one-line `DROP TABLE IF EXISTS` can be added to `init_schema` for cleanliness; decide at impl time.
- `backend/src/media_library_viewer_api/config.py`: remove `monitoring_poll_interval_seconds` (56), `monitoring_poll_initial_delay_seconds` (57), `monitoring_action_retention_days` (58).
### Backend — tests to fix
- `backend/tests/test_api.py`:
- `TestMonitoring.test_disk` (line 615) — **remove** (tests `/api/monitoring/disk`).
- `TestMonitoring.test_prometheus_targets_empty` and `..._returns_enabled_ssh_node_exporter`**keep** (test the surviving `/prometheus-targets`).
- `TestSettingsMachines`**keep** but verify they still pass after the `poller` calls are removed from `post/put_machine`.
- `TestAlertmanager`**keep** (untouched).
- The `disk_usage` reference at line 576/586 is the **Jobs** test (`/api/jobs/run`), NOT the monitoring poller — **keep** (the job template stays).
### Frontend — delete
- `frontend/src/components/DiskSpaceCard.tsx`**verified orphaned** (only `__tests__/DiskSpaceCard.test.tsx` imports it; no page uses it).
- `frontend/src/components/__tests__/DiskSpaceCard.test.tsx`.
- `frontend/src/types/index.ts` `DiskSpace` interface (line 279) — remove after confirming no importer (grep shows none outside the type file).
### Docs
- `AGENTS.md` line 25 ("starts the mail queue and monitoring poller") → "...starts the mail queue and backup alert poller."
- `docs/monitoring-logging-design.md` line 65 (describes the poller) → update or strike the poller paragraph.
- `docs/MIGRATION_PLAN.md` line 110 (`/api/monitoring/disk` row) → remove the row.
- `docs/REQUIREMENTS.md` → add a note that Manage-side system-metric scraping is retired in favour of the external observability stack.
- `docs/superpowers/specs/2026-05-11-backup-monitoring-design.md` is a historical spec; leave as-is (it is an archived design doc).
## 4. Slice plan (≤400 lines each, build+pytest green per slice)
1. **Slice 1 — Backend removal (endpoints + poller + actions + store + config).** Delete `monitoring_poller.py`, `monitoring_actions.py`, `test_monitoring_actions.py`; edit `main.py`, `dependencies.py`, `routers/monitoring.py`, `routers/settings.py`, `settings_store.py`, `config.py`; fix `test_api.py` (`test_disk` removed, `TestSettingsMachines` re-checked). Gate: `cd backend && PYTHONPATH=src pytest`.
2. **Slice 2 — Frontend orphan removal.** Delete `DiskSpaceCard.tsx` + its test + `DiskSpace` type. Gate: `cd frontend && npm run build && npm run lint && npm test`.
3. **Slice 3 — Docs.** `AGENTS.md`, `docs/monitoring-logging-design.md`, `docs/MIGRATION_PLAN.md`, `docs/REQUIREMENTS.md`. Gate: none (docs); commit standalone.
Estimated total: ~500700 lines deleted, ~50100 added (edits). Each slice well under 400.
## 5. Risks & verification
- **Hidden caller of `run_machine_operation` / `poll_machine_snapshot`**: mitigated — grep shows exactly the callers listed; re-grep at slice-1 start.
- **`TestSettingsMachines` breakage** once `poller.start()/kick()` is removed from `post/put_machine`: those tests mock `write_prometheus_targets` and don't assert on the poller; should pass. If they reference `get_monitoring_poller`, fix by dropping the assertion.
- **Orphaned SQLite table on existing DBs**: harmless (empty, unused). Optional `DROP TABLE IF EXISTS monitoring_machine_actions` in `init_schema` for cleanliness.
- **No browser smoke**: same caveat as the UI rework; backend covered by pytest.
- **`_resolve_machine` in monitoring.py** may become unused after `/disk` + `/actions` removal; remove if so.
## 6. Acceptance
- `cd backend && PYTHONPATH=src pytest` green (with `test_disk` + `test_monitoring_actions.py` removed).
- `grep -rnE 'MonitoringPoller|poll_machine_snapshot|/monitoring/disk|monitoring_machine_actions|monitoring_poll_interval_seconds|DiskSpaceCard' backend/ frontend/src/` → only historical/docs hits (spec.md archive is fine).
- `cd frontend && npm run build && npm run lint && npm test` green.
- Docs updated to reflect Manage no longer scrapes its own metrics.
## 7. Open questions for the user (none blocking, defaults shown)
- Q1. Existing DBs' orphaned `monitoring_machine_actions` table — (a) add `DROP TABLE IF EXISTS` to `init_schema` for a clean slate [default], or (b) leave it harmless?
- Q2. Commit/PR mechanics — same as the UI rework (commit per slice, no push until you say)?