diff --git a/AGENTS.md b/AGENTS.md index 7f24131..02405ea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,12 +1,14 @@ # AGENTS.md ## Layout + - Current app is `backend/` (FastAPI) plus `frontend/` (Vite React); ignore Streamlit-era commands in `CONTRIBUTING.md`. - Backend entrypoint: `backend/src/media_library_viewer_api/main.py` (`media_library_viewer_api.main:app`). - Frontend entrypoint: `frontend/src/main.tsx`. - Backend uses a `src/` layout; tests live in `backend/tests/`. ## Commands + - Backend setup: `cd backend && python -m venv .venv && source .venv/bin/activate && pip install -e '.[dev]'` - Backend run: `uvicorn media_library_viewer_api.main:app --reload --port 8000`; if not installed, use `PYTHONPATH=src uvicorn media_library_viewer_api.main:app --reload --port 8000`. - Backend tests: run `pytest` from `backend/`; focused checks can use `pytest tests/test_api.py` or `pytest -k `; if the package is not installed, use `PYTHONPATH=src pytest`. @@ -17,12 +19,13 @@ - Production stack: `docker compose up --build` ## Repo-Specific Gotchas + - Root compose files rely on environment-variable interpolation, not `env_file`; export required values before running them. - Production compose needs the host/cert and OIDC variables from `docker-compose.yml` (`BACKEND_APP_HOST`, `FRONTEND_APP_HOST`, `CERT_RESOLVER`, and the frontend OIDC vars). - Dev compose runs with auth off and does not need SSH key material unless you add remote SSH machines. - `backend_cache` persists the media index and the managed `known_hosts` file. - SSH host-key checking is strict, but the first successful connect records the host key into backend-managed `known_hosts`. -- Backend startup validates auth settings, rebuilds managed `known_hosts`, and starts the mail queue and monitoring poller. +- Backend startup validates auth settings, rebuilds managed `known_hosts`, and starts the mail queue and backup alert poller. - Machine-level settings now own Jellyfin/Jellyseerr/SSH config; the backend seeds a local machine automatically. - Remote job templates live in `backend/src/media_library_viewer_api/jobs.py`; keep shell quoting intact. - Backend Ruff config is in `backend/pyproject.toml` and uses line length 120 with Python 3.11. diff --git a/docs/MIGRATION_PLAN.md b/docs/MIGRATION_PLAN.md index a505b9e..964aad8 100644 --- a/docs/MIGRATION_PLAN.md +++ b/docs/MIGRATION_PLAN.md @@ -105,9 +105,9 @@ repo/ | `/api/dashboard/counts` | GET | `jellyfin.media_counts()` | Movie/series/episode totals | | `/api/dashboard/libraries` | GET | `jellyfin.library_item_counts()` | Per-library breakdown | | `/api/dashboard/now-playing` | GET | `jellyfin.active_sessions()` | Active sessions + transcode info | -| `/api/monitoring/status` | GET | `resources.resource_collector_status()` | Collector running? | -| `/api/monitoring/metrics` | GET | `resources.read_resource_metrics()` | Last-hour JSONL samples | -| `/api/monitoring/disk` | GET | `resources.disk_space()` | df for media root | +| `/api/monitoring/status` | GET | `resources.resource_collector_status()` | Collector running? *(legacy/removed)* | +| `/api/monitoring/metrics` | GET | `resources.read_resource_metrics()` | Last-hour JSONL samples *(legacy/removed)* | +| `/api/monitoring/disk` | GET | `resources.disk_space()` | df for media root *(removed 2026-06-17; metrics now in Prometheus/Grafana)* | | `/api/monitoring/start` | POST | `resources.start_resource_collector()` | Start collector | | `/api/monitoring/stop` | POST | `resources.stop_resource_collector()` | Stop collector | | `/api/monitoring/restart` | POST | `resources.restart_resource_collector()` | Restart collector | diff --git a/docs/REQUIREMENTS.md b/docs/REQUIREMENTS.md index c5e427f..cdad3ec 100644 --- a/docs/REQUIREMENTS.md +++ b/docs/REQUIREMENTS.md @@ -37,6 +37,12 @@ fully removed (web-ui-rework; see decision log 2026-06-17). not rendered graphs. - The legacy in-app D3 monitoring charts and the POSIX remote resource collector are superseded by this Grafana-based model (see decision log 2026-06-13 and 2026-06-17). +- **Manage no longer scrapes its own system metrics** (decision 2026-06-17). The backend + `MonitoringPoller` (which SSH-ran `df` on every machine every 5 minutes into a local + SQLite `monitoring_machine_actions` table), the `/api/monitoring/disk`, `/poller`, and + `/machines/{id}/actions` endpoints, and the frontend `DiskSpaceCard` have been removed. + Disk/CPU/memory visibility is owned by Prometheus + node_exporter + Grafana. The + `disk_usage` **job template** in Actions remains as a manual on-demand SSH check. ### Tables @@ -252,6 +258,7 @@ fully removed (web-ui-rework; see decision log 2026-06-17). ## Decision Log +- 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template. - 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring` → `/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`. - 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`. - 2026-06-13 (Phase 1): Added Prometheus, Loki, Grafana Alloy, Grafana, Alertmanager, and Node Exporter services to `docker-compose.yml` and `docker-compose.dev.yml`. Provisioned Grafana datasources and an initial `Manage Overview` dashboard as code. Configured Alloy to tail Docker logs and ship to Loki. Added Grafana generic OAuth configuration via `monitoring/grafana/grafana.ini` and a dedicated Traefik host rule. Added Alertmanager email routing with env-var interpolation. Added `/grafana` proxy to the Vite dev server for iframe embedding. diff --git a/docs/monitoring-logging-design.md b/docs/monitoring-logging-design.md index 4e390df..c39d9ea 100644 --- a/docs/monitoring-logging-design.md +++ b/docs/monitoring-logging-design.md @@ -60,10 +60,16 @@ The existing POSIX remote collector will be removed, and the Python backup alert ### Metrics -- `backend/src/media_library_viewer_api/clients/resources.py` deploys a POSIX shell collector to `/tmp` on each remote machine. -- The collector samples `/proc/stat`, `/proc/meminfo`, `/proc/net/dev`, and `/sys/block/*/stat` every 10s and writes JSONL to `/tmp/media_library_viewer_metrics.jsonl`. -- `MonitoringPoller` (`monitoring_poller.py`) runs every 5 minutes, reads the remote JSONL, and stores snapshots in SQLite (`monitoring_machine_actions`). -- Retention defaults to 30 days with periodic pruning. +> **Historical note (2026-06-17):** The legacy Manage-side `MonitoringPoller` that +> SSH-scraped `/proc` + `df` into a local SQLite table (`monitoring_machine_actions`) +> has been **decommissioned**. System metrics now live entirely in the external +> observability stack: `node_exporter` on each machine is scraped by **Prometheus** +> and visualised in **Grafana** (see the standalone `docker-compose.observability.yml` +> stack). Manage is a thin dashboard: it surfaces Alertmanager alerts + Prometheus +> target health + Grafana deep-links, and does not collect or store its own metrics. + +- `main.py` has a `log_requests` middleware that emits method, path, client IP, status code, and elapsed time. +- Frontend uses standard `console.log` / browser dev tools; no server-side log aggregation. ### Alerting @@ -210,7 +216,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert - Manage API overview (request rate, latency, errors). - Manage operations (SSH commands, media index builds, mail queue). - Backup runs and alert history. -- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and ` kiosk` mode. +- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and `kiosk` mode. ### Manage React UI @@ -309,6 +315,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert - [x] Wire Grafana OAuth to Authentik. **Phase 1 files**: + - `monitoring/prometheus/prometheus.yml` - `monitoring/prometheus/rules/backup_alerts.yml` - `monitoring/loki/loki.yml` @@ -336,6 +343,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert - [x] Remove POSIX collector fallback. The legacy collector code in `backend/src/media_library_viewer_api/clients/resources.py` has been deleted, the collector control endpoints were removed from `routers/monitoring.py`, and `disk_space` was relocated to `services/monitoring_actions.py` as a lightweight SSH/local helper. Metrics are now sourced exclusively from Prometheus/Node Exporter. **Phase 2 files**: + - `backend/src/media_library_viewer_api/jobs.py` (Node Exporter job templates). - `backend/src/media_library_viewer_api/routers/settings.py` (machine input fields + target regeneration). - `backend/src/media_library_viewer_api/services/settings_store.py` (machine persistence fields). @@ -364,6 +372,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert - [x] Added tests for the Alertmanager endpoints and the backup success gauge. **Phase 3 files**: + - `backend/src/media_library_viewer_api/routers/monitoring.py` (`/alerts` and `/alertmanager-status` endpoints). - `backend/src/media_library_viewer_api/observability.py` (`BACKUP_RUNS_LAST_SUCCESS` gauge + updated `record_backup_run`). - `backend/src/media_library_viewer_api/routers/backups.py` (pass `success=True` to `record_backup_run` on successful reports). @@ -386,6 +395,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert - [x] Wire the new `/observability` route into `App.tsx` and the sidebar navigation. **Phase 4 files**: + - `frontend/src/components/ObservabilityPage.tsx` (page component). - `frontend/src/hooks/useObservability.ts` (React Query hooks). - `frontend/src/api/client.ts` (API client functions). @@ -406,6 +416,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert - [ ] Optional: add OpenTelemetry Collector as a translation layer for traces later. **Phase 5 files**: + - `docker-compose.yml` and `docker-compose.dev.yml` (health checks, resource limits, `depends_on` conditions). - `monitoring/prometheus/prometheus.yml` (additional scrape jobs for observability services). - `monitoring/prometheus/rules/backup_alerts.yml` (renamed scope to include observability health alerts).