docs(monitoring): record legacy poller decommission (slice 3)

Update docs to reflect that Manage no longer scrapes its own system metrics (slices 1-2). AGENTS.md, REQUIREMENTS.md (decision log + observability section), monitoring-logging-design.md, MIGRATION_PLAN.md. Gate: docs only; backend pytest (173) + frontend build/lint/test (22/63) remain green from slices 1-2.
2026-06-17 20:55:24 +00:00
parent a8eb751322
commit bb8b040657
4 changed files with 30 additions and 9 deletions
@@ -105,9 +105,9 @@ repo/
 | `/api/dashboard/counts`         | GET    | `jellyfin.media_counts()`                | Movie/series/episode totals      |
 | `/api/dashboard/libraries`      | GET    | `jellyfin.library_item_counts()`         | Per-library breakdown            |
 | `/api/dashboard/now-playing`    | GET    | `jellyfin.active_sessions()`             | Active sessions + transcode info |
-| `/api/monitoring/status`        | GET    | `resources.resource_collector_status()`  | Collector running?               |
-| `/api/monitoring/metrics`       | GET    | `resources.read_resource_metrics()`      | Last-hour JSONL samples          |
-| `/api/monitoring/disk`          | GET    | `resources.disk_space()`                 | df for media root                |
+| `/api/monitoring/status`        | GET    | `resources.resource_collector_status()`  | Collector running? *(legacy/removed)* |
+| `/api/monitoring/metrics`       | GET    | `resources.read_resource_metrics()`      | Last-hour JSONL samples *(legacy/removed)* |
+| `/api/monitoring/disk`          | GET    | `resources.disk_space()`                 | df for media root *(removed 2026-06-17; metrics now in Prometheus/Grafana)* |
 | `/api/monitoring/start`         | POST   | `resources.start_resource_collector()`   | Start collector                  |
 | `/api/monitoring/stop`          | POST   | `resources.stop_resource_collector()`    | Stop collector                   |
 | `/api/monitoring/restart`       | POST   | `resources.restart_resource_collector()` | Restart collector                |
@@ -37,6 +37,12 @@ fully removed (web-ui-rework; see decision log 2026-06-17).
  not rendered graphs.
 - The legacy in-app D3 monitoring charts and the POSIX remote resource collector are
  superseded by this Grafana-based model (see decision log 2026-06-13 and 2026-06-17).
+- **Manage no longer scrapes its own system metrics** (decision 2026-06-17). The backend
+  `MonitoringPoller` (which SSH-ran `df` on every machine every 5 minutes into a local
+  SQLite `monitoring_machine_actions` table), the `/api/monitoring/disk`, `/poller`, and
+  `/machines/{id}/actions` endpoints, and the frontend `DiskSpaceCard` have been removed.
+  Disk/CPU/memory visibility is owned by Prometheus + node_exporter + Grafana. The
+  `disk_usage` **job template** in Actions remains as a manual on-demand SSH check.

 ### Tables

@@ -252,6 +258,7 @@ fully removed (web-ui-rework; see decision log 2026-06-17).

 ## Decision Log

+- 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template.
 - 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring` → `/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`.
 - 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`.
 - 2026-06-13 (Phase 1): Added Prometheus, Loki, Grafana Alloy, Grafana, Alertmanager, and Node Exporter services to `docker-compose.yml` and `docker-compose.dev.yml`. Provisioned Grafana datasources and an initial `Manage Overview` dashboard as code. Configured Alloy to tail Docker logs and ship to Loki. Added Grafana generic OAuth configuration via `monitoring/grafana/grafana.ini` and a dedicated Traefik host rule. Added Alertmanager email routing with env-var interpolation. Added `/grafana` proxy to the Vite dev server for iframe embedding.
@@ -60,10 +60,16 @@ The existing POSIX remote collector will be removed, and the Python backup alert

 ### Metrics

- `backend/src/media_library_viewer_api/clients/resources.py` deploys a POSIX shell collector to `/tmp` on each remote machine.
- The collector samples `/proc/stat`, `/proc/meminfo`, `/proc/net/dev`, and `/sys/block/*/stat` every 10s and writes JSONL to `/tmp/media_library_viewer_metrics.jsonl`.
- `MonitoringPoller` (`monitoring_poller.py`) runs every 5 minutes, reads the remote JSONL, and stores snapshots in SQLite (`monitoring_machine_actions`).
- Retention defaults to 30 days with periodic pruning.
+> **Historical note (2026-06-17):** The legacy Manage-side `MonitoringPoller` that
+> SSH-scraped `/proc` + `df` into a local SQLite table (`monitoring_machine_actions`)
+> has been **decommissioned**. System metrics now live entirely in the external
+> observability stack: `node_exporter` on each machine is scraped by **Prometheus**
+> and visualised in **Grafana** (see the standalone `docker-compose.observability.yml`
+> stack). Manage is a thin dashboard: it surfaces Alertmanager alerts + Prometheus
+> target health + Grafana deep-links, and does not collect or store its own metrics.
+
+- `main.py` has a `log_requests` middleware that emits method, path, client IP, status code, and elapsed time.
+- Frontend uses standard `console.log` / browser dev tools; no server-side log aggregation.

 ### Alerting

@@ -210,7 +216,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
  - Manage API overview (request rate, latency, errors).
  - Manage operations (SSH commands, media index builds, mail queue).
  - Backup runs and alert history.
- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and ` kiosk` mode.
+- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and `kiosk` mode.

 ### Manage React UI

@@ -309,6 +315,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
 - [x] Wire Grafana OAuth to Authentik.

 **Phase 1 files**:
+
 - `monitoring/prometheus/prometheus.yml`
 - `monitoring/prometheus/rules/backup_alerts.yml`
 - `monitoring/loki/loki.yml`
@@ -336,6 +343,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
 - [x] Remove POSIX collector fallback. The legacy collector code in `backend/src/media_library_viewer_api/clients/resources.py` has been deleted, the collector control endpoints were removed from `routers/monitoring.py`, and `disk_space` was relocated to `services/monitoring_actions.py` as a lightweight SSH/local helper. Metrics are now sourced exclusively from Prometheus/Node Exporter.

 **Phase 2 files**:
+
 - `backend/src/media_library_viewer_api/jobs.py` (Node Exporter job templates).
 - `backend/src/media_library_viewer_api/routers/settings.py` (machine input fields + target regeneration).
 - `backend/src/media_library_viewer_api/services/settings_store.py` (machine persistence fields).
@@ -364,6 +372,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
 - [x] Added tests for the Alertmanager endpoints and the backup success gauge.

 **Phase 3 files**:
+
 - `backend/src/media_library_viewer_api/routers/monitoring.py` (`/alerts` and `/alertmanager-status` endpoints).
 - `backend/src/media_library_viewer_api/observability.py` (`BACKUP_RUNS_LAST_SUCCESS` gauge + updated `record_backup_run`).
 - `backend/src/media_library_viewer_api/routers/backups.py` (pass `success=True` to `record_backup_run` on successful reports).
@@ -386,6 +395,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
 - [x] Wire the new `/observability` route into `App.tsx` and the sidebar navigation.

 **Phase 4 files**:
+
 - `frontend/src/components/ObservabilityPage.tsx` (page component).
 - `frontend/src/hooks/useObservability.ts` (React Query hooks).
 - `frontend/src/api/client.ts` (API client functions).
@@ -406,6 +416,7 @@ The existing POSIX remote collector will be removed, and the Python backup alert
 - [ ] Optional: add OpenTelemetry Collector as a translation layer for traces later.

 **Phase 5 files**:
+
 - `docker-compose.yml` and `docker-compose.dev.yml` (health checks, resource limits, `depends_on` conditions).
 - `monitoring/prometheus/prometheus.yml` (additional scrape jobs for observability services).
 - `monitoring/prometheus/rules/backup_alerts.yml` (renamed scope to include observability health alerts).