alex/manage
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(observability): add Prometheus/Grafana/Loki/Alertmanager/Alloy stack and remove legacy Monitoring UI

Browse Source
This commit is contained in:
Developer
2026-06-16 11:08:32 +00:00
parent 615e02e970
commit e2ad731b5f
80 changed files with 5020 additions and 3665 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/REQUIREMENTS.md
+6
View File
@@ -197,6 +197,12 @@ Phase 1: Jellyfin media index, SSH-based remote filesystem inspection, server mo
## Decision Log
- 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`.
- 2026-06-13 (Phase 1): Added Prometheus, Loki, Grafana Alloy, Grafana, Alertmanager, and Node Exporter services to `docker-compose.yml` and `docker-compose.dev.yml`. Provisioned Grafana datasources and an initial `Manage Overview` dashboard as code. Configured Alloy to tail Docker logs and ship to Loki. Added Grafana generic OAuth configuration via `monitoring/grafana/grafana.ini` and a dedicated Traefik host rule. Added Alertmanager email routing with env-var interpolation. Added `/grafana` proxy to the Vite dev server for iframe embedding.
- 2026-06-13 (Phase 2): Extended machine settings with `node_exporter_enabled`, `node_exporter_port`, and `node_exporter_scrape_host`. Added Node Exporter install/restart/status job templates to `jobs.py`. Implemented `media_library_viewer_api.services.targets` to generate Prometheus file-SD target files and wired target regeneration into machine create/update/delete. Added `/api/monitoring/prometheus-targets` for live target previews. Configured Prometheus with a `node-exporter-remote` job reading file SD from the backend cache volume. Added a minimal `Node Exporter Overview` Grafana dashboard. Added unit and integration tests for target generation and the new endpoint.
- 2026-06-13 (Phase 3): Added `ALERTMANAGER_URL` setting and `GET /api/monitoring/alerts` plus `GET /api/monitoring/alertmanager-status` endpoints that proxy/summarize Alertmanager for the UI. Added `manage_backup_runs_last_success_timestamp` Prometheus gauge and wired `record_backup_run` to update it on success so the existing `BackupJobStuck` Alertmanager rule works. Updated `docker-compose.yml`, `docker-compose.dev.yml`, and `.env.example` with `ALERTMANAGER_URL`. Added tests for the new endpoints and metric helpers.
- 2026-06-13 (Phase 4): Added `/observability` React page (`frontend/src/components/ObservabilityPage.tsx`) with health cards for Alertmanager, active alerts, Prometheus targets, and machines; a recent-alerts list; machine selector; and Grafana iframe panels for Node Exporter metrics and Loki logs. Added `useObservability` hook and API client wrappers for `/api/monitoring/alerts`, `/api/monitoring/alertmanager-status`, and `/api/monitoring/prometheus-targets`. Added TypeScript types for Alertmanager summary/status and Prometheus targets. Wired the new route into `App.tsx` and the sidebar. Added shadcn/ui `card`, `badge`, `alert`, `skeleton`, and `select` components. Frontend build (`npm run build`) passes; lint has only pre-existing warnings.
- 2026-06-13 (Phase 5): Hardened observability containers with health checks and resource limits in both compose files; added service-healthy `depends_on` conditions. Added Prometheus scrape jobs for Loki, Alertmanager, and Grafana, plus new `observability_health` alert rules. Added `ALERTMANAGER_WEBHOOK_URL` backend setting, `POST /api/monitoring/alertmanager-webhook` receiver, and Alertmanager `webhook` receiver config. Added `docs/observability-runbooks.md` with operational playbooks. Updated `.env.example` to include all observability variables.
- 2026-05-03: Reaffirmed that the Monitoring tab charts should be rendered directly with D3 and expose brush-based time-range selection plus moving averages.
- 2026-05-03: Added hover tooltips, summary chips, a moving vertical cursor, snapped point markers, and a selected-range label to the D3 Monitoring charts for faster visual inspection.
- 2026-05-03: Combined network download/upload into one traffic chart and disk read/write into one I/O chart for clearer Monitoring layout.