Developer
439 lines
26 KiB
Markdown
439 lines
26 KiB
Markdown
# Monitoring and Logging Design — Manage
|
||
|
||
## Executive Summary
|
||
|
||
Manage currently uses ad-hoc observability: plain-text Python logs, a custom POSIX shell metrics collector on remote machines, and a background poller that stores snapshots in SQLite. This works for a single-instance homelab but becomes painful as the fleet grows and as users need faster incident response.
|
||
|
||
This document proposes a dedicated, self-hosted observability subsystem built on the standard Prometheus/Grafana stack:
|
||
|
||
- **Metrics**: Prometheus pulling from Node Exporter on machines and from application exporters in containers.
|
||
- **Logs**: Structured JSON logs shipped to **Grafana Loki** by **Promtail/Grafana Alloy**.
|
||
- **Dashboards**: Grafana for deep-dive dashboards, embedded in the Manage React UI via iframes.
|
||
- **Alerting**: Prometheus Alertmanager for routing and notifications (email first, webhooks later).
|
||
- **Auth**: Grafana authenticates through the existing OIDC/Authentik provider.
|
||
|
||
The existing POSIX remote collector will be removed, and the Python backup alert engine will be migrated to Alertmanager rules.
|
||
|
||
---
|
||
|
||
## Goals
|
||
|
||
1. **Fast query and alerting**: move from SQLite scan-based history to a real time-series database and indexed log store.
|
||
2. **Unified view**: monitor both local containers/apps and remote Linux machines from one place.
|
||
3. **Standard tooling**: use de-facto open-source tools so dashboards, exporters, and runbooks are reusable.
|
||
4. **Room to grow**: design supports adding traces, more notification channels, and longer retention later without re-architecture.
|
||
|
||
## Non-Goals
|
||
|
||
1. **Traces**: deferred to a later phase; the data flow and collector choice (Promtail/Alloy) will be trace-ready.
|
||
2. **Multi-tenant RBAC**: Manage is single-instance/homelab; Grafana teams are sufficient for now.
|
||
3. **SLA/SLO framework**: out of scope; we focus on metrics, logs, and alerts, not SLO budgeting.
|
||
4. **Cloud-hosted observability vendors**: all components run self-hosted in Docker Compose.
|
||
|
||
---
|
||
|
||
## Decisions
|
||
|
||
| Area | Decision | Rationale |
|
||
|------|----------|-----------|
|
||
| Coupling | Dedicated observability subsystem consumed by Manage | Keeps Manage fast and lets the observability stack evolve independently. |
|
||
| Metrics backend | Prometheus | Pull model, huge ecosystem, standard exporters, easy Grafana integration. |
|
||
| Machine metrics | Node Exporter | Rich OS metrics, reusable dashboards, no custom shell to maintain. |
|
||
| Log backend | Grafana Loki | Prometheus-style labels, low resource use, tight Grafana integration. |
|
||
| Log collection | Promtail / Grafana Alloy | Tails Docker logs and journald; no per-app network calls. |
|
||
| App logs | Structured JSON to stdout | Standard 12-factor pattern; collector handles routing. |
|
||
| Dashboards | Grafana + iframe embeds | Fast to implement, rich dashboards, Manage UI stays focused on summary. |
|
||
| Alerting | Prometheus Alertmanager | Mature routing, silencing, inhibition; single source of truth for infra alerts. |
|
||
| Auth | Grafana OAuth via Authentik | Reuses existing identity provider; consistent UX. |
|
||
| Retention | 30 days metrics, 30 days logs | Matches current retention policy; disk usage stays predictable. |
|
||
| Migration | Remove POSIX collector, migrate backup alerts | Eliminates duplicate alerting paths and custom remote code. |
|
||
|
||
---
|
||
|
||
## Current State
|
||
|
||
### Logging
|
||
|
||
- `backend/src/media_library_viewer_api/logging_utils.py` configures stdlib `logging` with a plain-text format.
|
||
- `main.py` has a `log_requests` middleware that emits method, path, client IP, status code, and elapsed time.
|
||
- Frontend uses standard `console.log` / browser dev tools; no server-side log aggregation.
|
||
|
||
### Metrics
|
||
|
||
- `backend/src/media_library_viewer_api/clients/resources.py` deploys a POSIX shell collector to `/tmp` on each remote machine.
|
||
- The collector samples `/proc/stat`, `/proc/meminfo`, `/proc/net/dev`, and `/sys/block/*/stat` every 10s and writes JSONL to `/tmp/media_library_viewer_metrics.jsonl`.
|
||
- `MonitoringPoller` (`monitoring_poller.py`) runs every 5 minutes, reads the remote JSONL, and stores snapshots in SQLite (`monitoring_machine_actions`).
|
||
- Retention defaults to 30 days with periodic pruning.
|
||
|
||
### Alerting
|
||
|
||
- `backup_alert_engine.py` / `backup_poller.py` generate backup-related alerts (failure, anomaly, missed schedule) and store them in SQLite.
|
||
- No general infrastructure alerting (disk full, machine down, high CPU, etc.).
|
||
|
||
---
|
||
|
||
## Target Architecture
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||
│ Docker Compose Network │
|
||
│ │
|
||
│ ┌─────────────┐ scrape ┌──────────────┐ │
|
||
│ │ Prometheus │◄────────────────│ Node Exporter│◄── host / remote hosts │
|
||
│ │ (TSDB) │ └──────────────┘ │
|
||
│ └──────┬──────┘ │
|
||
│ │ query │
|
||
│ ▼ │
|
||
│ ┌─────────────┐ alert ┌─────────────┐ email ┌──────────┐ │
|
||
│ │ Grafana │──────────────►│ Alertmanager│──────────────►│ SMTP │ │
|
||
│ │ (OAuth) │ └─────────────┘ └──────────┘ │
|
||
│ └──────┬──────┘ │
|
||
│ │ embed (iframe) │
|
||
│ ▼ │
|
||
│ ┌─────────────────┐ │
|
||
│ │ Manage React │ │
|
||
│ │ (summary) │ │
|
||
│ └─────────────────┘ │
|
||
│ │
|
||
│ Logs: │
|
||
│ Manage / containers ──stdout──► Promtail/Alloy ──push──► Loki ◄──────┐ │
|
||
│ host / remote journald ───────► Promtail/Alloy ──push──► Loki │ │
|
||
│ │ │
|
||
│ Grafana queries Loki for logs ◄──────────────────────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Component Responsibilities
|
||
|
||
| Component | Responsibility |
|
||
|-----------|----------------|
|
||
| **Prometheus** | Scrape and store metrics; evaluate alert rules. |
|
||
| **Node Exporter** | Expose host-level metrics (CPU, memory, disk, network, filesystem). |
|
||
| **Loki** | Store and index log streams by labels. |
|
||
| **Promtail / Alloy** | Discover log sources, parse labels, and push to Loki. |
|
||
| **Grafana** | Visualize metrics and logs; serve as the alert UI. |
|
||
| **Alertmanager** | Deduplicate, group, route, and deliver alerts. |
|
||
| **Manage backend** | Emit structured logs; expose `/metrics` for Prometheus; forward health/status to summary endpoints. |
|
||
| **Manage frontend** | Embed Grafana panels; show high-level status cards. |
|
||
|
||
---
|
||
|
||
## Instrumentation Changes
|
||
|
||
### Backend Logging
|
||
|
||
1. Switch to structured JSON logging via `python-json-logger` or `structlog`.
|
||
2. Include fields:
|
||
- `timestamp`, `level`, `logger`, `message`
|
||
- `request_id` (correlation ID propagated from frontend or generated)
|
||
- `method`, `path`, `status_code`, `elapsed_ms`
|
||
- `user_id`, `machine_id` where relevant
|
||
- `error` / `error_type` / `traceback` for exceptions
|
||
3. Keep emitting to stdout; Promtail/Alloy will parse JSON.
|
||
|
||
**Status**: implemented in `backend/src/media_library_viewer_api/logging_utils.py` with `LOG_FORMAT=json|text`, secret-safe `sanitize_log_extra`, and request logging in `main.py`.
|
||
|
||
### Request Middleware
|
||
|
||
- Extend `log_requests` to attach `request_id` to `request.state`.
|
||
- Include `request_id` in response headers (`X-Request-Id`) so the frontend can correlate.
|
||
- Log all outbound SSH commands with `machine_id`, `action`, `duration_ms`, and `request_id`.
|
||
|
||
**Status**: implemented in `main.py` and `observability.py`; `record_ssh_command` is called from `monitoring_actions.py` for every machine operation.
|
||
|
||
### Application Metrics Endpoint
|
||
|
||
- Add a `/metrics` endpoint using `prometheus-client`.
|
||
- Initial counters/gauges:
|
||
- `manage_api_requests_total` (method, path, status)
|
||
- `manage_api_request_duration_seconds` histogram
|
||
- `manage_ssh_commands_total` (machine_id, action, status)
|
||
- `manage_ssh_command_duration_seconds` histogram
|
||
- `manage_media_index_build_duration_seconds`
|
||
- `manage_backup_runs_total` (job_name, status)
|
||
- `manage_mail_queue_size`, `manage_mail_queue_failures_total`
|
||
|
||
**Status**: implemented in `backend/src/media_library_viewer_api/observability.py` and wired into `main.py`, `monitoring_actions.py`, `backups.py`, `media.py`, and `mail_queue.py`.
|
||
|
||
### Frontend Observability
|
||
|
||
- Keep first phase minimal: capture JS errors and send them to the backend as structured log events.
|
||
- Optional later: expose RUM-style metrics (page loads, API call latencies) via Prometheus client library or manual instrumentation.
|
||
|
||
---
|
||
|
||
## Node Exporter Deployment
|
||
|
||
### Local / Docker Host
|
||
|
||
- Add a `node-exporter` service to `docker-compose.yml` with host PID/network mounts.
|
||
- Prometheus scrapes it as `job="node"`.
|
||
|
||
### Remote Machines
|
||
|
||
- Add a managed task/template in `jobs.py` to install/upgrade Node Exporter via the package manager or a static binary.
|
||
- Manage exposes a settings flag per machine: `node_exporter_enabled`.
|
||
- For machines behind NAT, use one of:
|
||
1. Reverse SSH tunnel from machine to Manage host.
|
||
2. VPN/Wireguard already in place.
|
||
3. Prometheus federation or pushgateway for unreachable targets (later phase).
|
||
- If Node Exporter cannot be installed, temporarily keep the POSIX collector as a fallback until migration is complete.
|
||
|
||
---
|
||
|
||
## Log Shipping
|
||
|
||
### Docker Compose Services
|
||
|
||
- Add `logging` driver config or Promtail sidecar to each service.
|
||
- Preferred: run **Grafana Alloy** as a single daemon container with `docker_sd_config` to discover all Compose services automatically.
|
||
|
||
### Host Logs
|
||
|
||
- Alloy mounts `/var/log` and `/var/lib/docker/containers` (read-only).
|
||
- Alloy also tails journald where available.
|
||
|
||
### Remote Machines
|
||
|
||
- Option A: install Alloy on remote hosts and have it push logs to Loki.
|
||
- Option B: keep logs on remote hosts and use Node Exporter logs only; defer centralized remote logs.
|
||
- Recommendation: **Option A** for important machines, **Option B** for constrained ones.
|
||
|
||
---
|
||
|
||
## Dashboards
|
||
|
||
### Grafana
|
||
|
||
- Provision dashboards from YAML/JSON in version control:
|
||
- Node Exporter Full dashboard (import from Grafana.com).
|
||
- Manage API overview (request rate, latency, errors).
|
||
- Manage operations (SSH commands, media index builds, mail queue).
|
||
- Backup runs and alert history.
|
||
- Manage iframe embeds point to specific dashboard panels using Grafana's `panelId` and ` kiosk` mode.
|
||
|
||
### Manage React UI
|
||
|
||
- Add an "Observability" page with:
|
||
- System health cards (Prometheus up, Loki up, Alertmanager up).
|
||
- Recent alerts summary from Alertmanager API.
|
||
- Iframe panels for key metrics (CPU/memory of selected machine, recent logs).
|
||
- Drill-down links open the full Grafana dashboard.
|
||
|
||
---
|
||
|
||
## Alerting
|
||
|
||
### Alertmanager Configuration
|
||
|
||
- Reuse existing SMTP settings for email notifications.
|
||
- Initial routing:
|
||
- `severity=critical` → email immediately.
|
||
- `severity=warning` → email with 5-minute group wait.
|
||
- `job=backup` → grouped by job name.
|
||
|
||
### Initial Alert Rules
|
||
|
||
- Infrastructure:
|
||
- Node down for > 5 minutes.
|
||
- Disk usage > 85% (warning), > 95% (critical).
|
||
- Memory usage > 90% for > 10 minutes.
|
||
- CPU iowait > 30% for > 10 minutes.
|
||
- Application:
|
||
- Manage API 5xx rate > 1% over 5 minutes.
|
||
- SSH command failure rate > 10% over 5 minutes.
|
||
- Mail queue growing or failures increasing.
|
||
- Backup:
|
||
- Backup job failed (`manage_backup_runs_total{status="failure"}`).
|
||
- Backup job missing for > 1.5× schedule interval.
|
||
- Backup run duration or size anomaly compared to rolling median.
|
||
|
||
### Backup Alert Migration
|
||
|
||
- Re-implement rules as Prometheus recording/alerting rules where possible.
|
||
- Keep historical comparison logic (median duration/size) as a small scheduled task that writes anomaly metrics to a Pushgateway or custom exporter, then Alertmanager consumes them.
|
||
- Preserve acknowledge/resolve workflow by storing Alertmanager webhook events in SQLite if needed, or by using Grafana alert annotations.
|
||
|
||
---
|
||
|
||
## Authentication
|
||
|
||
- Grafana configured with generic OAuth pointing at Authentik (same issuer as Manage).
|
||
- Grafana role mapping: default `Viewer`; admin group mapped to `Admin`.
|
||
- Traefik routes `grafana.${BACKEND_APP_HOST}` or a sub-path.
|
||
- Iframe embedding requires Grafana `allow_embedding = true` and matching cookie domain/samesite settings.
|
||
|
||
---
|
||
|
||
## Retention and Storage
|
||
|
||
| Store | Retention | Notes |
|
||
|-------|-----------|-------|
|
||
| Prometheus | 30 days | Default TSDB block compaction. |
|
||
| Loki | 30 days | Single-store boltdb-shipper or filesystem target. |
|
||
| Grafana | persistent SQLite/Postgres later | Dashboards and users are config, not runtime data. |
|
||
|
||
- Volumes: `prometheus-data`, `loki-data`, `grafana-data`.
|
||
- Backups: snapshot these volumes alongside existing `backend_cache`.
|
||
|
||
---
|
||
|
||
## Security
|
||
|
||
1. Network: all observability services on an internal Docker network; exposed only through Traefik where needed.
|
||
2. Node Exporter: bind to localhost on remote hosts and use a reverse tunnel, or firewall to Manage IP only.
|
||
3. Secrets: SMTP password, OIDC client secret, and any remote scrape credentials in environment variables or Docker secrets; never commit them.
|
||
4. Logs: sanitize tokens, passwords, and private keys before JSON serialization.
|
||
5. Alertmanager: disable unauthenticated UI if exposed publicly; rely on OIDC/Traefik.
|
||
|
||
---
|
||
|
||
## Implementation Plan
|
||
|
||
### Phase 0 — Foundation and Cleanup
|
||
|
||
- [x] Add `prometheus-client` and `python-json-logger` to `backend/pyproject.toml`.
|
||
- [x] Refactor `logging_utils.py` to emit JSON when `LOG_FORMAT=json`.
|
||
- [x] Add `request_id` propagation in `log_requests` middleware.
|
||
- [x] Add `/metrics` endpoint with initial counters/gauges.
|
||
- [x] Remove the POSIX remote collector code in `resources.py`; keep `disk_space` as a lightweight SSH/local helper in `monitoring_actions.py`.
|
||
- [x] Add `X-Request-Id` response header.
|
||
|
||
### Phase 1 — Local Observability Stack
|
||
|
||
- [x] Add services to `docker-compose.yml`: Prometheus, Loki, Grafana, Alertmanager, Grafana Alloy.
|
||
- [x] Add `node-exporter` service for the Docker host.
|
||
- [x] Configure Alloy to scrape all Docker container logs and ship to Loki.
|
||
- [x] Configure Prometheus to scrape `node-exporter` and Manage `/metrics`.
|
||
- [x] Provision Grafana datasources and a basic Manage API dashboard.
|
||
- [x] Wire Grafana OAuth to Authentik.
|
||
|
||
**Phase 1 files**:
|
||
- `monitoring/prometheus/prometheus.yml`
|
||
- `monitoring/prometheus/rules/backup_alerts.yml`
|
||
- `monitoring/loki/loki.yml`
|
||
- `monitoring/alloy/config.alloy`
|
||
- `monitoring/alertmanager/alertmanager.yml`
|
||
- `monitoring/grafana/grafana.ini`
|
||
- `monitoring/grafana/provisioning/datasources/datasources.yml`
|
||
- `monitoring/grafana/provisioning/dashboards-json/dashboards.yml`
|
||
- `monitoring/grafana/provisioning/dashboards-json/dashboards/manage-overview.json`
|
||
- `docker-compose.yml` and `docker-compose.dev.yml` updated with observability services.
|
||
- `frontend/vite.config.ts` updated with `/grafana` dev proxy.
|
||
- `.env.example` updated with Grafana OAuth and alerting variables.
|
||
|
||
### Phase 2 — Remote Machine Metrics
|
||
|
||
- [x] Add Node Exporter install/restart/status job templates in `jobs.py` (`install_node_exporter`, `restart_node_exporter`, `node_exporter_status`).
|
||
- [x] Add `node_exporter_enabled`, `node_exporter_port`, and `node_exporter_scrape_host` fields to `MonitoringMachineInput` and `SettingsStore`.
|
||
- [x] Implement `media_library_viewer_api.services.targets` to build Prometheus file-SD target lists for enabled SSH machines and write them to `PROMETHEUS_FILE_SD_DIR/node_exporter_targets.json`.
|
||
- [x] Regenerate file-SD targets on machine create/update/delete in `routers/settings.py`.
|
||
- [x] Add `/api/monitoring/prometheus-targets` endpoint returning live targets from the store.
|
||
- [x] Configure Prometheus `node-exporter-remote` job with `file_sd_configs` reading `/etc/prometheus/file-sd/node_exporter_targets.json`.
|
||
- [x] Mount the backend cache `prometheus-file-sd` directory into the Prometheus container as a read-only file-SD source.
|
||
- [x] Add `PROMETHEUS_FILE_SD_DIR` setting and `.env.example` entry.
|
||
- [x] Provision a minimal `Node Exporter Overview` Grafana dashboard (`monitoring/grafana/provisioning/dashboards-json/dashboards/node-exporter-overview.json`) covering CPU, memory, root disk, and network traffic.
|
||
- [x] Remove POSIX collector fallback. The legacy collector code in `backend/src/media_library_viewer_api/clients/resources.py` has been deleted, the collector control endpoints were removed from `routers/monitoring.py`, and `disk_space` was relocated to `services/monitoring_actions.py` as a lightweight SSH/local helper. Metrics are now sourced exclusively from Prometheus/Node Exporter.
|
||
|
||
**Phase 2 files**:
|
||
- `backend/src/media_library_viewer_api/jobs.py` (Node Exporter job templates).
|
||
- `backend/src/media_library_viewer_api/routers/settings.py` (machine input fields + target regeneration).
|
||
- `backend/src/media_library_viewer_api/services/settings_store.py` (machine persistence fields).
|
||
- `backend/src/media_library_viewer_api/services/targets.py` (file-SD target builder/writer).
|
||
- `backend/src/media_library_viewer_api/routers/monitoring.py` (`/prometheus-targets` endpoint).
|
||
- `backend/src/media_library_viewer_api/config.py` (`prometheus_file_sd_dir` setting).
|
||
- `backend/tests/test_targets.py` and `backend/tests/test_api.py` (target + endpoint tests).
|
||
- `monitoring/prometheus/prometheus.yml` (`node-exporter-remote` file SD job).
|
||
- `monitoring/grafana/provisioning/dashboards-json/dashboards/node-exporter-overview.json`.
|
||
- `docker-compose.yml` and `docker-compose.dev.yml` (file-SD volume mount + backend env var).
|
||
- `.env.example` (`PROMETHEUS_FILE_SD_DIR`).
|
||
|
||
### Phase 3 — Alerting
|
||
|
||
- [x] Define initial Prometheus alert rules for backup failures (infrastructure rules deferred to Phase 2/3).
|
||
- [x] Configure Alertmanager with email routing using existing SMTP settings; `monitoring/alertmanager/alertmanager.yml` uses env vars for SMTP and routing.
|
||
- [x] Migrate backup alert rules to Alertmanager:
|
||
- `BackupJobFailed` triggers on `increase(manage_backup_runs_total{status="failed"}[1h]) > 0`.
|
||
- `BackupJobStuck` triggers on `time() - manage_backup_runs_last_success_timestamp > 86400`.
|
||
- Added `manage_backup_runs_last_success_timestamp` gauge in `observability.py` and updated `routers/backups.py` to set it on successful runs.
|
||
- SQLite backup alerts (`backup_alert_engine.py` and `backup_poller.py`) are preserved for now alongside Alertmanager rules; the UI can consume either source during transition.
|
||
- [x] Add Alertmanager status summary endpoints in Manage backend:
|
||
- `GET /api/monitoring/alerts` proxies `/api/v1/alerts` and returns a UI-friendly summary (total, by_severity, alerts list).
|
||
- `GET /api/monitoring/alertmanager-status` proxies `/api/v2/status` and returns `up`, `version`, `uptime`, `peers`.
|
||
- [x] Added `alertmanager_url` setting to `config.py` (default `http://alertmanager:9093`) and `ALERTMANAGER_URL` env var in both compose files and `.env.example`.
|
||
- [x] Added tests for the Alertmanager endpoints and the backup success gauge.
|
||
|
||
**Phase 3 files**:
|
||
- `backend/src/media_library_viewer_api/routers/monitoring.py` (`/alerts` and `/alertmanager-status` endpoints).
|
||
- `backend/src/media_library_viewer_api/observability.py` (`BACKUP_RUNS_LAST_SUCCESS` gauge + updated `record_backup_run`).
|
||
- `backend/src/media_library_viewer_api/routers/backups.py` (pass `success=True` to `record_backup_run` on successful reports).
|
||
- `backend/src/media_library_viewer_api/config.py` (`alertmanager_url` setting).
|
||
- `monitoring/alertmanager/alertmanager.yml` (SMTP + routing config).
|
||
- `monitoring/prometheus/rules/backup_alerts.yml` (backup alert rules).
|
||
- `docker-compose.yml` / `docker-compose.dev.yml` (`ALERTMANAGER_URL` env var).
|
||
- `.env.example` (`ALERTMANAGER_URL`).
|
||
- `backend/tests/test_api.py` (`TestAlertmanager` tests).
|
||
- `backend/tests/test_observability.py` (backup metric tests).
|
||
|
||
### Phase 4 — Manage UI Integration
|
||
|
||
- [x] Add "Observability" page in React with summary cards (Alertmanager health, active alerts, Prometheus targets, machines) and Grafana iframe panels.
|
||
- [x] Add recent alerts list from Alertmanager API via `GET /api/monitoring/alerts`.
|
||
- [x] Add drill-down links to full Grafana dashboards for Node Exporter metrics and Loki logs.
|
||
- [x] Handle iframe sandbox attributes (`allow-scripts allow-same-origin allow-popups allow-forms`); CSP is delegated to the reverse proxy / Grafana `allow_embedding` configuration.
|
||
- [x] Add `useObservability` hook and API client wrappers for alerts, Alertmanager status, and Prometheus targets.
|
||
- [x] Add TypeScript types for Alertmanager summary/status and Prometheus targets.
|
||
- [x] Wire the new `/observability` route into `App.tsx` and the sidebar navigation.
|
||
|
||
**Phase 4 files**:
|
||
- `frontend/src/components/ObservabilityPage.tsx` (page component).
|
||
- `frontend/src/hooks/useObservability.ts` (React Query hooks).
|
||
- `frontend/src/api/client.ts` (API client functions).
|
||
- `frontend/src/types/index.ts` (new interfaces).
|
||
- `frontend/src/App.tsx` (route + nav item).
|
||
- `frontend/src/components/ui/{card,badge,alert,skeleton,select}.tsx` (shadcn/ui components).
|
||
- `frontend/vite.config.ts` already has `/grafana` dev proxy for iframe source.
|
||
|
||
### Phase 5 — Hardening and Future-Proofing
|
||
|
||
- [x] Add health checks and `deploy.resources` limits for Prometheus, Loki, Alloy, Grafana, Alertmanager, and Node Exporter in both compose files.
|
||
- [x] Use `depends_on` with `condition: service_healthy` for Alloy → Loki and Grafana → Prometheus/Loki.
|
||
- [x] Add Prometheus scrape jobs for Loki, Alertmanager, and Grafana so their `up` metrics are available for health alerts.
|
||
- [x] Add observability health alerting rules (`PrometheusTargetMissing`, `AlertmanagerDown`, `GrafanaDown`).
|
||
- [x] Add `ALERTMANAGER_WEBHOOK_URL` backend setting, `POST /api/monitoring/alertmanager-webhook` receiver, and Alertmanager `webhook` receiver config.
|
||
- [x] Document runbooks for common alerts.
|
||
- [x] Add volume backups for Prometheus/Loki/Grafana data. Backup/restore procedures for `prometheus_data`, `loki_data`, `grafana_data`, and `alertmanager_data` are documented in `docs/observability-runbooks.md`.
|
||
- [ ] Optional: add OpenTelemetry Collector as a translation layer for traces later.
|
||
|
||
**Phase 5 files**:
|
||
- `docker-compose.yml` and `docker-compose.dev.yml` (health checks, resource limits, `depends_on` conditions).
|
||
- `monitoring/prometheus/prometheus.yml` (additional scrape jobs for observability services).
|
||
- `monitoring/prometheus/rules/backup_alerts.yml` (renamed scope to include observability health alerts).
|
||
- `backend/src/media_library_viewer_api/config.py` (`alertmanager_webhook_url` setting).
|
||
- `backend/src/media_library_viewer_api/routers/monitoring.py` (`POST /api/monitoring/alertmanager-webhook`).
|
||
- `monitoring/alertmanager/alertmanager.yml` (`webhook` receiver).
|
||
- `backend/tests/test_api.py` (`TestAlertmanagerWebhook`).
|
||
- `docs/observability-runbooks.md` (new runbook documentation).
|
||
|
||
---
|
||
|
||
## Risks and Mitigations
|
||
|
||
| Risk | Mitigation |
|
||
|------|------------|
|
||
| Node Exporter hard to install on NAS/minimal hosts | Keep POSIX collector as opt-in fallback; document manual install steps. |
|
||
| Grafana iframe embedding blocked by CSP or cookies | Test early in Phase 4; use matching domains and `allow_embedding`. |
|
||
| Prometheus storage grows faster than expected | Start with 30-day retention; add compaction alerts. |
|
||
| Backup alert semantics lost in migration | Write tests comparing old Python alerts vs new Alertmanager rules. |
|
||
| OIDC configuration drift between Manage and Grafana | Use same env vars/Authentik application for both. |
|
||
| Remote scrape requires network path | Use reverse SSH tunnels or defer remote scraping until VPN is ready. |
|
||
|
||
---
|
||
|
||
## Open Questions
|
||
|
||
1. ~~What sub-domain or sub-path should Grafana use? (`grafana.${BACKEND_APP_HOST}` vs `${BACKEND_APP_HOST}/grafana`)~~ Decided: dedicated `GRAFANA_APP_HOST` subdomain in production; dev uses port 3000 and a `/grafana` proxy in Vite.
|
||
2. Should remote machines run Node Exporter as a systemd service or a container?
|
||
3. Do we need remote log aggregation immediately, or can it wait until after metrics alerting is stable?
|
||
4. Should the existing backup alert acknowledgement/resolve UI be rebuilt on top of Alertmanager, or replaced by Grafana alert annotations?
|