From d4f95b64d4fb1824be29806267877d1c7405a5db Mon Sep 17 00:00:00 2001 From: Developer Date: Tue, 23 Jun 2026 21:20:07 +0000 Subject: [PATCH] chore(observability): externalize stack from root compose files Manage now connects to existing Grafana/Prometheus/Alertmanager instances and never deploys its own stack. - docker-compose.yml / docker-compose.dev.yml: removed prometheus, loki, alloy, grafana, alertmanager, node-exporter services, the monitoring network, and observability named volumes; they now ship only backend + frontend. Dev frontend now joins the web network so the Vite dev proxy can reach the backend. - backend: alertmanager_url default is now empty; /api/monitoring/alerts and /alertmanager-status return graceful "not configured" responses when ALERTMANAGER_URL is unset. Added not-configured tests. - docker-compose.observability.yml: kept as the optional standalone example; header clarifies Manage does not deploy it. - Removed orphaned combined monitoring/prometheus/prometheus.yml (standalone stack uses prometheus.standalone.yml). - Docs (README, REQUIREMENTS decision log, monitoring-logging-design, observability-runbooks, context.md, MIGRATION_PLAN, frontend/README, CHANGELOG) updated to the connect-to-existing model. VITE_GRAFANA_URL / VITE_PROMETHEUS_URL remain as optional frontend deep-link overrides. .env.example still needs a manual update (safety policy blocks assistant edits): set ALERTMANAGER_URL empty/optional and move standalone-only vars out of the root file. --- CHANGELOG.md | 19 ++ README.md | 8 +- .../src/media_library_viewer_api/config.py | 2 +- .../routers/monitoring.py | 13 +- backend/tests/test_api.py | 21 ++ backend/tests/test_widgets.py | 1 - context.md | 30 ++- docker-compose.dev.yml | 219 ++--------------- docker-compose.observability.yml | 11 +- docker-compose.yml | 226 +----------------- docs/MIGRATION_PLAN.md | 2 +- docs/REQUIREMENTS.md | 3 +- docs/monitoring-logging-design.md | 11 + docs/observability-runbooks.md | 6 +- frontend/README.md | 19 +- monitoring/prometheus/prometheus.yml | 48 ---- 16 files changed, 146 insertions(+), 493 deletions(-) delete mode 100644 monitoring/prometheus/prometheus.yml diff --git a/CHANGELOG.md b/CHANGELOG.md index 5711100..5de025a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,25 @@ All notable changes to Manage. Breaking changes are marked with **BREAKING**. ## [Unreleased] +### Changed — Observability is now external only + +- **Removed** all observability services from `docker-compose.yml` and + `docker-compose.dev.yml`. They now deploy **only** the backend and frontend. + The `monitoring` network and the `prometheus`/`loki`/`alloy`/`grafana`/ + `alertmanager`/`node-exporter` services and their named volumes were deleted, + and the `GRAFANA_APP_HOST` Traefik rule was removed. +- Manage now connects to **existing** Grafana/Prometheus/Alertmanager instances + and never ships its own stack. The previous in-compose stack is preserved as + an optional, deploy-it-yourself example in `docker-compose.observability.yml` + (config under `monitoring/`, documented in `docs/observability-runbooks.md`). +- The backend `alertmanager_url` default is now empty. The + `/api/monitoring/alerts` and `/alertmanager-status` endpoints return a + graceful "not configured" response when `ALERTMANAGER_URL` is unset. +- Removed the now-orphaned combined `monitoring/prometheus/prometheus.yml`; the + standalone stack uses `monitoring/prometheus/prometheus.standalone.yml`. +- `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` remain as optional frontend deep-link + overrides. `ALERTMANAGER_URL`/`ALERTMANAGER_WEBHOOK_URL` are optional. + ### Added — Service registry - Runtime **service registry** persisted in the backend SQLite database. External diff --git a/README.md b/README.md index ea11026..978efe7 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,8 @@ Open the app at . The production Compose file requires OIDC and Traefik variables; see [Configuration](#configuration) below. Copy `.env.example` to `.env`, fill in the required values, and export them in your shell before running `docker compose up`. +> **Observability is external.** Manage only ships its **backend** and **frontend**. It does **not** deploy Grafana, Prometheus, Loki, Alertmanager, Alloy, or Node Exporter. The backend exposes a `/metrics` endpoint and optional Alertmanager proxy endpoints so an *existing* observability deployment can scrape and consume them. For a ready-to-run example stack you can deploy alongside Manage, see [`docker-compose.observability.yml`](docker-compose.observability.yml) and [`docs/observability-runbooks.md`](docs/observability-runbooks.md). + Local development with hot reload: ```bash @@ -81,7 +83,6 @@ Production-style example with shell exports: ```bash export BACKEND_APP_HOST=api.manage.example.com export FRONTEND_APP_HOST=manage.example.com -export GRAFANA_APP_HOST=grafana.manage.example.com export CERT_RESOLVER=letsencrypt export VITE_OIDC_ISSUER=https://auth.example.com/application/o/manage/ export VITE_OIDC_CLIENT_ID=manage @@ -97,7 +98,7 @@ docker compose up --build Inline one-liner example: ```bash -BACKEND_APP_HOST=api.manage.example.com FRONTEND_APP_HOST=manage.example.com GRAFANA_APP_HOST=grafana.manage.example.com CERT_RESOLVER=letsencrypt VITE_OIDC_ISSUER=https://auth.example.com/application/o/manage/ VITE_OIDC_CLIENT_ID=manage VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/ VITE_GRAFANA_URL=https://grafana.manage.example.com VITE_PROMETHEUS_URL=https://prometheus.manage.example.com docker compose up --build +BACKEND_APP_HOST=api.manage.example.com FRONTEND_APP_HOST=manage.example.com CERT_RESOLVER=letsencrypt VITE_OIDC_ISSUER=https://auth.example.com/application/o/manage/ VITE_OIDC_CLIENT_ID=manage VITE_OIDC_REDIRECT_URI=https://manage.example.com/oidc/callback VITE_OIDC_POST_LOGOUT_REDIRECT_URI=https://manage.example.com/ VITE_GRAFANA_URL=https://grafana.manage.example.com VITE_PROMETHEUS_URL=https://prometheus.manage.example.com docker compose up --build ``` For local development, no SSH key is required unless you want to connect to remote SSH machines later: @@ -182,6 +183,7 @@ cd frontend && npx tsc --noEmit && npm run build - Jellyfin server root URL required (not `/web`). The client strips trailing `/web` defensively. - SSH commands run through `/bin/sh -c` regardless of remote login shell. - Job templates are shell-quoted. Add new templates in `backend/src/media_library_viewer_api/jobs.py`. -- Root-level Docker Compose files are provided for production (`docker-compose.yml`) and local development (`docker-compose.dev.yml`), and both rely on Compose interpolation rather than `env_file` entries. +- Root-level Docker Compose files are provided for production (`docker-compose.yml`) and local development (`docker-compose.dev.yml`), and both rely on Compose interpolation rather than `env_file` entries. They deploy **only** the backend and frontend; Manage never deploys its own observability stack (see `docker-compose.observability.yml` for an optional standalone example). - The configurable dashboard stores widget instances in the backend SQLite settings database. New installs seed default Jellyfin activity and Backups widgets automatically. - Grafana and Prometheus widget adapters resolve URLs from service records configured in the app; `VITE_GRAFANA_URL` / `VITE_PROMETHEUS_URL` are only used for frontend deep-links. No credentials are stored in widget config; service API keys are encrypted at rest with `MANAGE_ENCRYPTION_KEY`. +- `ALERTMANAGER_URL` is optional. When unset, the Alertmanager proxy endpoints return graceful "not configured" responses instead of erroring. diff --git a/backend/src/media_library_viewer_api/config.py b/backend/src/media_library_viewer_api/config.py index 7056b28..3a08c1d 100644 --- a/backend/src/media_library_viewer_api/config.py +++ b/backend/src/media_library_viewer_api/config.py @@ -55,7 +55,7 @@ class Settings(BaseSettings): # Observability prometheus_enabled: bool = True prometheus_file_sd_dir: str = "/app/backend/.cache/prometheus-file-sd" - alertmanager_url: str = "http://alertmanager:9093" + alertmanager_url: str = "" alertmanager_webhook_url: str = "" # Optional receiver for alertmanager webhook notifications # Remote paths diff --git a/backend/src/media_library_viewer_api/routers/monitoring.py b/backend/src/media_library_viewer_api/routers/monitoring.py index 125a370..1d70143 100644 --- a/backend/src/media_library_viewer_api/routers/monitoring.py +++ b/backend/src/media_library_viewer_api/routers/monitoring.py @@ -88,11 +88,13 @@ def get_alertmanager_alerts() -> dict[str, Any]: """Return a summary of active Alertmanager alerts for the UI. Proxies the Alertmanager `/api/v1/alerts` endpoint and reshapes the payload - into a stable, UI-friendly format. If Alertmanager is unreachable, the - endpoint returns an empty summary and logs the failure so the UI can still - render a health card instead of an error page. + into a stable, UI-friendly format. If Alertmanager is unreachable (or not + configured via ``ALERTMANAGER_URL``), the endpoint returns an empty summary + so the UI can still render a health card instead of an error page. """ session, base_url = _alertmanager_client() + if not base_url: + return {"total": 0, "by_severity": {}, "alerts": [], "error": "alertmanager_not_configured"} try: response = session.get(f"{base_url}/api/v1/alerts", timeout=5) response.raise_for_status() @@ -115,9 +117,12 @@ def get_alertmanager_status() -> dict[str, Any]: Uses the Alertmanager `/api/v2/status` endpoint and exposes only the high- level fields the UI needs: uptime, version, and whether the cluster is - healthy. + healthy. Returns ``up=False`` when Alertmanager is unreachable or not + configured via ``ALERTMANAGER_URL``. """ session, base_url = _alertmanager_client() + if not base_url: + return {"up": False, "version": "", "uptime": ""} try: response = session.get(f"{base_url}/api/v2/status", timeout=5) response.raise_for_status() diff --git a/backend/tests/test_api.py b/backend/tests/test_api.py index bbc7ff9..bb89a1e 100644 --- a/backend/tests/test_api.py +++ b/backend/tests/test_api.py @@ -677,6 +677,17 @@ class TestAlertmanager: assert data["total"] == 0 assert data["error"] == "alertmanager_unreachable" + def test_alerts_endpoint_when_not_configured(self, test_client): + with patch("media_library_viewer_api.routers.monitoring._alertmanager_client") as mock_client: + session = MagicMock() + mock_client.return_value = (session, "") + response = test_client.get("/api/monitoring/alerts") + assert response.status_code == 200 + data = response.json() + assert data["total"] == 0 + assert data["error"] == "alertmanager_not_configured" + session.get.assert_not_called() + def test_alerts_endpoint_returns_summary(self, test_client): with patch("media_library_viewer_api.routers.monitoring._alertmanager_client") as mock_client: session = MagicMock() @@ -700,6 +711,16 @@ class TestAlertmanager: assert data["by_severity"]["critical"] == 1 assert data["alerts"][0]["name"] == "BackupJobFailed" + def test_alertmanager_status_endpoint_when_not_configured(self, test_client): + with patch("media_library_viewer_api.routers.monitoring._alertmanager_client") as mock_client: + session = MagicMock() + mock_client.return_value = (session, "") + response = test_client.get("/api/monitoring/alertmanager-status") + assert response.status_code == 200 + data = response.json() + assert data["up"] is False + session.get.assert_not_called() + def test_alertmanager_status_endpoint_when_unreachable(self, test_client): with patch("media_library_viewer_api.routers.monitoring._alertmanager_client") as mock_client: session = MagicMock() diff --git a/backend/tests/test_widgets.py b/backend/tests/test_widgets.py index a033307..68736fa 100644 --- a/backend/tests/test_widgets.py +++ b/backend/tests/test_widgets.py @@ -369,7 +369,6 @@ async def test_ssh_task_adapter_records_history_on_run(client): fake_result = SimpleNamespace(exit_status=0, stdout="hi\n", stderr="") fake_client = SimpleNamespace(run=lambda *a, **k: fake_result) - from media_library_viewer_api.services.task_runner import build_ssh_client from media_library_viewer_api.widgets.sources import SshTaskWidgetSource adapter = SshTaskWidgetSource() diff --git a/context.md b/context.md index 0a5d3d1..17a5e51 100644 --- a/context.md +++ b/context.md @@ -1,6 +1,16 @@ # Code Context +> **Status (2026-06-23):** Manage no longer deploys an observability stack. +> The root `docker-compose.yml` / `docker-compose.dev.yml` ship **only** the +> backend and frontend; Grafana, Prometheus, Loki, Alertmanager, Alloy, and Node +> Exporter were removed from them. Manage connects to **existing** instances. +> The standalone example stack lives in `docker-compose.observability.yml`. Some +> snippets below still reference the former in-compose services and are kept as +> historical context; treat `docker-compose.observability.yml` as authoritative +> for the stack layout. + ## Files Retrieved + 1. `docker-compose.yml` (lines 1–262) – production Compose stack; defines observability services and Traefik routing. 2. `docker-compose.dev.yml` (lines 1–234) – development Compose stack; same observability services but with host ports exposed and auth disabled. 3. `.env.example` (lines 1–55) – template with all required environment variables for the stack, including Prometheus/Grafana/Alertmanager/Alloy/Loki and Node Exporter settings. @@ -20,7 +30,9 @@ ## Key Code ### Backend `/metrics` endpoint + `backend/src/media_library_viewer_api/main.py`: + ```python @app.middleware("http") async def enforce_jwt_auth(request: Request, call_next): @@ -36,7 +48,9 @@ def metrics() -> Response: ``` ### Metrics emitted by the backend + `backend/src/media_library_viewer_api/observability.py`: + ```python REQUESTS_TOTAL = Counter("manage_api_requests_total", "Total API requests", ["method", "path", "status_code"]) REQUEST_DURATION = Histogram("manage_api_request_duration_seconds", "API request duration", ["method", "path"], ...) @@ -48,7 +62,9 @@ MAIL_QUEUE_SIZE = Counter("manage_mail_queue_messages_total", "Total messages en ``` ### Prometheus scrape configuration + `monitoring/prometheus/prometheus.yml`: + ```yaml scrape_configs: - job_name: manage-backend @@ -71,7 +87,9 @@ scrape_configs: ``` ### Backend-managed remote Node Exporter targets + `backend/src/media_library_viewer_api/services/targets.py`: + ```python def build_node_exporter_targets(store: SettingsStore) -> list[dict[str, Any]]: ... @@ -96,6 +114,7 @@ The observability stack is a standard self-hosted Prometheus/Grafana/Loki/Alertm - **Alertmanager** routes alerts by severity (critical vs warning) and delivers email notifications (and optionally a webhook back to the backend). The backend bridges the stack with the application: + - It exposes `/metrics` (unauthenticated, along with `/api/health` and `/api/version`). - On startup it writes `${PROMETHEUS_FILE_SD_DIR}/node_exporter_targets.json` based on enabled SSH machines in the settings store. - It provides proxy endpoints (`/api/monitoring/alerts`, `/api/monitoring/alertmanager-status`, `/api/monitoring/prometheus-targets`) consumed by the frontend. @@ -105,6 +124,7 @@ The backend bridges the stack with the application: Open `monitoring/prometheus/prometheus.yml` first to understand what is scraped and how the backend is wired, then read `backend/src/media_library_viewer_api/observability.py` to see the metric names and labels. For environment requirements, read `.env.example`. ## Supervisor coordination + Not needed — this is a read-only scouting summary. --- @@ -113,7 +133,7 @@ Not needed — this is a read-only scouting summary. ## 1. Observability services defined in Compose -Both `docker-compose.yml` and `docker-compose.dev.yml` define the following services: +Both `docker-compose.yml` and `docker-compose.dev.yml` define **only the backend and frontend**. The observability services (Prometheus, Loki, Grafana, Alertmanager, Alloy, Node Exporter) were extracted to the standalone `docker-compose.observability.yml` example stack and are **no longer** deployed by Manage. Summary of what remains in the app compose files: | Service | Image | Internal endpoint | Purpose | |---------|-------|-------------------|---------| @@ -126,6 +146,7 @@ Both `docker-compose.yml` and `docker-compose.dev.yml` define the following serv | `backend` | Build from `backend/Dockerfile` | `http://backend:8000` | FastAPI app exposing `/metrics` | Differences: + - Production (`docker-compose.yml`): services attach to an external `web` network for Traefik, use `expose` instead of host ports for most services, and require OIDC/auth variables. - Development (`docker-compose.dev.yml`): Prometheus/Grafana/Loki/Alertmanager/Node Exporter are published on host ports `9090`, `3000`, `3100`, `9093`, `9100`; auth is disabled (`AUTH_ENABLED=false`). @@ -134,6 +155,7 @@ Differences: From `.env.example` and the Compose files, the variables relevant to the observability stack are: ### Backend / metrics + - `PROMETHEUS_ENABLED` – enable metrics endpoint (set to `"true"` in both compose files). - `PROMETHEUS_FILE_SD_DIR` – directory where the backend writes `node_exporter_targets.json` (default `/app/backend/.cache/prometheus-file-sd`). - `ALERTMANAGER_URL` – backend proxy target (default `http://alertmanager:9093`). @@ -141,6 +163,7 @@ From `.env.example` and the Compose files, the variables relevant to the observa - `BACKEND_CACHE_DIR` – host directory mounted into backend and Prometheus for file-SD. ### Grafana + - `GRAFANA_APP_HOST` – public hostname for Grafana (production; required). - `GRAFANA_APP_PORT` – defaults to `3000`. - `GRAFANA_APP_NAME` – defaults to `grafana`. @@ -152,16 +175,19 @@ From `.env.example` and the Compose files, the variables relevant to the observa - `GF_AUTH_GENERIC_OAUTH_API_URL` ### Alertmanager + - `SMTP_HOST` / `SMTP_PORT` - `SMTP_USERNAME` / `SMTP_PASSWORD` - `SMTP_FROM_ADDRESS` - `ALERT_EMAIL_TO` ### Traefik / network (production) + - `BACKEND_APP_HOST` / `FRONTEND_APP_HOST` / `GRAFANA_APP_HOST` - `CERT_RESOLVER` – e.g. `letsencrypt` ### General + - `LOG_LEVEL` / `LOG_FORMAT` – also passed to Grafana and backend. ## 3. Monitoring config files @@ -193,7 +219,7 @@ Remote Node Exporter targets are not static: the backend reads machine settings ## 5. Setup steps and gotchas -- The observability stack is brought up with the app itself: +- Manage's own compose stack does **not** include observability services. To run a full local stack, bring up the app and the standalone observability example separately: - Production: `docker compose -f docker-compose.yml up --build` - Development: `docker compose -f docker-compose.dev.yml up --build` - Production requires the external `web` network and Traefik already configured; `docker-compose.dev.yml` does not use Traefik and binds ports directly. diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index b163424..bfc73fc 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -1,3 +1,12 @@ +# Manage development stack — backend + frontend only. +# +# Manage does NOT deploy its own Grafana/Prometheus/Loki/Alertmanager stack. +# It exposes a /metrics endpoint and optional Alertmanager proxy endpoints so +# an *existing* observability deployment can scrape/consume them. +# +# For a ready-to-use example observability stack, see +# docker-compose.observability.yml and docs/observability-runbooks.md. + services: backend: build: @@ -15,7 +24,7 @@ services: PROMETHEUS_ENABLED: "true" SSH_KNOWN_HOSTS_PATH: /app/backend/.cache/known_hosts PROMETHEUS_FILE_SD_DIR: /app/backend/.cache/prometheus-file-sd - ALERTMANAGER_URL: ${ALERTMANAGER_URL:-http://alertmanager:9093} + ALERTMANAGER_URL: ${ALERTMANAGER_URL:-} ALERTMANAGER_WEBHOOK_URL: ${ALERTMANAGER_WEBHOOK_URL:-} MANAGE_ENCRYPTION_KEY: ${MANAGE_ENCRYPTION_KEY:?set MANAGE_ENCRYPTION_KEY in your .env} ports: @@ -25,7 +34,6 @@ services: - ${BACKEND_CACHE_DIR:-./backend-cache}:/app/backend/.cache networks: - web - - monitoring restart: unless-stopped frontend: @@ -38,8 +46,8 @@ services: VITE_API_URL: "/api" VITE_OIDC_ENABLED: "false" VITE_DEV_API_PROXY_TARGET: "http://backend:8000" - VITE_GRAFANA_URL: "http://localhost:3000" - VITE_PROMETHEUS_URL: "http://localhost:9090" + VITE_GRAFANA_URL: ${VITE_GRAFANA_URL:-http://localhost:3000} + VITE_PROMETHEUS_URL: ${VITE_PROMETHEUS_URL:-http://localhost:9090} ports: - "5173:5173" volumes: @@ -47,211 +55,12 @@ services: - frontend_node_modules:/app/frontend/node_modules depends_on: - backend - restart: unless-stopped - - prometheus: - image: prom/prometheus:v2.55.1 - container_name: prometheus - command: - - "--config.file=/etc/prometheus/prometheus.yml" - - "--storage.tsdb.path=/prometheus" - - "--storage.tsdb.retention.time=30d" - - "--web.console.libraries=/usr/share/prometheus/console_libraries" - - "--web.console.templates=/usr/share/prometheus/consoles" - - "--web.enable-lifecycle" - volumes: - - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - - ./monitoring/prometheus/rules:/etc/prometheus/rules:ro - - ${BACKEND_CACHE_DIR:-./backend-cache}/prometheus-file-sd:/etc/prometheus/file-sd:ro - - prometheus_data:/prometheus - ports: - - "9090:9090" networks: - - monitoring + - web restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9090/-/healthy"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "1.00" - memory: 1G - reservations: - cpus: "0.25" - memory: 256M - - loki: - image: grafana/loki:3.1.1 - container_name: loki - command: -config.file=/etc/loki/loki.yml - volumes: - - ./monitoring/loki/loki.yml:/etc/loki/loki.yml:ro - - loki_data:/loki - ports: - - "3100:3100" - networks: - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3100/ready"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "1.00" - memory: 1G - reservations: - cpus: "0.25" - memory: 256M - - alloy: - image: grafana/alloy:v1.5.0 - container_name: alloy - command: - - run - - /etc/alloy/config.alloy - - --storage.path=/var/lib/alloy - volumes: - - ./monitoring/alloy/config.alloy:/etc/alloy/config.alloy:ro - - /var/run/docker.sock:/var/run/docker.sock:ro - - /var/lib/docker/containers:/var/lib/docker/containers:ro - networks: - - monitoring - depends_on: - loki: - condition: service_healthy - restart: unless-stopped - user: root - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:12345/-/healthy"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "0.50" - memory: 512M - reservations: - cpus: "0.10" - memory: 128M - - grafana: - image: grafana/grafana:11.3.1 - container_name: grafana - environment: - GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin} - GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin} - GF_AUTH_GENERIC_OAUTH_CLIENT_ID: ${GF_AUTH_GENERIC_OAUTH_CLIENT_ID:-} - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: ${GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:-} - GF_AUTH_GENERIC_OAUTH_AUTH_URL: ${GF_AUTH_GENERIC_OAUTH_AUTH_URL:-} - GF_AUTH_GENERIC_OAUTH_TOKEN_URL: ${GF_AUTH_GENERIC_OAUTH_TOKEN_URL:-} - GF_AUTH_GENERIC_OAUTH_API_URL: ${GF_AUTH_GENERIC_OAUTH_API_URL:-} - LOG_LEVEL: ${LOG_LEVEL:-INFO} - volumes: - - ./monitoring/grafana/grafana.ini:/etc/grafana/grafana.ini:ro - - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro - - grafana_data:/var/lib/grafana - ports: - - "3000:3000" - networks: - - monitoring - depends_on: - prometheus: - condition: service_healthy - loki: - condition: service_healthy - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/api/health"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "1.00" - memory: 512M - reservations: - cpus: "0.25" - memory: 128M - - alertmanager: - image: prom/alertmanager:v0.27.0 - container_name: alertmanager - command: - - "--config.file=/etc/alertmanager/alertmanager.yml" - - "--storage.path=/alertmanager" - environment: - SMTP_HOST: ${SMTP_HOST:-smtp.example.com} - SMTP_PORT: ${SMTP_PORT:-587} - SMTP_USERNAME: ${SMTP_USERNAME:-} - SMTP_PASSWORD: ${SMTP_PASSWORD:-} - SMTP_FROM_ADDRESS: ${SMTP_FROM_ADDRESS:-no-reply@example.com} - ALERT_EMAIL_TO: ${ALERT_EMAIL_TO:-admin@example.com} - volumes: - - ./monitoring/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro - - alertmanager_data:/alertmanager - ports: - - "9093:9093" - networks: - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9093/-/healthy"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "0.50" - memory: 256M - reservations: - cpus: "0.10" - memory: 64M - - node-exporter: - image: prom/node-exporter:v1.8.2 - container_name: node-exporter - command: - - "--path.rootfs=/host" - volumes: - - /:/host:ro,rslave - networks: - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9100/"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 10s - deploy: - resources: - limits: - cpus: "0.25" - memory: 128M - reservations: - cpus: "0.05" - memory: 32M networks: - monitoring: + web: volumes: frontend_node_modules: - prometheus_data: - loki_data: - grafana_data: - alertmanager_data: diff --git a/docker-compose.observability.yml b/docker-compose.observability.yml index 62ab10e..0b84c28 100644 --- a/docker-compose.observability.yml +++ b/docker-compose.observability.yml @@ -1,4 +1,13 @@ -# Standalone observability stack. +# Standalone observability stack (OPTIONAL EXAMPLE). +# +# Manage itself does NOT deploy Grafana/Prometheus/Loki/Alertmanager/Alloy/Node +# Exporter. The root docker-compose.yml / docker-compose.dev.yml ship only the +# backend and frontend, and Manage connects to EXISTING Grafana/Prometheus/ +# Alertmanager instances. +# +# This file is a ready-to-use example you can run on a host where you want the +# full Grafana, Prometheus, Loki, Alertmanager, Alloy and Node Exporter stack +# without the rest of Manage. It is not required to run Manage. # # Run this on a host where you want Grafana, Prometheus, Loki, Alertmanager, # Alloy and Node Exporter without the rest of Manage. diff --git a/docker-compose.yml b/docker-compose.yml index 85362a1..a705f6e 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,3 +1,12 @@ +# Manage production stack — backend + frontend only. +# +# Manage does NOT deploy its own Grafana/Prometheus/Loki/Alertmanager stack. +# It exposes a /metrics endpoint and optional Alertmanager proxy endpoints so +# an *existing* observability deployment can scrape/consume them. +# +# For a ready-to-use example observability stack, see +# docker-compose.observability.yml and docs/observability-runbooks.md. + services: backend: build: @@ -26,7 +35,7 @@ services: SMTP_TIMEOUT: ${SMTP_TIMEOUT:-30} SSH_KNOWN_HOSTS_PATH: /app/backend/.cache/known_hosts PROMETHEUS_FILE_SD_DIR: ${PROMETHEUS_FILE_SD_DIR:-/app/backend/.cache/prometheus-file-sd} - ALERTMANAGER_URL: ${ALERTMANAGER_URL:-http://alertmanager:9093} + ALERTMANAGER_URL: ${ALERTMANAGER_URL:-} ALERTMANAGER_WEBHOOK_URL: ${ALERTMANAGER_WEBHOOK_URL:-} MANAGE_ENCRYPTION_KEY: ${MANAGE_ENCRYPTION_KEY:?generate one with python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"} volumes: @@ -34,7 +43,6 @@ services: restart: unless-stopped networks: - web - - monitoring expose: - "8000" labels: @@ -71,7 +79,7 @@ services: VITE_OIDC_POST_LOGOUT_REDIRECT_URI: ${VITE_OIDC_POST_LOGOUT_REDIRECT_URI:?set VITE_OIDC_POST_LOGOUT_REDIRECT_URI} VITE_DEV_API_PROXY_TARGET: ${VITE_DEV_API_PROXY_TARGET:-http://backend:8000} VITE_GRAFANA_URL: ${VITE_GRAFANA_URL:-https://grafana.example.com} - VITE_PROMETHEUS_URL: ${VITE_PROMETHEUS_URL:-http://localhost:9090} + VITE_PROMETHEUS_URL: ${VITE_PROMETHEUS_URL:-http://prometheus.example.com} VITE_APP_VERSION: ${APP_VERSION:-0.1.0} VITE_APP_BUILD_INFO: ${APP_BUILD_INFO:-dev} depends_on: @@ -89,218 +97,6 @@ services: - "8080:80" restart: unless-stopped - prometheus: - image: prom/prometheus:v2.55.1 - container_name: prometheus - command: - - "--config.file=/etc/prometheus/prometheus.yml" - - "--storage.tsdb.path=/prometheus" - - "--storage.tsdb.retention.time=30d" - - "--web.console.libraries=/usr/share/prometheus/console_libraries" - - "--web.console.templates=/usr/share/prometheus/consoles" - - "--web.enable-lifecycle" - volumes: - - ./monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - - ./monitoring/prometheus/rules:/etc/prometheus/rules:ro - - ${BACKEND_CACHE_DIR:-./backend-cache}/prometheus-file-sd:/etc/prometheus/file-sd:ro - - prometheus_data:/prometheus - expose: - - "9090" - networks: - - web - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9090/-/healthy"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "1.00" - memory: 1G - reservations: - cpus: "0.25" - memory: 256M - - loki: - image: grafana/loki:3.1.1 - container_name: loki - command: -config.file=/etc/loki/loki.yml - volumes: - - ./monitoring/loki/loki.yml:/etc/loki/loki.yml:ro - - loki_data:/loki - expose: - - "3100" - networks: - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3100/ready"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "1.00" - memory: 1G - reservations: - cpus: "0.25" - memory: 256M - - alloy: - image: grafana/alloy:v1.5.0 - container_name: alloy - command: - - run - - /etc/alloy/config.alloy - - --storage.path=/var/lib/alloy - volumes: - - ./monitoring/alloy/config.alloy:/etc/alloy/config.alloy:ro - - /var/run/docker.sock:/var/run/docker.sock:ro - - /var/lib/docker/containers:/var/lib/docker/containers:ro - networks: - - monitoring - depends_on: - loki: - condition: service_healthy - restart: unless-stopped - user: root - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:12345/-/healthy"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "0.50" - memory: 512M - reservations: - cpus: "0.10" - memory: 128M - - grafana: - image: grafana/grafana:11.3.1 - container_name: grafana - environment: - GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin} - GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin} - GF_AUTH_GENERIC_OAUTH_CLIENT_ID: ${GF_AUTH_GENERIC_OAUTH_CLIENT_ID:-} - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: ${GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:-} - GF_AUTH_GENERIC_OAUTH_AUTH_URL: ${GF_AUTH_GENERIC_OAUTH_AUTH_URL:-} - GF_AUTH_GENERIC_OAUTH_TOKEN_URL: ${GF_AUTH_GENERIC_OAUTH_TOKEN_URL:-} - GF_AUTH_GENERIC_OAUTH_API_URL: ${GF_AUTH_GENERIC_OAUTH_API_URL:-} - LOG_LEVEL: ${LOG_LEVEL:-INFO} - volumes: - - ./monitoring/grafana/grafana.ini:/etc/grafana/grafana.ini:ro - - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro - - grafana_data:/var/lib/grafana - expose: - - "3000" - networks: - - web - - monitoring - depends_on: - prometheus: - condition: service_healthy - loki: - condition: service_healthy - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/api/health"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - labels: - - "traefik.enable=true" - - "traefik.http.routers.${GRAFANA_APP_NAME:-grafana}.rule=Host(`${GRAFANA_APP_HOST:?set GRAFANA_APP_HOST}`)" - - "traefik.http.routers.${GRAFANA_APP_NAME:-grafana}.entrypoints=websecure" - - "traefik.http.routers.${GRAFANA_APP_NAME:-grafana}.tls.certresolver=${CERT_RESOLVER:?set CERT_RESOLVER}" - - "traefik.http.services.${GRAFANA_APP_NAME:-grafana}.loadbalancer.server.port=${GRAFANA_APP_PORT:-3000}" - deploy: - resources: - limits: - cpus: "1.00" - memory: 512M - reservations: - cpus: "0.25" - memory: 128M - - alertmanager: - image: prom/alertmanager:v0.27.0 - container_name: alertmanager - command: - - "--config.file=/etc/alertmanager/alertmanager.yml" - - "--storage.path=/alertmanager" - environment: - SMTP_HOST: ${SMTP_HOST:-smtp.example.com} - SMTP_PORT: ${SMTP_PORT:-587} - SMTP_USERNAME: ${SMTP_USERNAME:-} - SMTP_PASSWORD: ${SMTP_PASSWORD:-} - SMTP_FROM_ADDRESS: ${SMTP_FROM_ADDRESS:-no-reply@example.com} - ALERT_EMAIL_TO: ${ALERT_EMAIL_TO:-admin@example.com} - volumes: - - ./monitoring/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro - - alertmanager_data:/alertmanager - expose: - - "9093" - networks: - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9093/-/healthy"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 30s - deploy: - resources: - limits: - cpus: "0.50" - memory: 256M - reservations: - cpus: "0.10" - memory: 64M - - node-exporter: - image: prom/node-exporter:v1.8.2 - container_name: node-exporter - command: - - "--path.rootfs=/host" - volumes: - - /:/host:ro,rslave - networks: - - monitoring - restart: unless-stopped - healthcheck: - test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9100/"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 10s - deploy: - resources: - limits: - cpus: "0.25" - memory: 128M - reservations: - cpus: "0.05" - memory: 32M - networks: web: external: true - monitoring: - -volumes: - prometheus_data: - loki_data: - grafana_data: - alertmanager_data: diff --git a/docs/MIGRATION_PLAN.md b/docs/MIGRATION_PLAN.md index 964aad8..d4f310b 100644 --- a/docs/MIGRATION_PLAN.md +++ b/docs/MIGRATION_PLAN.md @@ -92,7 +92,7 @@ repo/ │ └── types/ │ └── index.ts │ -├── docker-compose.yml # Optional unified deployment +├── docker-compose.yml # Backend + frontend deployment (observability is external) └── README.md ``` diff --git a/docs/REQUIREMENTS.md b/docs/REQUIREMENTS.md index bdf015c..64498a3 100644 --- a/docs/REQUIREMENTS.md +++ b/docs/REQUIREMENTS.md @@ -197,7 +197,7 @@ fully removed (web-ui-rework; see decision log 2026-06-17). - Dashboard shortcuts should support a small icon/preview field so cards can be visually recognizable without changing the underlying model later. - Support OIDC login in the frontend using an OIDC client library, with backend JWT validation for protected API requests. - Persist frontend OIDC auth state across tab reloads by storing the OIDC user and request state in browser localStorage. -- Provide Docker Compose deployment files at the repository root for production and local development. +- Provide Docker Compose deployment files at the repository root for production and local development. These deploy **only** the backend and frontend; Manage connects to *existing* Grafana/Prometheus/Alertmanager instances and never ships its own observability stack (see `docker-compose.observability.yml` for an optional standalone example). - SSH private keys should be managed as reusable saved secrets in Settings, independent of any one machine, and SSH machines should select from that saved-key list. - The web UI should allow both importing an existing private key and generating a new SSH keypair for that saved-key list. - Saved SSH keys should display their derived public key, fingerprint, and machine usage count so administrators can audit them at a glance. @@ -331,6 +331,7 @@ the widget/addon-pages model were removed. `MANAGE_ENCRYPTION_KEY` is now requir ## Decision Log +- 2026-06-23: Removed all observability services from the root Compose files. `docker-compose.yml` and `docker-compose.dev.yml` now deploy **only** the backend and frontend; the `monitoring` network and the `prometheus`/`loki`/`alloy`/`grafana`/`alertmanager`/`node-exporter` services and their named volumes were deleted, and the `GRAFANA_APP_HOST` Traefik rule was removed. Manage connects to **existing** Grafana/Prometheus/Alertmanager instances and does not deploy them itself. The backend `alertmanager_url` default is now empty, and the `/api/monitoring/alerts` and `/alertmanager-status` endpoints return graceful "not configured" responses when `ALERTMANAGER_URL` is unset. The previous in-compose stack is preserved as an optional standalone example in `docker-compose.observability.yml` (config under `monitoring/`, documented in `docs/observability-runbooks.md`). `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` remain as optional frontend deep-link overrides. - 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template. - 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring` → `/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`. - 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`. diff --git a/docs/monitoring-logging-design.md b/docs/monitoring-logging-design.md index c39d9ea..9f80156 100644 --- a/docs/monitoring-logging-design.md +++ b/docs/monitoring-logging-design.md @@ -1,5 +1,16 @@ # Monitoring and Logging Design — Manage +> **Status (2026-06-23):** This document is the **reference architecture** for the +> self-hosted observability stack Manage integrates with. Manage itself **does +> not deploy** Grafana, Prometheus, Loki, Alertmanager, Alloy, or Node Exporter. +> The root `docker-compose.yml` / `docker-compose.dev.yml` ship **only** the +> backend and frontend; Manage connects to **existing** Grafana / Prometheus / +> Alertmanager instances. The stack described here is available as an optional, +> deploy-it-yourself example in `docker-compose.observability.yml` (see +> `docs/observability-runbooks.md`). The "Add services to `docker-compose.yml`" +> phase notes below describe the **original** in-compose deployment, which has +> since been extracted to the standalone compose file. + ## Executive Summary Manage currently uses ad-hoc observability: plain-text Python logs, a custom POSIX shell metrics collector on remote machines, and a background poller that stores snapshots in SQLite. This works for a single-instance homelab but becomes painful as the fleet grows and as users need faster incident response. diff --git a/docs/observability-runbooks.md b/docs/observability-runbooks.md index 2ad8ed8..e7e32ea 100644 --- a/docs/observability-runbooks.md +++ b/docs/observability-runbooks.md @@ -1,6 +1,10 @@ # Observability Runbooks -Operational playbooks for the Manage self-hosted observability stack (Prometheus, Grafana, Loki, Alertmanager). +Operational playbooks for the **standalone example observability stack** +(`docker-compose.observability.yml`) that can be deployed alongside Manage. +Manage itself does **not** deploy these services; it connects to existing +Grafana / Prometheus / Alertmanager instances. These runbooks cover operating +the standalone stack shipped under `monitoring/`. ## Service Overview diff --git a/frontend/README.md b/frontend/README.md index 9e62af2..90f9803 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -8,10 +8,9 @@ React + TypeScript SPA for Manage, consuming the FastAPI backend. - **React 18+** — UI framework - **TypeScript** — Type safety - **@tanstack/react-query** — Data fetching/caching -- **ag-grid-react** — Data tables (media, file browser) -- **recharts** — Monitoring charts +- **@tanstack/react-table** — Data tables (media, file browser) - **react-router-dom** — Client-side routing -- **Tailwind CSS** — Styling +- **Tailwind CSS** + **shadcn/ui** — Styling ## Setup @@ -26,7 +25,7 @@ npm install npm run dev ``` -Runs on http://localhost:5173 with API requests proxied to http://localhost:8000. +Runs on with API requests proxied to . Make sure the backend is running: @@ -45,12 +44,13 @@ Output goes to `frontend/dist/`. ## Pages -- **Dashboard** (`/`) — Now playing, backend-collected per-machine monitoring table with 10-minute averages/min/max, library stats, frontend/backend version chips in the shell header -- **Monitoring** (`/monitoring`) — Per-machine CPU/IO wait/RAM/network/disk charts, collector controls, and backend-collected recent action history +- **Dashboard** (`/`) — Now playing, library stats, configurable widgets and shortcuts, frontend/backend version chips in the shell header +- **Observability** (`/observability`) — Thin dashboard: Alertmanager alerts, Prometheus target health, machine status, and Grafana deep-links (no in-app charting) - **Media** (`/media`) — Full-library table with sort/filter/search - **Users** (`/users`) — Read-only Jellyfin user list with optional Jellyseerr enrichment - **File Browser** (`/files`) — Remote directory browsing, ffprobe preview, jobs -- **Settings** (`/settings`) — Persistent monitoring machine definitions and setup workflow +- **Settings** (`/settings`) — Persistent monitoring machine definitions, SSH keys, and setup workflow +- **Actions** (`/actions`) — Saved server tasks (shell/python) targeting ssh_tasks services ## Environment Variables @@ -65,7 +65,6 @@ VITE_API_URL=http://your-backend-host:8000 - **Local development**: run `docker compose -f docker-compose.dev.yml up --build`, then open the app and add monitoring machines in the **Settings** tab. - **Production**: export the required Compose variables in your shell, run `docker compose up --build`, and manage local/remote machines from **Settings**. -- **Dashboard monitoring UI**: the dashboard shows a compact, sortable table with one row per configured machine plus poller status. -- **Monitoring UI**: the **Monitoring** tab shows one card per configured machine, including a recent action-history table populated by the backend poller. A machine can be `local` (the API host itself) or `ssh` (a remote host), and the UI treats both the same after configuration. The page also shows the backend poller health badge. +- **Observability**: Manage only deploys backend + frontend. It connects to **existing** Grafana/Prometheus/Alertmanager instances; see `docker-compose.observability.yml` for an optional standalone example stack. A machine can be `local` (the API host itself) or `ssh` (a remote host), and the UI treats both the same after configuration. -In development, the Vite proxy handles `/api` requests automatically. +In development, the Vite proxy handles `/api` requests automatically. `VITE_GRAFANA_URL` / `VITE_PROMETHEUS_URL` are optional frontend deep-link overrides for your existing instances. diff --git a/monitoring/prometheus/prometheus.yml b/monitoring/prometheus/prometheus.yml deleted file mode 100644 index 1c02edc..0000000 --- a/monitoring/prometheus/prometheus.yml +++ /dev/null @@ -1,48 +0,0 @@ -global: - scrape_interval: 15s - evaluation_interval: 15s - external_labels: - monitor: manage - -alerting: - alertmanagers: - - static_configs: - - targets: - - alertmanager:9093 - -rule_files: - - /etc/prometheus/rules/*.yml - -scrape_configs: - - job_name: prometheus - static_configs: - - targets: - - localhost:9090 - - - job_name: manage-backend - static_configs: - - targets: - - backend:8000 - metrics_path: /metrics - scrape_interval: 15s - - - job_name: node-exporter - static_configs: - - targets: - - node-exporter:9100 - - - job_name: loki - static_configs: - - targets: - - loki:3100 - - - job_name: alertmanager - static_configs: - - targets: - - alertmanager:9093 - - - job_name: node-exporter-remote - file_sd_configs: - - files: - - /etc/prometheus/file-sd/node_exporter_targets.json - refresh_interval: 30s