chore(observability): remove remaining observability env vars, docs

Slice 5 (final) of observability-service-registry. Completes the move to service-registry-only observability config: no observability service env vars remain. - config.py: removed alertmanager_url + alertmanager_webhook_url fields. - docker-compose.yml / docker-compose.dev.yml: removed ALERTMANAGER_URL, ALERTMANAGER_WEBHOOK_URL (backend env), and VITE_GRAFANA_URL, VITE_PROMETHEUS_URL (frontend build args / dev env). - frontend/Dockerfile: removed the VITE_GRAFANA_URL / VITE_PROMETHEUS_URL ARG, build-stage ENV, and dev-stage ENV lines. - docs: REQUIREMENTS decision-log entry; CHANGELOG Added/Changed/BREAKING for the observability service registry; backend/README monitoring section (Observability page, services page config, http_sd_configs, new health endpoints, log-only webhook). The only observability env var remaining is PROMETHEUS_ENABLED (Manage's own /metrics toggle). Grep-gated: no live references to the removed vars/fields in backend src, frontend src, compose, or Dockerfile. ruff clean; 239 backend tests pass; frontend 0 lint errors, build clean, 72 tests pass. .env.example is assistant-edit-blocked; user follow-up noted in the SDD tasks: drop the removed vars there too.
2026-06-24 08:49:53 +00:00
parent b200025daa
commit b1a66a1ab7
7 changed files with 35 additions and 25 deletions
@@ -4,6 +4,20 @@ All notable changes to Manage. Breaking changes are marked with **BREAKING**.

 ## [Unreleased]

+### Added — Observability service registry
+
+- **Alertmanager is now a service type.** Configure Alertmanager, Grafana, and
+  Prometheus instances in the UI on the Services page; all three are first-class
+  service-registry entries with dashboard widgets (`active_alerts`, Grafana link,
+  Prometheus metric).
+- New monitoring endpoints resolve the configured service instance and probe its
+  health: `GET /api/monitoring/grafana-status`, `/prometheus-status`. The
+  `/alerts` and `/alertmanager-status` endpoints now take an optional
+  `service_id` and pick the first enabled alertmanager instance by default.
+- The Observability page discovers Grafana/Prometheus/Alertmanager from the
+  registry and renders health cards; the dashboard `active_alerts` widget sums
+  firing alerts by severity.
+
 ### Changed — Observability is now external only

 - **Removed** all observability services from `docker-compose.yml` and
@@ -15,13 +29,22 @@ All notable changes to Manage. Breaking changes are marked with **BREAKING**.
  and never ships its own stack. The previous in-compose stack is preserved as
  an optional, deploy-it-yourself example in `docker-compose.observability.yml`
  (config under `monitoring/`, documented in `docs/observability-runbooks.md`).
- The backend `alertmanager_url` default is now empty. The
-  `/api/monitoring/alerts` and `/alertmanager-status` endpoints return a
-  graceful "not configured" response when `ALERTMANAGER_URL` is unset.
 - Removed the now-orphaned combined `monitoring/prometheus/prometheus.yml`; the
  standalone stack uses `monitoring/prometheus/prometheus.standalone.yml`.
- `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` remain as optional frontend deep-link
-  overrides. `ALERTMANAGER_URL`/`ALERTMANAGER_WEBHOOK_URL` are optional.
+- Removed the Prometheus file-SD bridge (`PROMETHEUS_FILE_SD_DIR` + the
+  `write_prometheus_targets` file writer). External Prometheus instances now
+  consume node-exporter targets via `http_sd_configs` against
+  `GET /api/monitoring/prometheus-targets`. The webhook receiver is log-only.
+
+### **BREAKING**
+
+- Observability is configured entirely via the service registry; the backend
+  `alertmanager_url`/`alertmanager_webhook_url` and frontend
+  `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` environment variables, plus
+  `PROMETHEUS_FILE_SD_DIR`, were **removed**. Re-create your Alertmanager /
+  Grafana / Prometheus instances on the Services page after upgrading. The only
+  observability env var remaining is `PROMETHEUS_ENABLED` (toggles Manage's own
+  `/metrics` endpoint).

 ### Added — Service registry

@@ -132,7 +132,7 @@ docker compose up --build
   - Use **Validate SSH + trust host** in the machine editor before saving if you want to test the banner/auth flow explicitly.
   - The first successful SSH connection uses trust-on-first-use: the backend records that machine's host key into its managed `known_hosts` file automatically, then continues verifying it strictly on later connects.

-2. Open **Monitoring** to see Alertmanager alerts, Prometheus scrape targets, and Grafana deep-links for configured machines. System metrics (disk, CPU, memory) are owned by the external observability stack (Prometheus + node_exporter + Grafana), not by the Manage backend.
+2. Open **Observability** to see Alertmanager alerts, Prometheus scrape targets, and Grafana deep-links for configured machines. Alertmanager, Grafana, and Prometheus are configured as service instances on the **Services** page; system metrics (disk, CPU, memory) are owned by the external observability stack (Prometheus + node_exporter + Grafana), not by the Manage backend.

 For local development, `docker compose -f docker-compose.dev.yml up --build` does not require an SSH key unless you configure remote SSH machines in the Settings tab.

@@ -142,10 +142,12 @@ For local development, `docker compose -f docker-compose.dev.yml up --build` doe
 - `GET /api/dashboard/libraries` — Per-library breakdown
 - `GET /api/dashboard/now-playing` — Active playback sessions
 - `GET /api/monitoring/machines` — Persistent monitoring machine definitions
- `GET /api/monitoring/prometheus-targets` — Prometheus file-SD targets for remote Node Exporters
- `GET /api/monitoring/alerts` — Active Alertmanager alerts summary
+- `GET /api/monitoring/prometheus-targets` — Prometheus scrape targets for remote Node Exporters (consumed by external Prometheus via `http_sd_configs`)
+- `GET /api/monitoring/alerts` — Active Alertmanager alerts summary (resolves the configured alertmanager service)
 - `GET /api/monitoring/alertmanager-status` — Alertmanager cluster/status
- `POST /api/monitoring/alertmanager-webhook` — Receive/forward Alertmanager webhooks
+- `GET /api/monitoring/grafana-status` — Grafana service health
+- `GET /api/monitoring/prometheus-status` — Prometheus service health
+- `POST /api/monitoring/alertmanager-webhook` — Receive Alertmanager webhooks (log-only)
 - `GET /api/settings/machines` — Manage machine definitions
 - `GET /api/media/status` — Index status
 - `POST /api/media/build` — Rebuild index
@@ -54,8 +54,6 @@ class Settings(BaseSettings):

    # Observability
    prometheus_enabled: bool = True
-    alertmanager_url: str = ""
-    alertmanager_webhook_url: str = ""  # Optional receiver for alertmanager webhook notifications

    # Remote paths
    remote_media_root: str = ""
@@ -23,8 +23,6 @@ services:
      LOG_FORMAT: ${LOG_FORMAT:-text}
      PROMETHEUS_ENABLED: "true"
      SSH_KNOWN_HOSTS_PATH: /app/backend/.cache/known_hosts
-      ALERTMANAGER_URL: ${ALERTMANAGER_URL:-}
-      ALERTMANAGER_WEBHOOK_URL: ${ALERTMANAGER_WEBHOOK_URL:-}
      MANAGE_ENCRYPTION_KEY: ${MANAGE_ENCRYPTION_KEY:?set MANAGE_ENCRYPTION_KEY in your .env}
    ports:
      - "8000:8000"
@@ -45,8 +43,6 @@ services:
      VITE_API_URL: "/api"
      VITE_OIDC_ENABLED: "false"
      VITE_DEV_API_PROXY_TARGET: "http://backend:8000"
-      VITE_GRAFANA_URL: ${VITE_GRAFANA_URL:-http://localhost:3000}
-      VITE_PROMETHEUS_URL: ${VITE_PROMETHEUS_URL:-http://localhost:9090}
    ports:
      - "5173:5173"
    volumes:
@@ -34,8 +34,6 @@ services:
      SMTP_USE_SSL: ${SMTP_USE_SSL:-false}
      SMTP_TIMEOUT: ${SMTP_TIMEOUT:-30}
      SSH_KNOWN_HOSTS_PATH: /app/backend/.cache/known_hosts
-      ALERTMANAGER_URL: ${ALERTMANAGER_URL:-}
-      ALERTMANAGER_WEBHOOK_URL: ${ALERTMANAGER_WEBHOOK_URL:-}
      MANAGE_ENCRYPTION_KEY: ${MANAGE_ENCRYPTION_KEY:?generate one with python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"}
    volumes:
      - ${BACKEND_CACHE_DIR:-./backend-cache}:/app/backend/.cache
@@ -77,8 +75,6 @@ services:
        VITE_OIDC_REDIRECT_URI: ${VITE_OIDC_REDIRECT_URI:?set VITE_OIDC_REDIRECT_URI}
        VITE_OIDC_POST_LOGOUT_REDIRECT_URI: ${VITE_OIDC_POST_LOGOUT_REDIRECT_URI:?set VITE_OIDC_POST_LOGOUT_REDIRECT_URI}
        VITE_DEV_API_PROXY_TARGET: ${VITE_DEV_API_PROXY_TARGET:-http://backend:8000}
-        VITE_GRAFANA_URL: ${VITE_GRAFANA_URL:-https://grafana.example.com}
-        VITE_PROMETHEUS_URL: ${VITE_PROMETHEUS_URL:-http://prometheus.example.com}
        VITE_APP_VERSION: ${APP_VERSION:-0.1.0}
        VITE_APP_BUILD_INFO: ${APP_BUILD_INFO:-dev}
    depends_on:
@@ -331,6 +331,7 @@ the widget/addon-pages model were removed. `MANAGE_ENCRYPTION_KEY` is now requir

 ## Decision Log

+- 2026-06-23: Made all observability integration service-registry first-class with zero service env config. Alertmanager is now a 6th service type (`integrations/alertmanager.py`) with an `active_alerts` widget, mirroring Grafana/Prometheus. The monitoring router resolves alertmanager/grafana/prometheus from service records (first-enabled-instance default, optional `?service_id=`); added `/api/monitoring/grafana-status` and `/prometheus-status` health probes. The Observability page discovers services from the registry (dropped `import.meta.env.VITE_GRAFANA_URL`) and renders health cards; the dashboard gained an `active_alerts` widget. Removed the backend `alertmanager_url`/`alertmanager_webhook_url` fields and the frontend `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` build args — observability services are configured in the UI only. Removed the `PROMETHEUS_FILE_SD_DIR` shared-file bridge + `write_prometheus_targets` file writer; external Prometheus now uses `http_sd_configs` against `GET /api/monitoring/prometheus-targets` (`build_node_exporter_targets` + that endpoint stay). The webhook receiver is log-only (dropped the outbound forward). The only observability env var remaining is `PROMETHEUS_ENABLED` (Manage's own `/metrics` toggle). See `openspec/changes/observability-service-registry/`.
 - 2026-06-23: Removed all observability services from the root Compose files. `docker-compose.yml` and `docker-compose.dev.yml` now deploy **only** the backend and frontend; the `monitoring` network and the `prometheus`/`loki`/`alloy`/`grafana`/`alertmanager`/`node-exporter` services and their named volumes were deleted, and the `GRAFANA_APP_HOST` Traefik rule was removed. Manage connects to **existing** Grafana/Prometheus/Alertmanager instances and does not deploy them itself. The backend `alertmanager_url` default is now empty, and the `/api/monitoring/alerts` and `/alertmanager-status` endpoints return graceful "not configured" responses when `ALERTMANAGER_URL` is unset. The previous in-compose stack is preserved as an optional standalone example in `docker-compose.observability.yml` (config under `monitoring/`, documented in `docs/observability-runbooks.md`). `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` remain as optional frontend deep-link overrides.
 - 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template.
 - 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring` → `/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`.
@@ -15,8 +15,6 @@ ARG VITE_OIDC_SCOPE=openid profile email
 ARG VITE_OIDC_REDIRECT_URI=
 ARG VITE_OIDC_POST_LOGOUT_REDIRECT_URI=
 ARG VITE_DEV_API_PROXY_TARGET=http://backend:8000
-ARG VITE_GRAFANA_URL=https://grafana.example.com
-ARG VITE_PROMETHEUS_URL=http://localhost:9090
 ARG VITE_APP_VERSION=0.1.0
 ARG VITE_APP_BUILD_INFO=dev

@@ -28,8 +26,6 @@ ENV VITE_API_URL=${VITE_API_URL} \
    VITE_OIDC_REDIRECT_URI=${VITE_OIDC_REDIRECT_URI} \
    VITE_OIDC_POST_LOGOUT_REDIRECT_URI=${VITE_OIDC_POST_LOGOUT_REDIRECT_URI} \
    VITE_DEV_API_PROXY_TARGET=${VITE_DEV_API_PROXY_TARGET} \
-    VITE_GRAFANA_URL=${VITE_GRAFANA_URL} \
-    VITE_PROMETHEUS_URL=${VITE_PROMETHEUS_URL} \
    VITE_APP_VERSION=${VITE_APP_VERSION} \
    VITE_APP_BUILD_INFO=${VITE_APP_BUILD_INFO}

@@ -54,8 +50,6 @@ COPY frontend/ ./
 ENV VITE_API_URL=/api \
    VITE_OIDC_ENABLED=false \
    VITE_DEV_API_PROXY_TARGET=http://backend:8000 \
-    VITE_GRAFANA_URL=http://localhost:3000 \
-    VITE_PROMETHEUS_URL=http://localhost:9090 \
    VITE_APP_VERSION=0.1.0 \
    VITE_APP_BUILD_INFO=dev