manage/context.md

Code Context

Status (2026-06-23): Manage no longer deploys an observability stack, and observability is no longer env-configured. The root docker-compose.yml / docker-compose.dev.yml ship only the backend and frontend; Grafana, Prometheus, Loki, Alertmanager, Alloy, and Node Exporter were removed from them. Manage connects to existing instances, and Alertmanager/Grafana/ Prometheus are now service-registry instances configured in the UI (no ALERTMANAGER_URL, VITE_GRAFANA_URL, VITE_PROMETHEUS_URL, or PROMETHEUS_FILE_SD_DIR env vars). The file-SD writer was removed; external Prometheus uses http_sd_configs against GET /api/monitoring/prometheus-targets. The standalone example stack lives in docker-compose.observability.yml. The detailed snippets below describe the former in-compose, env-configured model and are kept as historical context only; treat the service registry (Services page UI + openspec/changes/observability-service-registry/) and docker-compose.observability.yml as authoritative.

Files Retrieved

1. docker-compose.yml (lines 1–262) – production Compose stack; defines observability services and Traefik routing.
2. docker-compose.dev.yml (lines 1–234) – development Compose stack; same observability services but with host ports exposed and auth disabled.
3. .env.example (lines 1–55) – template with all required environment variables for the stack, including Prometheus/Grafana/Alertmanager/Alloy/Loki and Node Exporter settings.
4. monitoring/prometheus/prometheus.yml (all lines) – Prometheus scrape configuration including the manage-backend, node-exporter, and file-SD remote targets.
5. monitoring/prometheus/rules/backup_alerts.yml (all lines) – Prometheus alerting rules for backup jobs and observability-stack health.
6. monitoring/alertmanager/alertmanager.yml (all lines) – Alertmanager routing, email/webhook receivers, and inhibition rules.
7. monitoring/grafana/grafana.ini (all lines) – Grafana server, embedding, and generic OAuth (Authentik) configuration.
8. monitoring/alloy/config.alloy (all lines) – Alloy pipeline to discover Docker containers and push logs to Loki.
9. monitoring/loki/loki.yml (all lines) – Single-node Loki configuration with filesystem storage and 30-day retention.
10. backend/src/media_library_viewer_api/observability.py (all lines) – Prometheus metrics definitions and helper functions.
11. backend/src/media_library_viewer_api/main.py (lines 1–128) – FastAPI entrypoint exposing /metrics and wiring request/observability middleware.
12. backend/src/media_library_viewer_api/services/targets.py (all lines) – Backend writes file-SD target list for remote Node Exporters.
13. backend/src/media_library_viewer_api/config.py (lines 1–88) – Settings including prometheus_enabled, prometheus_file_sd_dir, and alertmanager_url.
14. docs/monitoring-logging-design.md (all lines) – Architecture/design document describing the observability stack.
15. docs/observability-runbooks.md (all lines) – Operational runbooks for the observability services.

Key Code

Backend /metrics endpoint

backend/src/media_library_viewer_api/main.py:

@app.middleware("http")
async def enforce_jwt_auth(request: Request, call_next):
    if request.url.path in {"/api/health", "/api/version", "/metrics"}:
        return await call_next(request)
    return await require_jwt_auth(request, call_next)

@app.get("/metrics")
def metrics() -> Response:
    """Expose Prometheus metrics."""
    data, content_type = metrics_payload()
    return FastAPIResponse(content=data, media_type=content_type)

Metrics emitted by the backend

backend/src/media_library_viewer_api/observability.py:

REQUESTS_TOTAL = Counter("manage_api_requests_total", "Total API requests", ["method", "path", "status_code"])
REQUEST_DURATION = Histogram("manage_api_request_duration_seconds", "API request duration", ["method", "path"], ...)
SSH_COMMANDS_TOTAL = Counter("manage_ssh_commands_total", "Total SSH/local commands executed", ["machine_id", "action", "status"])
MEDIA_INDEX_BUILDS_TOTAL = Counter("manage_media_index_builds_total", "Total media index build attempts", ["status"])
BACKUP_RUNS_TOTAL = Counter("manage_backup_runs_total", "Total backup runs", ["job_name", "status"])
BACKUP_RUNS_LAST_SUCCESS = Gauge("manage_backup_runs_last_success_timestamp", "Unix timestamp of the last successful backup run per job", ["job_name"])
MAIL_QUEUE_SIZE = Counter("manage_mail_queue_messages_total", "Total messages enqueued", ["status"])

Prometheus scrape configuration

monitoring/prometheus/prometheus.yml:

scrape_configs:
  - job_name: manage-backend
    static_configs:
      - targets:
          - backend:8000
    metrics_path: /metrics
    scrape_interval: 15s

  - job_name: node-exporter
    static_configs:
      - targets:
          - node-exporter:9100

  - job_name: node-exporter-remote
    file_sd_configs:
      - files:
          - /etc/prometheus/file-sd/node_exporter_targets.json
        refresh_interval: 30s

Backend-managed remote Node Exporter targets

backend/src/media_library_viewer_api/services/targets.py:

def build_node_exporter_targets(store: SettingsStore) -> list[dict[str, Any]]:
    ...

def write_prometheus_targets(store: SettingsStore, file_sd_dir: Path | None = None) -> Path:
    ...
    file_path = file_sd_dir / "node_exporter_targets.json"

Architecture

The observability stack is a standard self-hosted Prometheus/Grafana/Loki/Alertmanager deployment running inside the same Docker Compose project as the Manage application:

• Prometheus pulls metrics from:
  • The Manage FastAPI backend via /metrics (job="manage-backend").
  • The local Docker host via node-exporter (job="node-exporter").
  • Remote machines that have Node Exporter enabled through Manage settings (job="node-exporter-remote"), discovered through a file-SD JSON file generated by the backend.
  • The observability services themselves: Loki, Alertmanager, Grafana, and Prometheus self-scrape.
• Grafana visualizes metrics and logs; it is configured for OAuth login through Authentik and is embedded in the Manage React UI via iframes.
• Loki stores logs; retention is 30 days.
• Alloy (Grafana Alloy) collects Docker container logs by mounting the Docker socket and the container log directory, then pushes them to Loki.
• Alertmanager routes alerts by severity (critical vs warning) and delivers email notifications (and optionally a webhook back to the backend).

The backend bridges the stack with the application:

• It exposes /metrics (unauthenticated, along with /api/health and /api/version).
• On startup it writes ${PROMETHEUS_FILE_SD_DIR}/node_exporter_targets.json based on enabled SSH machines in the settings store.
• It provides proxy endpoints (/api/monitoring/alerts, /api/monitoring/alertmanager-status, /api/monitoring/prometheus-targets) consumed by the frontend.

Start Here

Open monitoring/prometheus/prometheus.yml first to understand what is scraped and how the backend is wired, then read backend/src/media_library_viewer_api/observability.py to see the metric names and labels. For environment requirements, read .env.example.

Supervisor coordination

Not needed — this is a read-only scouting summary.

---

Monitoring/Observability Setup Summary

1. Observability services defined in Compose

Both docker-compose.yml and docker-compose.dev.yml define only the backend and frontend. The observability services (Prometheus, Loki, Grafana, Alertmanager, Alloy, Node Exporter) were extracted to the standalone docker-compose.observability.yml example stack and are no longer deployed by Manage. Summary of what remains in the app compose files:

Service	Image	Internal endpoint	Purpose
prometheus	prom/prometheus:v2.55.1	http://prometheus:9090	Metrics TSDB and alert evaluator
loki	grafana/loki:3.1.1	http://loki:3100	Log aggregation
alloy	grafana/alloy:v1.5.0	http://alloy:12345	Docker log collection agent
grafana	grafana/grafana:11.3.1	http://grafana:3000	Dashboards and visualization
alertmanager	prom/alertmanager:v0.27.0	http://alertmanager:9093	Alert routing/delivery
node-exporter	prom/node-exporter:v1.8.2	http://node-exporter:9100	Host metrics for the Docker host
backend	Build from backend/Dockerfile	http://backend:8000	FastAPI app exposing /metrics

Differences:

• Production (docker-compose.yml): services attach to an external web network for Traefik, use expose instead of host ports for most services, and require OIDC/auth variables.
• Development (docker-compose.dev.yml): Prometheus/Grafana/Loki/Alertmanager/Node Exporter are published on host ports 9090, 3000, 3100, 9093, 9100; auth is disabled (AUTH_ENABLED=false).

2. Required environment variables

From .env.example and the Compose files, the variables relevant to the observability stack are:

Backend / metrics

• PROMETHEUS_ENABLED – enable metrics endpoint (set to "true" in both compose files).
• PROMETHEUS_FILE_SD_DIR – directory where the backend writes node_exporter_targets.json (default /app/backend/.cache/prometheus-file-sd).
• ALERTMANAGER_URL – backend proxy target (default http://alertmanager:9093).
• ALERTMANAGER_WEBHOOK_URL – optional webhook receiver for Alertmanager.
• BACKEND_CACHE_DIR – host directory mounted into backend and Prometheus for file-SD.

Grafana

• GRAFANA_APP_HOST – public hostname for Grafana (production; required).
• GRAFANA_APP_PORT – defaults to 3000.
• GRAFANA_APP_NAME – defaults to grafana.
• GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD – local admin credentials.
• GF_AUTH_GENERIC_OAUTH_CLIENT_ID
• GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET
• GF_AUTH_GENERIC_OAUTH_AUTH_URL
• GF_AUTH_GENERIC_OAUTH_TOKEN_URL
• GF_AUTH_GENERIC_OAUTH_API_URL

Alertmanager

• SMTP_HOST / SMTP_PORT
• SMTP_USERNAME / SMTP_PASSWORD
• SMTP_FROM_ADDRESS
• ALERT_EMAIL_TO

Traefik / network (production)

• BACKEND_APP_HOST / FRONTEND_APP_HOST / GRAFANA_APP_HOST
• CERT_RESOLVER – e.g. letsencrypt

General

• LOG_LEVEL / LOG_FORMAT – also passed to Grafana and backend.

3. Monitoring config files

File	Purpose
monitoring/prometheus/prometheus.yml	Scrape jobs: backend /metrics, local node-exporter, remote node-exporter via file-SD, Loki, Alertmanager, Grafana, and self-scrape.
monitoring/prometheus/rules/backup_alerts.yml	Alert rules: BackupJobFailed, BackupJobStuck, PrometheusTargetMissing, AlertmanagerDown, GrafanaDown.
monitoring/alertmanager/alertmanager.yml	Routes alerts by severity, sends email to ALERT_EMAIL_TO, optional webhook to backend, and inhibits warnings when a critical alert fires.
monitoring/grafana/grafana.ini	Enables iframe embedding, OAuth via Authentik, role mapping from groups, and defaults users to Viewer.
monitoring/alloy/config.alloy	Discovers Docker containers, relabels container/stream labels, and writes logs to http://loki:3100/loki/api/v1/push.
monitoring/loki/loki.yml	Single-node Loki with filesystem storage, tsdb index, 30-day retention (720h).

4. Backend metrics and Prometheus scraping

• The backend exposes Prometheus metrics at /metrics on port 8000.
• The endpoint is unauthenticated (bypassed in enforce_jwt_auth).
• Prometheus scrapes it as job="manage-backend" with scrape_interval: 15s.
• Key application metrics include:
  • manage_api_requests_total{method, path, status_code}
  • manage_api_request_duration_seconds{method, path}
  • manage_ssh_commands_total{machine_id, action, status}
  • manage_media_index_builds_total{status}
  • manage_backup_runs_total{job_name, status}
  • manage_backup_runs_last_success_timestamp{job_name}
  • manage_mail_queue_messages_total{status}

Remote Node Exporter targets are not static: the backend reads machine settings from SQLite and writes ${PROMETHEUS_FILE_SD_DIR}/node_exporter_targets.json. Prometheus reloads this file every 30 seconds via file_sd_configs.

5. Setup steps and gotchas

• Manage's own compose stack does not include observability services. To run a full local stack, bring up the app and the standalone observability example separately:
  • Production: docker compose -f docker-compose.yml up --build
  • Development: docker compose -f docker-compose.dev.yml up --build
• Production requires the external web network and Traefik already configured; docker-compose.dev.yml does not use Traefik and binds ports directly.
• Export/copy .env.example to .env and fill required values (OIDC_ISSUER_URL, OIDC_AUDIENCE, BACKEND_APP_HOST, FRONTEND_APP_HOST, GRAFANA_APP_HOST, CERT_RESOLVER, SMTP credentials, Grafana OAuth secrets).
• BACKEND_CACHE_DIR is shared between the backend and Prometheus so file-SD target updates are visible to Prometheus.
• Alloy must run as user: root and mount /var/run/docker.sock and /var/lib/docker/containers; without these mounts it cannot collect Docker logs.
• Node Exporter mounts the host root filesystem read-only (/:/host:ro,rslave) to report host-level metrics; on production, this exposes the Docker host.
• Grafana iframe embedding requires allow_embedding = true in grafana.ini and matching cookie settings; also ensure the reverse proxy/CSP permits embedding.
• After changing Prometheus rules/config, trigger a reload with curl -X POST http://localhost:9090/-/reload (production needs Traefik/network access).
• Observability data is stored in named volumes: prometheus_data, loki_data, grafana_data, alertmanager_data. Back them up as documented in docs/observability-runbooks.md.
• Default retention is 30 days for both Prometheus TSDB and Loki logs.