manage/context.md

# Code Context

## Files Retrieved
1. `docker-compose.yml` (lines 1–262) – production Compose stack; defines observability services and Traefik routing.
2. `docker-compose.dev.yml` (lines 1–234) – development Compose stack; same observability services but with host ports exposed and auth disabled.
3. `.env.example` (lines 1–55) – template with all required environment variables for the stack, including Prometheus/Grafana/Alertmanager/Alloy/Loki and Node Exporter settings.
4. `monitoring/prometheus/prometheus.yml` (all lines) – Prometheus scrape configuration including the `manage-backend`, `node-exporter`, and file-SD remote targets.
5. `monitoring/prometheus/rules/backup_alerts.yml` (all lines) – Prometheus alerting rules for backup jobs and observability-stack health.
6. `monitoring/alertmanager/alertmanager.yml` (all lines) – Alertmanager routing, email/webhook receivers, and inhibition rules.
7. `monitoring/grafana/grafana.ini` (all lines) – Grafana server, embedding, and generic OAuth (Authentik) configuration.
8. `monitoring/alloy/config.alloy` (all lines) – Alloy pipeline to discover Docker containers and push logs to Loki.
9. `monitoring/loki/loki.yml` (all lines) – Single-node Loki configuration with filesystem storage and 30-day retention.
10. `backend/src/media_library_viewer_api/observability.py` (all lines) – Prometheus metrics definitions and helper functions.
11. `backend/src/media_library_viewer_api/main.py` (lines 1–128) – FastAPI entrypoint exposing `/metrics` and wiring request/observability middleware.
12. `backend/src/media_library_viewer_api/services/targets.py` (all lines) – Backend writes file-SD target list for remote Node Exporters.
13. `backend/src/media_library_viewer_api/config.py` (lines 1–88) – Settings including `prometheus_enabled`, `prometheus_file_sd_dir`, and `alertmanager_url`.
14. `docs/monitoring-logging-design.md` (all lines) – Architecture/design document describing the observability stack.
15. `docs/observability-runbooks.md` (all lines) – Operational runbooks for the observability services.

## Key Code

### Backend `/metrics` endpoint
`backend/src/media_library_viewer_api/main.py`:
```python
@app.middleware("http")
async def enforce_jwt_auth(request: Request, call_next):
    if request.url.path in {"/api/health", "/api/version", "/metrics"}:
        return await call_next(request)
    return await require_jwt_auth(request, call_next)

@app.get("/metrics")
def metrics() -> Response:
    """Expose Prometheus metrics."""
    data, content_type = metrics_payload()
    return FastAPIResponse(content=data, media_type=content_type)
```

### Metrics emitted by the backend
`backend/src/media_library_viewer_api/observability.py`:
```python
REQUESTS_TOTAL = Counter("manage_api_requests_total", "Total API requests", ["method", "path", "status_code"])
REQUEST_DURATION = Histogram("manage_api_request_duration_seconds", "API request duration", ["method", "path"], ...)
SSH_COMMANDS_TOTAL = Counter("manage_ssh_commands_total", "Total SSH/local commands executed", ["machine_id", "action", "status"])
MEDIA_INDEX_BUILDS_TOTAL = Counter("manage_media_index_builds_total", "Total media index build attempts", ["status"])
BACKUP_RUNS_TOTAL = Counter("manage_backup_runs_total", "Total backup runs", ["job_name", "status"])
BACKUP_RUNS_LAST_SUCCESS = Gauge("manage_backup_runs_last_success_timestamp", "Unix timestamp of the last successful backup run per job", ["job_name"])
MAIL_QUEUE_SIZE = Counter("manage_mail_queue_messages_total", "Total messages enqueued", ["status"])
```

### Prometheus scrape configuration
`monitoring/prometheus/prometheus.yml`:
```yaml
scrape_configs:
  - job_name: manage-backend
    static_configs:
      - targets:
          - backend:8000
    metrics_path: /metrics
    scrape_interval: 15s

  - job_name: node-exporter
    static_configs:
      - targets:
          - node-exporter:9100

  - job_name: node-exporter-remote
    file_sd_configs:
      - files:
          - /etc/prometheus/file-sd/node_exporter_targets.json
        refresh_interval: 30s
```

### Backend-managed remote Node Exporter targets
`backend/src/media_library_viewer_api/services/targets.py`:
```python
def build_node_exporter_targets(store: SettingsStore) -> list[dict[str, Any]]:
    ...

def write_prometheus_targets(store: SettingsStore, file_sd_dir: Path | None = None) -> Path:
    ...
    file_path = file_sd_dir / "node_exporter_targets.json"
```

## Architecture

The observability stack is a standard self-hosted Prometheus/Grafana/Loki/Alertmanager deployment running inside the same Docker Compose project as the Manage application:

- **Prometheus** pulls metrics from:
  - The Manage FastAPI backend via `/metrics` (`job="manage-backend"`).
  - The local Docker host via `node-exporter` (`job="node-exporter"`).
  - Remote machines that have Node Exporter enabled through Manage settings (`job="node-exporter-remote"`), discovered through a file-SD JSON file generated by the backend.
  - The observability services themselves: Loki, Alertmanager, Grafana, and Prometheus self-scrape.
- **Grafana** visualizes metrics and logs; it is configured for OAuth login through Authentik and is embedded in the Manage React UI via iframes.
- **Loki** stores logs; retention is 30 days.
- **Alloy** (Grafana Alloy) collects Docker container logs by mounting the Docker socket and the container log directory, then pushes them to Loki.
- **Alertmanager** routes alerts by severity (critical vs warning) and delivers email notifications (and optionally a webhook back to the backend).

The backend bridges the stack with the application:
- It exposes `/metrics` (unauthenticated, along with `/api/health` and `/api/version`).
- On startup it writes `${PROMETHEUS_FILE_SD_DIR}/node_exporter_targets.json` based on enabled SSH machines in the settings store.
- It provides proxy endpoints (`/api/monitoring/alerts`, `/api/monitoring/alertmanager-status`, `/api/monitoring/prometheus-targets`) consumed by the frontend.

## Start Here

Open `monitoring/prometheus/prometheus.yml` first to understand what is scraped and how the backend is wired, then read `backend/src/media_library_viewer_api/observability.py` to see the metric names and labels. For environment requirements, read `.env.example`.

## Supervisor coordination
Not needed — this is a read-only scouting summary.

---

# Monitoring/Observability Setup Summary

## 1. Observability services defined in Compose

Both `docker-compose.yml` and `docker-compose.dev.yml` define the following services:

| Service | Image | Internal endpoint | Purpose |
|---------|-------|-------------------|---------|
| `prometheus` | `prom/prometheus:v2.55.1` | `http://prometheus:9090` | Metrics TSDB and alert evaluator |
| `loki` | `grafana/loki:3.1.1` | `http://loki:3100` | Log aggregation |
| `alloy` | `grafana/alloy:v1.5.0` | `http://alloy:12345` | Docker log collection agent |
| `grafana` | `grafana/grafana:11.3.1` | `http://grafana:3000` | Dashboards and visualization |
| `alertmanager` | `prom/alertmanager:v0.27.0` | `http://alertmanager:9093` | Alert routing/delivery |
| `node-exporter` | `prom/node-exporter:v1.8.2` | `http://node-exporter:9100` | Host metrics for the Docker host |
| `backend` | Build from `backend/Dockerfile` | `http://backend:8000` | FastAPI app exposing `/metrics` |

Differences:
- Production (`docker-compose.yml`): services attach to an external `web` network for Traefik, use `expose` instead of host ports for most services, and require OIDC/auth variables.
- Development (`docker-compose.dev.yml`): Prometheus/Grafana/Loki/Alertmanager/Node Exporter are published on host ports `9090`, `3000`, `3100`, `9093`, `9100`; auth is disabled (`AUTH_ENABLED=false`).

## 2. Required environment variables

From `.env.example` and the Compose files, the variables relevant to the observability stack are:

### Backend / metrics
- `PROMETHEUS_ENABLED` – enable metrics endpoint (set to `"true"` in both compose files).
- `PROMETHEUS_FILE_SD_DIR` – directory where the backend writes `node_exporter_targets.json` (default `/app/backend/.cache/prometheus-file-sd`).
- `ALERTMANAGER_URL` – backend proxy target (default `http://alertmanager:9093`).
- `ALERTMANAGER_WEBHOOK_URL` – optional webhook receiver for Alertmanager.
- `BACKEND_CACHE_DIR` – host directory mounted into backend and Prometheus for file-SD.

### Grafana
- `GRAFANA_APP_HOST` – public hostname for Grafana (production; required).
- `GRAFANA_APP_PORT` – defaults to `3000`.
- `GRAFANA_APP_NAME` – defaults to `grafana`.
- `GRAFANA_ADMIN_USER` / `GRAFANA_ADMIN_PASSWORD` – local admin credentials.
- `GF_AUTH_GENERIC_OAUTH_CLIENT_ID`
- `GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET`
- `GF_AUTH_GENERIC_OAUTH_AUTH_URL`
- `GF_AUTH_GENERIC_OAUTH_TOKEN_URL`
- `GF_AUTH_GENERIC_OAUTH_API_URL`

### Alertmanager
- `SMTP_HOST` / `SMTP_PORT`
- `SMTP_USERNAME` / `SMTP_PASSWORD`
- `SMTP_FROM_ADDRESS`
- `ALERT_EMAIL_TO`

### Traefik / network (production)
- `BACKEND_APP_HOST` / `FRONTEND_APP_HOST` / `GRAFANA_APP_HOST`
- `CERT_RESOLVER` – e.g. `letsencrypt`

### General
- `LOG_LEVEL` / `LOG_FORMAT` – also passed to Grafana and backend.

## 3. Monitoring config files

| File | Purpose |
|------|---------|
| `monitoring/prometheus/prometheus.yml` | Scrape jobs: backend `/metrics`, local node-exporter, remote node-exporter via file-SD, Loki, Alertmanager, Grafana, and self-scrape. |
| `monitoring/prometheus/rules/backup_alerts.yml` | Alert rules: `BackupJobFailed`, `BackupJobStuck`, `PrometheusTargetMissing`, `AlertmanagerDown`, `GrafanaDown`. |
| `monitoring/alertmanager/alertmanager.yml` | Routes alerts by severity, sends email to `ALERT_EMAIL_TO`, optional webhook to backend, and inhibits warnings when a critical alert fires. |
| `monitoring/grafana/grafana.ini` | Enables iframe embedding, OAuth via Authentik, role mapping from groups, and defaults users to `Viewer`. |
| `monitoring/alloy/config.alloy` | Discovers Docker containers, relabels container/stream labels, and writes logs to `http://loki:3100/loki/api/v1/push`. |
| `monitoring/loki/loki.yml` | Single-node Loki with filesystem storage, tsdb index, 30-day retention (`720h`). |

## 4. Backend metrics and Prometheus scraping

- The backend exposes Prometheus metrics at `/metrics` on port `8000`.
- The endpoint is unauthenticated (bypassed in `enforce_jwt_auth`).
- Prometheus scrapes it as `job="manage-backend"` with `scrape_interval: 15s`.
- Key application metrics include:
  - `manage_api_requests_total{method, path, status_code}`
  - `manage_api_request_duration_seconds{method, path}`
  - `manage_ssh_commands_total{machine_id, action, status}`
  - `manage_media_index_builds_total{status}`
  - `manage_backup_runs_total{job_name, status}`
  - `manage_backup_runs_last_success_timestamp{job_name}`
  - `manage_mail_queue_messages_total{status}`

Remote Node Exporter targets are not static: the backend reads machine settings from SQLite and writes `${PROMETHEUS_FILE_SD_DIR}/node_exporter_targets.json`. Prometheus reloads this file every 30 seconds via `file_sd_configs`.

## 5. Setup steps and gotchas

- The observability stack is brought up with the app itself:
  - Production: `docker compose -f docker-compose.yml up --build`
  - Development: `docker compose -f docker-compose.dev.yml up --build`
- Production requires the external `web` network and Traefik already configured; `docker-compose.dev.yml` does not use Traefik and binds ports directly.
- Export/copy `.env.example` to `.env` and fill required values (`OIDC_ISSUER_URL`, `OIDC_AUDIENCE`, `BACKEND_APP_HOST`, `FRONTEND_APP_HOST`, `GRAFANA_APP_HOST`, `CERT_RESOLVER`, SMTP credentials, Grafana OAuth secrets).
- `BACKEND_CACHE_DIR` is shared between the backend and Prometheus so file-SD target updates are visible to Prometheus.
- Alloy must run as `user: root` and mount `/var/run/docker.sock` and `/var/lib/docker/containers`; without these mounts it cannot collect Docker logs.
- Node Exporter mounts the host root filesystem read-only (`/:/host:ro,rslave`) to report host-level metrics; on production, this exposes the Docker host.
- Grafana iframe embedding requires `allow_embedding = true` in `grafana.ini` and matching cookie settings; also ensure the reverse proxy/CSP permits embedding.
- After changing Prometheus rules/config, trigger a reload with `curl -X POST http://localhost:9090/-/reload` (production needs Traefik/network access).
- Observability data is stored in named volumes: `prometheus_data`, `loki_data`, `grafana_data`, `alertmanager_data`. Back them up as documented in `docs/observability-runbooks.md`.
- Default retention is 30 days for both Prometheus TSDB and Loki logs.