manage/docs/REQUIREMENTS.md

# Manage - Requirements and Decision Log

This is a living document for the project. Update it whenever requirements, UX expectations, architecture, constraints, or implementation plans change.

## Product Goal

Build Manage, a compact web application for browsing a remote Jellyfin media library and inspecting the corresponding media files on disk over SSH. The app should support library metadata review, direct file-system navigation, detailed media metadata inspection, and safe remote maintenance/job workflows.

## Current Phase

Phase 1: Jellyfin media index, SSH-based remote filesystem inspection, server monitoring, and safe job templates.

## Frontend Design System & Architecture

The Manage frontend is a React + TypeScript SPA built on a **single design system**.
The legacy Material UI (MUI v9) / Emotion / recharts / D3 / `theme.ts` stack has been
fully removed (web-ui-rework; see decision log 2026-06-17).

### Design system

- **shadcn/ui** components + **Tailwind CSS v4** + **lucide-react** icons are the only UI layer.
- Design tokens live as CSS `@theme` tokens in `frontend/src/index.css` (light + `.dark`),
  with the primary brand color `#4f8cff`.
- The `chart-1`..`chart-5` color tokens are **repurposed as status / Grafana-link color
  cues** (not charts): `chart-1`=info/brand, `chart-2`=success/healthy, `chart-3`=warning,
  `chart-4`=destructive, `chart-5`=neutral accent. No token value changed.
- Removed from the frontend dependency tree: `@mui/material`, `@mui/icons-material`,
  `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the
  no-op `src/theme.ts` shim.

### Thin-dashboard observability model

- The app does **no in-app charting**. Metrics, charts, and logs live in the external,
  decoupled observability stack (Prometheus / Loki / Grafana / Alertmanager).
- In-app observability surfaces (`/observability`) show **Alertmanager alerts, Prometheus
  target health, machine health, and Grafana deep-links** (per-machine metric/log panels),
  not rendered graphs.
- The legacy in-app D3 monitoring charts and the POSIX remote resource collector are
  superseded by this Grafana-based model (see decision log 2026-06-13 and 2026-06-17).
- **Manage no longer scrapes its own system metrics** (decision 2026-06-17). The backend
  `MonitoringPoller` (which SSH-ran `df` on every machine every 5 minutes into a local
  SQLite `monitoring_machine_actions` table), the `/api/monitoring/disk`, `/poller`, and
  `/machines/{id}/actions` endpoints, and the frontend `DiskSpaceCard` have been removed.
  Disk/CPU/memory visibility is owned by Prometheus + node_exporter + Grafana. The
  `disk_usage` **job template** in Actions remains as a manual on-demand SSH check.

### Tables

- Tabular surfaces use **TanStack Table** (`@tanstack/react-table`) behind a `DataTable`
  wrapper (`components/ui/data-table.tsx`).
- Parity is **visibility-only**: pagination, row selection, row click, and column
  visibility are supported. There is **no client sorting and no column resizing**.
- Media uses **server-driven pagination** (`manualPagination` + `rowCount`); the File
  Browser renders the full listing without pagination.
- The Media and File Browser tables previously used `@mui/x-data-grid`; both now use the
  TanStack `DataTable` (earlier "AG Grid" / `@mui/x-data-grid` references are superseded).

### Reconciled information architecture

- **Backups** is a top-level navigation item at `/backups`.
- The media/applications surface is named **Media** and lives at `/media`; `/applications`
  redirects to `/media`, mirroring the existing `/monitoring` → `/observability` redirect.
- User deep-links (`/users?user=<id>`), dashboard shortcut deep-links, and the Media →
  File Browser row-click navigation are preserved under the reconciled routes.

### Frontend testing

- Component tests run on **Vitest + @testing-library/react** (`npm test`), with the
  `@testing-library/jest-dom` matchers.
- Legacy plain-Node suites (`frontend/tests/*.test.mjs`) run via
  `node --test tests/*.test.mjs` (npm script `test:node`).
- The build/lint gate is `npm run build` (`tsc -b` + `vite build`) + `npm run lint` (ESLint).

## Core Requirements

### Jellyfin Library

- Connect to a remote Jellyfin server using an API key.
- Use the Jellyfin server root URL, not the `/web` UI URL.
- Handle API-key auth correctly:
  - use `GET /Users` to list available users;
  - allow a manual `JELLYFIN_USER_ID` override;
  - do not rely on `/Users/Me` for API-key auth.
- List Jellyfin libraries for the selected user.
- Browse library items with search, pagination, media type filtering, and poster cards.
- Show item details including overview, genres, ratings where available, file path, media sources/streams, and raw Jellyfin JSON.
- Provide a Media tab with a paginated inventory table for large libraries.
- Media inventory should show title, series name, season, episode number, type, year, runtime, file size, bitrate, explicit HDR yes/no flag, video codec, resolution, date added, and path where available.
- Media inventory should use Jellyfin metadata only for now; full ffprobe enrichment for all media should be deferred to a cached/background scan to avoid expensive per-item SSH probing.
- Media inventory should support multi-library selection, type, search, full-index sort/order, HDR filter, page size, and page controls.
- Media inventory should use a local SQLite index so sorting/filtering by nested/derived fields such as size, bitrate, HDR, codec, resolution, series, season, and episode can apply to the whole indexed library instead of only one Jellyfin page.
- Media inventory table should be read-only; full-index sorting/filtering should be handled by the service/query layer rather than relying on frontend table sorting.
- Media inventory should use row-based table selection and automatically sync the File browser tab to the selected row's containing directory.
- Media row selection should update both Media selected-row state and File browser location without requiring an extra action button.
- File browser interaction should stay explicit and simple: read-only listing plus explicit Open/Select actions rather than another row-selection grid.
- Use valid Jellyfin `Fields` query values only, because invalid field names can cause `400 Bad Request` responses.

### Users & Communication

- Provide a Users tab that lists all available users the system knows about.
- Use Jellyfin as the base source of truth for the user list.
- Optionally enrich Jellyfin users with Jellyseerr data when Jellyseerr is configured and reachable.
- Be tolerant of Jellyseerr response-shape differences across versions; for example, some endpoints may return a wrapped `{ users: [...] }` payload instead of a raw list.
- Surface whatever contact/identity fields are available from the configured source(s), such as email, avatar/thumb, role/permissions, and notification/contact eligibility.
- Email should only render actual email addresses; usernames or other non-email identifiers should be suppressed instead of shown as email.
- Keep communication actions separate from listing/identity data so the UI can support future email/notification workflows without redesigning the user list.
- SMTP-backed user messages should be queued asynchronously and return immediately; delivery must not block the rest of the API request path.
- The Users tab should expose a live queue status indicator so users can see when the outbound email queue is idle, busy, stopped, or failing.
- The queue status indicator should clearly show the current queue item count.
- The Users tab should include a one-click SMTP test action that validates connectivity/authentication without sending a real message.
- The SMTP test action should visibly show when it is running.
- The SMTP test should surface the chosen protocol/port and, for Fastmail, try both 465/SSL and 587/STARTTLS so configuration mismatches are easier to diagnose.
- Rework the Users data into an internal merged state so user identity can be combined with related now-playing/session data.
- Clicking a now-playing row should navigate to the Users tab and open the matching user detail drawer, keeping the selection deep-linkable.
- Provide an explicit "Open in Users" action in now-playing rows in addition to row-click navigation.
- Provide a compact per-field source summary for the Users detail drawer so it is obvious which backend source supplied name, email, avatar, and access data.
- Jellyseerr user list pagination must use `take`/`skip`, not `page`/`pageSize`.
- The Users tab table should stay compact and readable: center the avatar and email cells, keep backend source diagnostics out of the table itself, and prefer a simpler hand-built row layout when a dense grid makes text positioning awkward.
- The Users table activity column should stay compact and show only a brief status badge for playing/paused/idle/no-session state instead of a multi-line activity summary.
- The Dashboard activity panel should reuse the same compact session-table styling as the Users activity details so the two views feel consistent.
- In the shared session activity table, the user column should come before state, title/type, and device because the user is the most relevant identifier.
- The shared session table should keep a compact overall status summary line above the rows that reports total sessions plus playing, paused, and idle counts.
- The shared session table should keep the session identifier under the user name in a caption instead of giving it a full column, to keep the table tighter.
- The Users tab may open a read-only detail drawer for a selected user, but any communication actions in that drawer should remain clearly disabled/placeholders until the workflow is implemented.
- The frontend shell should use a polished two-row header with branding on the left, user/logout controls on the right, and primary navigation in a dedicated tab row beneath.
- The frontend shell and primary pages should remain responsive and mobile-safe, with compact navigation, stacked controls on narrow screens, and reduced table column density where needed.
- Dense management surfaces such as Actions and Settings should prefer compact tabbed or split-pane layouts over long single-column forms when that improves scanning and editing speed.
- Tab rails and subtabs should feel like a polished admin console: compact, clearly selected, and visually consistent across the app.
- When a setting subsection is not applicable to the currently selected mode or service, the UI should hide those fields rather than leaving them visible but disabled.
- The frontend should hydrate the API bearer token from persisted OIDC user storage immediately on reload so early requests do not race the auth provider lifecycle.
- The Media tab should persist its search/filter/sort/pagination state across reloads and tab switches.
- The File Browser should persist its current directory and selected file across reloads and tab switches.
- Backend startup should log a secret-safe configuration summary and request/activity diagnostics so configuration issues can be debugged without exposing API keys.

### Remote Filesystem over SSH

- Connect to a remote media server via SSH.
- Use strict SSH host key behavior, but synthesize and persist the managed `known_hosts` file from configured SSH machines instead of requiring users to mount their own `known_hosts` file.
- Browse remote directories and files rooted at a configurable default media path.
- File browser handoff should map Jellyfin paths to `REMOTE_MEDIA_ROOT` when possible (for example `/media/...` -> `/srv/media/...` when root is `/srv/media`).
- Media index paths should be stored in the SSH-visible form by default, using the same Jellyfin-to-SSH mapping so the Media tab and file browser agree on paths.
- Support a configurable Jellyfin-to-SSH fallback path prefix for cases where `REMOTE_MEDIA_ROOT` mapping alone is not sufficient.
- Support manual path entry and refresh.
- Remote file listing must be compact, structured, and navigable.
- The file table should be read-only.
- The file table should use row selection (single-select) in a TanStack `DataTable` format consistent with the Media tab (both migrated off the legacy `@mui/x-data-grid`/AG Grid).
- The file table should not expose a visible checkbox selection column.
- The file table should not show a visible `selected` column.
- Include a top `[UP] ..` row, when not at `/`, to navigate to the parent directory.
- The selected path should be visibly shown outside the table.
- Selecting a directory row (including `[UP] ..`) should open it immediately.
- Selecting a file row should set the selected file target for metadata/jobs.
- Avoid emojis and hard-to-render characters in labels.

### Remote Listing Controls

- Show compact listing summary information:
  - total entries;
  - directories;
  - files;
  - total file size for files in the current directory.
- Support filtering by:
  - all entries;
  - directories;
  - files;
  - file extension.
- Support case-insensitive filename search.
- Support sorting by:
  - name;
  - kind/type;
  - size;
  - modified time.
- Support ascending/descending sort order.
- Support pagination and configurable rows per page.

### Disk-Level Metadata

- Run `ffprobe` on selected remote files to inspect authoritative media metadata directly from disk.
- When a known video/movie file is selected in the remote file browser, automatically run a blocking `ffprobe` preview and show a spinner while it completes.
- Cache preview results briefly to keep repeated Streamlit reruns responsive; allow users to reload the preview manually.
- Display `ffprobe` results in separate sections instead of one sparse all-streams table:
  - container/format summary;
  - video streams;
  - audio streams;
  - subtitle streams.
- Video metadata should include codec, profile, resolution, pixel format, bitrate, frame rate, color range/space/transfer/primaries, side data/HDR-related metadata where available, language, title, and default flag.
- Audio metadata should include codec, profile, channels/layout, sample rate, bitrate, language, title, default, and forced flags.
- Subtitle metadata should include codec, language, title, default, forced, and hearing-impaired flags where available.
- Expose raw `ffprobe` JSON for deeper inspection.
- Keep a manual `ffprobe` action available for selected paths.
- Support `stat` on selected paths.

### Dashboard / Server Monitoring

- Provide a dashboard tab with a compact Jellyfin media library overview and a sortable table-style server resource overview covering all configured monitoring machines.
- Provide a dashboard shortcuts area that can open external websites now and later support internal shortcut types such as saved actions and user deep-links without redesigning the container.
- Dashboard shortcuts should support a small icon/preview field so cards can be visually recognizable without changing the underlying model later.
- Support OIDC login in the frontend using an OIDC client library, with backend JWT validation for protected API requests.
- Persist frontend OIDC auth state across tab reloads by storing the OIDC user and request state in browser localStorage.
- Provide Docker Compose deployment files at the repository root for production and local development. These deploy **only** the backend and frontend; Manage connects to *existing* Grafana/Prometheus/Alertmanager instances and never ships its own observability stack (see `docker-compose.observability.yml` for an optional standalone example).
- SSH private keys should be managed as reusable saved secrets in Settings, independent of any one machine, and SSH machines should select from that saved-key list.
- The web UI should allow both importing an existing private key and generating a new SSH keypair for that saved-key list.
- Saved SSH keys should display their derived public key, fingerprint, and machine usage count so administrators can audit them at a glance.
- The app should support optional SSH private key passphrases alongside the stored key material.
- The Settings tab should include a destructive local-database reset action protected by multiple acknowledgements and a typed confirmation phrase.
- Production compose should also pass the root `.env` into the backend container so runtime auth settings like `OIDC_ISSUER_URL` are available there, not just at compose interpolation time.
- The backend media index should persist in a Docker volume so a container restart or image rebuild does not force a new full index build.
- Compose deployment should not require `env_file`; required values should be supplied through environment interpolation or inline shell exports.
- The SSH key configuration should move from machine-local text areas to a saved SSH key registry; directory/name inputs remain only as legacy compatibility fields.
- Show Jellyfin media counts for movies, series, and series episodes on the dashboard.
- Show dashboard session activity from Jellyfin, including both currently playing sessions and logged-in idle sessions.
- Activity rows should include user, media title (or `(idle)`), playback state (`playing`/`paused`/`idle`), and whether transcoding is active.
- Provide a dashboard tab with a compact server resource overview over SSH.
- Provide a separate Monitoring tab for detailed resource charts, collector controls, diagnostics, raw samples, and per-machine recent action history gathered automatically by the backend.
- Expose backend poller status/configuration so the dashboard and Monitoring page can surface whether monitoring snapshots are being gathered automatically.
- The Monitoring tab should present one section per configured machine, and local vs remote machines should be treated the same in the UI with different connection/configuration data.
- Provide a Settings tab where monitoring machines can be added, edited, enabled/disabled, or deleted persistently.
- The app should start with no pre-seeded monitoring machines; users must explicitly add a local or SSH target before Monitoring shows anything.
- Local machines should be supported across Monitoring, Files, Jobs, and other remote-inspection tools without requiring SSH credentials.
- Creating a machine should automatically start the backend monitoring worker/poller so the new machine begins collecting snapshots without a separate manual step.
- The Monitoring tab should request all retained collector samples by default, while the dashboard overview can continue to use a shorter recent window.
- Show CPU and RAM usage for the last hour.
- Show IO wait percentage for the last hour.
- On the dashboard overview, summarize monitoring metrics as 10-minute averages with high/low values for quick inspection.
- Show average and spike/peak values for network throughput and disk I/O.
- Show used, available, and total disk space for the configured media root, falling back to `/`.
- The dashboard should present disk space as a single combined card with the progress/fill bar embedded inside the card and the size breakdown laid out clearly, with centered sub-card text for the Used/Free/Total breakdown and consistent vertical spacing across the dashboard cards.
- The disk usage bar should change color as usage increases so high utilization is easy to notice at a glance.
- The disk usage card should avoid redundant percentage labels next to the bar if the bar itself already communicates the value.
- (Superseded by the thin-dashboard observability model — 2026-06-17.) The app no longer renders in-app monitoring charts with D3; metrics/charts/logs live in the external Grafana stack, and the in-app Observability page surfaces Alertmanager alerts, Prometheus target health, and Grafana deep-links.
- Use a lightweight remote collector that reads Linux `/proc`, `/sys/block`, and `df` data into a JSONL file under `/tmp`.
- The collector should rotate/prune its JSONL metrics file so it does not grow unbounded; default retention is 7 days with a 70,000-line safety cap.
- The collector should be startable/stoppable/restartable from the dashboard and should not require installing a full monitoring stack.
- Machine definitions for monitoring should persist in backend-owned storage so local and remote monitoring targets survive restarts.
- Monitoring machine actions/history should also persist in backend-owned storage so each machine section can show recent status/metrics/disk/collector activity.
- The backend should periodically poll defined monitoring machines itself; no remote agent or push model should be required.
- Last-hour charts require the collector to have been running long enough to collect samples.
- Because the collector keeps only a bounded history, the Monitoring tab can safely load all retained samples up to the retention/max-lines cap.
- Network throughput should be shown as a combined traffic chart with download and upload lines.
- Network throughput should use bytes-per-second display units such as KB/s, MB/s, and GB/s to avoid bit/byte ambiguity.
- Disk throughput should be shown as a combined I/O chart with read and write lines.
- Network and disk throughput charts should scale values into readable units such as KB/s, MB/s, and GB/s.
- Each Monitoring chart should show compact summary chips such as min/avg/max for quick inspection.
- The Monitoring toolbar should offer quick time-range buttons such as 1h, 8h, 1 day, and 7 days in addition to free brush selection.
- The Monitoring brush selection should persist across data refreshes and tab reloads instead of resetting whenever new samples arrive, and the zoom buttons should stay in sync with the visible brush range.
- The Monitoring brush UI should be stable and remain visible after drags or zoom changes; it should be rendered independently from the chart redraw cycle, with usable resize handles for left/right edges.

### Saved Actions / Remote Jobs

- Provide an Actions tab for predefined server tasks that users can save and run later.
- The initial task types should support shell commands and Python scripts, while keeping the design flexible for future task types.
- Avoid arbitrary free-form command execution for ad-hoc execution; tasks should be stored records with an explicit name, type, content, enabled flag, default SSH task service, and notes.
- Support running tasks against `ssh_tasks` service instances only; local execution on the API host is no longer supported.
- Command/script content should be executed through the existing safe process helpers and shell-quoted where applicable.
- Future destructive actions should require explicit confirmations or dry-run style safeguards.
- Job templates should remain centralized in `jobs.py` for future extension.
- Remote job template values must be shell-quoted before execution.

## Service Registry and Dashboard Widgets

### Overview

External services (Grafana, Prometheus, Jellyfin, Nextcloud, SSH task runner) are
configured **in the app** and persisted in the backend SQLite database. Each
service instance holds non-secret config plus encrypted secret fields. Dashboard
widgets are either **service-bound** (reference a service instance + a widget
kind declared by that service) or **built-in / service-less** (backups, static
text).

Service definitions live as Pydantic modules in the backend
(`integrations/`); they declare the service config schema, secret fields, and
the widget kinds the service provides. There is no runtime plugin loading.

Every service `base_url` uses the shared `ServiceBaseUrl` type, which rejects
values missing an `http://` or `https://` schema with a clear validation error
(relative hosts break downstream HTTP clients).

### Services

- **Grafana** — base URL + optional API key; provides a dashboard-link widget.
- **Prometheus** — base URL + optional bearer token; provides a PromQL metric widget.
- **Jellyfin** — base URL + API key; provides a live-activity widget.
- **Nextcloud** — base URL + app password (no widgets yet).
- **SSH task runner** — host/port/username + saved SSH key reference + optional
  passphrase; provides a task-output widget. Tasks stay in the global saved-task
  registry; every run is recorded in `service_task_runs` as history.

Multiple instances per service type are supported. Services are managed from the
**Services** page (`/services`) and each instance has a detail page at
`/services/:serviceType/:serviceId`.

### Built-in widgets

- **Backups** — internal backup job summary and active alerts.
- **Static text** — plain text or markdown note.

These do not reference a service.

### Security

- Service secrets (API keys, tokens, passphrases) are **encrypted at rest** with
  Fernet using a single env-provided `MANAGE_ENCRYPTION_KEY`, which is always
  required to start the backend.
- Widget `config` and service `config` may not contain credential keys or
  secret-looking values; secrets go in the dedicated secret fields only.
- Plaintext secrets are never returned by the API; only `secrets_set` flags are
  surfaced.
- SSH task widgets only run tasks from the saved-task registry; arbitrary
  commands are not accepted.

### API

- `GET /api/services/types` — service definition metadata (config schema,
  secret fields, widget kinds).
- `GET /api/services/instances` — list service instances (no plaintext secrets).
- `POST /api/services/instances` — create instance.
- `PUT /api/services/instances/{id}` — update instance.
- `DELETE /api/services/instances/{id}` — delete instance (cascade-deletes
  widgets referencing it).
- `GET /api/widgets/builtin` — built-in (service-less) widget kinds.
- `GET /api/widgets/instances` — list widget instances.
- `POST/PUT/DELETE /api/widgets/instances/{id}` — widget CRUD.
- `GET /api/widgets/instances/{id}/data` — fetch widget data.

### Breaking change

Grafana/Prometheus URLs and credentials moved from environment variables into
service records. The legacy `GRAFANA_URL` / `PROMETHEUS_URL` backend settings and
the widget/addon-pages model were removed. `MANAGE_ENCRYPTION_KEY` is now required.

> **Follow-up (not in this change):** machine-level Jellyfin/Jellyseerr app
> config still powers the Media/Users/Files pages. Migrating those onto the
> service registry (and removing the machine app fields) is a separate change;
> see `openspec/changes/service-registry/design.md` §12.5.
>
> **Update (2026-06-23):** this follow-up is now complete — Jellyfin is no
> longer a machine service, and the dead machine-level `media_root`/`path_prefix`
> fields were removed. Jellyfin is configured exclusively via the service
> registry. The global `REMOTE_MEDIA_ROOT`/`REMOTE_PATH_PREFIX` config and
> `path_utils.py` remain for Jellyfin→SSH path resolution.

## Decision Log

- 2026-06-23: Completed the Jellyfin migration onto the service registry. Jellyfin is no longer a machine `services` tag (`DEFAULT_SERVICES` is now `["monitoring", "files"]`), and the dead machine-level `media_root`/`path_prefix` fields were removed from the settings store, `MonitoringMachineInput`, frontend types, and the Settings UI. Jellyfin is configured exclusively as a service-registry instance. The global `REMOTE_MEDIA_ROOT`/`REMOTE_PATH_PREFIX` config properties and `path_utils.py` are unchanged — `routers/files.py` and the media index still use them for Jellyfin→SSH path resolution. See `openspec/changes/jellyfin-service-registry/`.
- 2026-06-23: Made all observability integration service-registry first-class with zero service env config. Alertmanager is now a 6th service type (`integrations/alertmanager.py`) with an `active_alerts` widget, mirroring Grafana/Prometheus. The monitoring router resolves alertmanager/grafana/prometheus from service records (first-enabled-instance default, optional `?service_id=`); added `/api/monitoring/grafana-status` and `/prometheus-status` health probes. The Observability page discovers services from the registry (dropped `import.meta.env.VITE_GRAFANA_URL`) and renders health cards; the dashboard gained an `active_alerts` widget. Removed the backend `alertmanager_url`/`alertmanager_webhook_url` fields and the frontend `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` build args — observability services are configured in the UI only. Removed the `PROMETHEUS_FILE_SD_DIR` shared-file bridge + `write_prometheus_targets` file writer; external Prometheus now uses `http_sd_configs` against `GET /api/monitoring/prometheus-targets` (`build_node_exporter_targets` + that endpoint stay). The webhook receiver is log-only (dropped the outbound forward). The only observability env var remaining is `PROMETHEUS_ENABLED` (Manage's own `/metrics` toggle). See `openspec/changes/observability-service-registry/`.
- 2026-06-23: Removed all observability services from the root Compose files. `docker-compose.yml` and `docker-compose.dev.yml` now deploy **only** the backend and frontend; the `monitoring` network and the `prometheus`/`loki`/`alloy`/`grafana`/`alertmanager`/`node-exporter` services and their named volumes were deleted, and the `GRAFANA_APP_HOST` Traefik rule was removed. Manage connects to **existing** Grafana/Prometheus/Alertmanager instances and does not deploy them itself. The backend `alertmanager_url` default is now empty, and the `/api/monitoring/alerts` and `/alertmanager-status` endpoints return graceful "not configured" responses when `ALERTMANAGER_URL` is unset. The previous in-compose stack is preserved as an optional standalone example in `docker-compose.observability.yml` (config under `monitoring/`, documented in `docs/observability-runbooks.md`). (The `VITE_GRAFANA_URL`/`VITE_PROMETHEUS_URL` overrides were later removed entirely — see the service-registry entry above.)
- 2026-06-17: Decommissioned the legacy Manage-side system-metric scraping. Removed the backend `MonitoringPoller` (SSH-ran `df` on every machine every 5 min into a local SQLite `monitoring_machine_actions` table), the entire `services/monitoring_actions.py` module, the `/api/monitoring/poller`, `/api/monitoring/machines/{id}/actions`, and `/api/monitoring/disk` endpoints, the `monitoring_machine_actions` table (DROP on startup), the three `monitoring_poll_*` / `monitoring_action_retention_days` config knobs, and the orphaned frontend `DiskSpaceCard` + `DiskSpace` type. System metrics are now owned exclusively by Prometheus + node_exporter + Grafana. Kept the Alertmanager proxy (`/alerts`, `/alertmanager-status`, `/alertmanager-webhook`), `/prometheus-targets`, `/machines`, the `node_exporter_*` machine fields, and the on-demand `disk_usage` job template.
- 2026-06-17: Completed the web UI rework to a single design system. The frontend now uses **shadcn/ui + Tailwind CSS v4 + lucide-react** exclusively, with CSS `@theme` tokens in `src/index.css` (primary `#4f8cff`; `chart-1..5` repurposed as status/Grafana-link cues). Removed `@mui/material`, `@mui/icons-material`, `@mui/x-data-grid`, `@emotion/react`, `@emotion/styled`, `recharts`, `d3`, and the `src/theme.ts` shim. Tables moved from `@mui/x-data-grid`/AG Grid to a visibility-only TanStack `DataTable` wrapper (pagination, row selection, row click, column visibility — no sorting/resizing). Adopted the thin-dashboard observability model (no in-app charts; Alertmanager alerts + Prometheus target health + Grafana deep-links). Reconciled the information architecture: Backups is a top-level nav item at `/backups`, and the media surface is named Media at `/media` with `/applications` redirecting to `/media` (mirroring `/monitoring` → `/observability`). Frontend tests moved to Vitest + @testing-library/react (`npm test`), with legacy node suites in `frontend/tests`.
- 2026-06-13: Adopted a dedicated, self-hosted observability subsystem based on Prometheus, Grafana Loki, Grafana, and Alertmanager. Metrics will be pulled from Node Exporter on machines and from application exporters in containers; logs will be structured JSON shipped by Promtail/Grafana Alloy. The existing POSIX remote collector will be removed and backup alerts migrated to Alertmanager rules. See `docs/monitoring-logging-design.md`.
- 2026-06-13 (Phase 1): Added Prometheus, Loki, Grafana Alloy, Grafana, Alertmanager, and Node Exporter services to `docker-compose.yml` and `docker-compose.dev.yml`. Provisioned Grafana datasources and an initial `Manage Overview` dashboard as code. Configured Alloy to tail Docker logs and ship to Loki. Added Grafana generic OAuth configuration via `monitoring/grafana/grafana.ini` and a dedicated Traefik host rule. Added Alertmanager email routing with env-var interpolation. Added `/grafana` proxy to the Vite dev server for iframe embedding.
- 2026-06-13 (Phase 2): Extended machine settings with `node_exporter_enabled`, `node_exporter_port`, and `node_exporter_scrape_host`. Added Node Exporter install/restart/status job templates to `jobs.py`. Implemented `media_library_viewer_api.services.targets` to generate Prometheus file-SD target files and wired target regeneration into machine create/update/delete. Added `/api/monitoring/prometheus-targets` for live target previews. Configured Prometheus with a `node-exporter-remote` job reading file SD from the backend cache volume. Added a minimal `Node Exporter Overview` Grafana dashboard. Added unit and integration tests for target generation and the new endpoint.
- 2026-06-13 (Phase 3): Added `ALERTMANAGER_URL` setting and `GET /api/monitoring/alerts` plus `GET /api/monitoring/alertmanager-status` endpoints that proxy/summarize Alertmanager for the UI. Added `manage_backup_runs_last_success_timestamp` Prometheus gauge and wired `record_backup_run` to update it on success so the existing `BackupJobStuck` Alertmanager rule works. Updated `docker-compose.yml`, `docker-compose.dev.yml`, and `.env.example` with `ALERTMANAGER_URL`. Added tests for the new endpoints and metric helpers.
- 2026-06-13 (Phase 4): Added `/observability` React page (`frontend/src/components/ObservabilityPage.tsx`) with health cards for Alertmanager, active alerts, Prometheus targets, and machines; a recent-alerts list; machine selector; and Grafana iframe panels for Node Exporter metrics and Loki logs. Added `useObservability` hook and API client wrappers for `/api/monitoring/alerts`, `/api/monitoring/alertmanager-status`, and `/api/monitoring/prometheus-targets`. Added TypeScript types for Alertmanager summary/status and Prometheus targets. Wired the new route into `App.tsx` and the sidebar. Added shadcn/ui `card`, `badge`, `alert`, `skeleton`, and `select` components. Frontend build (`npm run build`) passes; lint has only pre-existing warnings.
- 2026-06-13 (Phase 5): Hardened observability containers with health checks and resource limits in both compose files; added service-healthy `depends_on` conditions. Added Prometheus scrape jobs for Loki, Alertmanager, and Grafana, plus new `observability_health` alert rules. Added `ALERTMANAGER_WEBHOOK_URL` backend setting, `POST /api/monitoring/alertmanager-webhook` receiver, and Alertmanager `webhook` receiver config. Added `docs/observability-runbooks.md` with operational playbooks. Updated `.env.example` to include all observability variables.
- 2026-05-03: Reaffirmed that the Monitoring tab charts should be rendered directly with D3 and expose brush-based time-range selection plus moving averages.
- 2026-05-03: Added hover tooltips, summary chips, a moving vertical cursor, snapped point markers, and a selected-range label to the D3 Monitoring charts for faster visual inspection.
- 2026-05-03: Combined network download/upload into one traffic chart and disk read/write into one I/O chart for clearer Monitoring layout.
- 2026-05-03: Confirmed the shared session activity table should keep the session identifier as a caption under the user name instead of a full column.
- 2026-05-03: Confirmed the shared session table should keep the compact overall status summary line above the rows.
- 2026-05-03: Updated the dashboard monitoring cards to show 10-minute averages with high/low subtext instead of only the latest sample.
- 2026-05-03: Added OIDC/JWT auth support plus root-level Docker Compose deployment files for production and dev workflows.
- 2026-05-04: Backend Docker Compose now mounts a host SSH directory into `/root/.ssh` so Paramiko can use a private key and strict host-key checking without baking secrets into the image.
- 2026-05-04: The frontend was adjusted to be more mobile-safe by making the app shell tabs scrollable, stacking header controls on narrow screens, and hiding low-priority table columns on smaller displays.
- 2026-05-04: The dashboard disk space area was consolidated into a single combined card with the progress bar embedded inside the card and the space stats reorganized into clearer, centered sub-panels with consistent vertical spacing.
- 2026-05-04: The disk usage bar was color-coded to shift from green to yellow to red as utilization increases, and the redundant percentage label beside the bar was removed.
- 2026-05-04: The Monitoring brush now persists its selected range in browser storage, avoids resetting when fresh monitoring data streams in, keeps the zoom buttons synchronized with the brush state, and renders the brush UI independently so data refreshes do not make it disappear.
- 2026-05-04: The Monitoring brush was rebuilt as a React overlay with explicit resize handles so mouse dragging is more reliable.
- 2026-05-04: The Media tab now persists search/filter/sort/pagination state, and the File Browser now persists the current directory plus selected file across reloads and tab switches.
- 2026-05-04: The app header was upgraded to a two-row branded layout with a left logo mark, right-side username/logout controls, and a separate navigation row.
- 2026-05-04: Frontend OIDC storage was switched from session-only defaults to localStorage-backed user/state stores so reloads keep the auth flow intact.
- 2026-05-04: API requests now fall back to the persisted OIDC user store for the bearer token so the first render after reload can avoid spurious 401s.
- 2026-05-04: Oversized frontend/backend modules were split into thin re-export entrypoints plus implementation modules to keep page/router/service code maintainable without changing behavior.
- 2026-05-04: The Monitoring charts and File Browser were also split into implementation modules behind thin entrypoints so the larger UI surfaces stay easier to navigate without changing runtime behavior.
- 2026-05-06: Monitoring became machine-based: a Settings tab now persists local/remote machine definitions, and the Monitoring tab renders a section per configured machine so API-host and remote targets are handled through the same UI model.
- 2026-05-06: Compose files were switched away from `env_file` and now rely on environment-variable interpolation, so deployments can be driven entirely by shell exports or inline environment values.
- 2026-05-06: Monitoring endpoints now translate machine-specific transport/runtime failures into user-facing HTTP errors so a broken machine only affects its own section instead of taking down the whole Monitoring page.
- 2026-05-06: Documentation now includes explicit Compose interpolation examples plus a monitoring-machine configuration workflow showing how to add local and SSH machines in the Settings tab.
- 2026-05-06: Monitoring machine action history was added so each machine section can display recent operation results, durations, and failures alongside the charts.
- 2026-05-06: Monitoring history collection was shifted to a backend-scheduled poller that reads the defined machines over SSH/local shell and stores snapshots in SQLite, avoiding any remote agent or push requirement.
- 2026-05-06: The dashboard monitoring section was converted from summary cards into a table of all configured machines, paired with backend poller status so the whole fleet can be reviewed at a glance.
- 2026-05-06: The dashboard monitoring table now shows 10-minute averages with min/max subtext and can be sorted by machine, status, and metric columns.
- 2026-05-07: The dashboard monitoring table should render each metric as a two-tier cell with a dominant centered 10-minute average in the upper area and a smaller lower area for min/max chips so the average is the visual focus.
- 2026-05-07: The lower min/max area in each monitoring metric cell should span the full width of the cell, and table/header spacing should stay tuned for a dense admin-console layout.
- 2026-05-07: The monitoring metric average should be explicitly labeled as a 10m average in the cell so the summary value is not ambiguous.
- 2026-05-07: The monitoring overview's Updated column should use a compact fixed timestamp format instead of locale-specific output for easier scanning, and the cell should show two stacked lines: a readable month/day timestamp and a compact clock-plus-age line.
- 2026-05-07: The monitoring overview table should allow horizontal scrolling when the dense column set exceeds the viewport width.
- 2026-05-07: The monitoring overview table was rewritten so the machine, mode, status, metric, and updated columns each have explicit widths and the metric cells use a clean two-tier layout with full-width min/max rows.
- 2026-05-07: Monitoring collector status should show a simple running/not-running state in the UI rather than surfacing backend process IDs.
- 2026-05-07: Creating or updating a monitoring machine should kick the poller immediately so the backend starts collecting snapshots right away instead of waiting for the next interval.
- 2026-05-07: The monitoring overview machine cell should reserve more room for the machine name and mode columns so the name does not overlap the mode text.
- 2026-05-06: The Monitoring page now includes a poller-health badge in the header so users can quickly see whether backend collection is active.
- 2026-05-06: The dashboard monitoring table now renders each metric summary with compact stacked low/high lines to keep the table narrower, and the activity/session table no longer hides columns on mobile so all details remain available.
- 2026-05-06: The dashboard monitoring table now renders the 10-minute value as the visual focus and keeps the low/high lines smaller as supporting detail.
- 2026-05-06: The file browser and media routes now support machine-specific SSH/Jellyfin selection via the request machine id, and the backend resolves clients from configured machines before falling back to legacy env-based defaults.
- 2026-05-06: Application settings now include per-machine Jellyfin/Jellyseerr configuration and multi-select service roles so the UI can manage app hosts from the same machine registry.
- 2026-05-06: The app shell now uses an Applications top-level tab with a Jellyfin subtab for media/library work and a placeholder Nextcloud subtab for future expansion.
- 2026-05-06: The settings model now treats Jellyfin/Jellyseerr as machine-level configuration instead of global env-only values, so app hosts can be edited alongside other machine services.
- 2026-05-06: The backend now synthesizes a managed `known_hosts` file from configured SSH machines at startup, avoiding a mounted SSH directory while keeping strict host-key verification enabled.
- 2026-05-06: SSH credentials were moved toward reusable saved key records in Settings, so machines can point at a shared SSH key instead of storing their own duplicate private key text.
- 2026-05-06: The Settings page now includes an SSH key registry UI with create/edit/delete flows and a generate-key action so users can make a reusable key directly in the web interface.
- 2026-05-06: Saved SSH keys now surface a derived public key, fingerprint, and per-key machine usage count in the Settings UI for easier auditing.
- 2026-05-06: The dev Compose stack now starts without any SSH key material at all unless a user later configures remote SSH machines.
- 2026-05-06: The Settings page now exposes a protected local-database reset flow that requires several explicit acknowledgements and a typed confirmation phrase before it can delete the cached app databases.
- 2026-05-06: The Actions page was redesigned into a compact tabbed workspace with a left tab rail of saved actions, and both new-action creation and editing now open in popups instead of inline forms.
- 2026-05-07: Added a reusable dashboard shortcuts container with persisted records so the dashboard can link to external websites now and later support action/user shortcut types from the same model.
- 2026-05-07: Dashboard shortcuts gained an optional icon/preview field so cards can be visually differentiated while keeping future shortcut types extensible.
- 2026-05-07: The dashboard shortcut editor was tightened with compact type guidance and shorter helper text so the popup stays readable without wasting vertical space.
- 2026-05-07: SSH key records should persist and display the derived public key and fingerprint, not just the private key blob, so imports and generated keys are auditable without recomputation.
- 2026-05-07: SSH machine creation/editing should present a saved-key dropdown and warn when no SSH keys exist yet, instead of forcing manual key-id entry.
- 2026-05-07: Saved task runs should return structured failure output for local execution problems instead of surfacing a generic 500 error.
- 2026-05-07: SSH dependency resolution should keep its cached tuple shape aligned with the legacy and machine-specific SSH settings so SSH clients can be created without tuple-unpack crashes.
- 2026-05-07: Machine creation was adjusted so dialog edits are controlled by the parent form state, ensuring all entered fields are actually saved.
- 2026-05-07: Local machines now work through the same Files/Jobs/monitoring tool paths without SSH credentials, and creating a machine starts the monitoring worker automatically.
- 2026-05-06: Closing an edited Action popup now warns before discarding unsaved changes.
- 2026-05-06: The Settings page was reworked into a compact tabbed layout with separate Machines, SSH Keys, and Danger Zone sections, and irrelevant machine/service fields now hide when that mode or service is not selected.
- 2026-05-06: The Settings machine list was converted into a compact table with popup editing so the page no longer repeats the full machine form for every entry.
- 2026-05-06: The Monitoring page was reworked into an overview tab plus one tab per configured machine, so each machine gets its own monitoring page.
- 2026-05-06: The Actions page now shows a visible empty-state container when no action is selected, and the Monitoring tabs were returned to the app's standard tab styling.
- 2026-05-06: The Monitoring tab spacing was tightened and the Applications page was updated to match the app's standard card-and-tab design.
- 2026-05-06: The dashboard library stats were moved into the Jellyfin Applications tab above the media table, and the dashboard activity panel was relabeled as Jellyfin activity.
- 2026-05-06: The dashboard, Monitoring overview, Applications, and File Browser layouts were normalized around shared titled containers, with Monitoring showing the fleet overview before machine tabs and File Browser using machine tabs with Browser/Media info/Jobs stacked in each machine view.
- 2026-05-06: The Applications and Monitoring tab containers were refactored into a reusable bordered tab-card pattern so the active tab's content sits inside the same card as the tab rail.
- 2026-05-06: The Settings Machines tab was changed from a repeated form layout to a left-hand machine list with a right-hand details panel and popup editing.
- 2026-05-06: The app no longer seeds a default local monitoring machine; a fresh database starts with an empty machine list until the user creates one.
- 2026-05-06: The File Browser was reworked into Browser / Media info / Jobs subtabs.
- 2026-05-06: The app shell received a small density pass that tightened container padding and tab widths to make the whole site feel more compact.
- 2026-05-06: The tab rails across Actions, Monitoring, Settings, and Files were restyled to be more enterprise-console-like with compact pills, clearer active states, and reduced visual noise.
- 2026-05-06: Added an Actions tab for saved server tasks, with backend persistence, per-task run history, and support for shell/Python task types on either local or SSH machines.
- 2026-05-06: Reusable dialog footers now keep cancel on the left and confirm on the right, and hover edit buttons now appear on the right edge of editable list rows in Actions and Settings.
- 2026-05-06: Library stats, Jellyfin activity, and Monitoring overview now use shared section-container patterns so subcontainers stay consistent across the app.
- 2026-05-07: The app versioning scheme should be hybrid: auto-detect package/build metadata when available, but allow explicit overrides for deployments that need fixed labels.
- 2026-05-07: The shell should display both frontend and backend version labels so deployed builds are easy to identify without opening a separate diagnostics screen.
- 2026-05-07: SSH host verification should use trust-on-first-use for new machines by recording the first observed host key into the backend-managed known_hosts file, while still rejecting later key mismatches.
- 2026-05-07: The SSH machine editor should expose a validation button that tests banner/auth flow and records the host key before save so users get clear feedback when a host is unreachable.
- 2026-05-07: Saving a monitoring-capable machine should validate the banner/auth flow, update the backend-managed known_hosts entry for the current host, and start the remote resource collector so charts populate without a separate manual step.
- 2026-05-07: Machine settings should visually separate Connection, Monitoring / Files, Jellyfin, Jellyseerr, and Notes into clearly labeled sections.

## Backup Monitoring

### Overview

The system receives backup execution reports from an external backup tool via HTTP API, stores job and run history, and provides alerting on failures, missed schedules, and anomalies.

### API

- `POST /api/backups/report` — Submit backup run (Bearer token auth)
- `POST /api/backups/report/start` — Mark backup as in_progress
- `GET /api/backups/jobs` — List jobs
- `GET /api/backups/runs` — List runs
- `GET /api/backups/alerts` — List alerts
- `POST /api/backups/alerts/{id}/acknowledge` — Acknowledge alert
- `GET /api/dashboard/backups` — Dashboard summary

### Data Model

- **BackupJob**: id, name, source, target, schedule_interval_seconds, created_at
- **BackupRun**: id, job_id, started_at, ended_at, status, bytes_transferred, duration_ms, error_message, details_json
- **BackupAlert**: id, job_id, run_id, alert_type, severity, message, acknowledged, resolved_at

### Alert Types

- `failed_status` — Backup reported failure (critical)
- `missed_schedule` — No run within 1.5x expected interval (warning)
- `anomaly_size` — Size is 0 or <10% / >300% of 7-day median (warning)
- `anomaly_duration` — Duration >300% of 7-day median (warning)

### Authentication

- Backup tool uses auto-generated Bearer API key
- Frontend uses existing OIDC/JWT auth

## Information Architecture (services-as-hub)

The app is organized around **services as the hub**. The top-level navigation
contains a small always-visible core plus conditional per-type entries and
user-created named dashboards.

### Top-level navigation

- **Main Dashboard** (`/`) — always visible, special (not deletable, default landing).
- **Named dashboards** (`/d/:slug`) — one top-level entry each, user-controlled order, composed of pinned service links (and widgets in a follow-up).
- **Conditional service-type entries** — appear only when at least one enabled instance of the type exists: `jellyfin`→Media, `ssh_tasks`→Files+Actions, `alertmanager`→Alerts, `grafana`→Grafana, `prometheus`→Prometheus, `backups`→Backups, `authentik`→Users. `nextcloud` contributes no entry.
- **Services** (`/services`) — always visible admin hub for managing service instances and named dashboards.
- **Settings** (`/settings`) — always visible.

### Service page

Every service page uses the tab skeleton `[Overview | type-specific content tabs | Widgets | Config]`. Content tabs per type: jellyfin=Media+Requests, ssh_tasks=Files+Actions, backups=Jobs, authentik=Users+Messaging, alertmanager=Alerts, grafana=Links, prometheus=Metrics. When >1 enabled instance of a type exists, an instance switcher appears at the top.

Routing: `/services/:type/:id` (specific instance), `/services/:type` (resolves first enabled instance, redirects).

### Service type registry

Eight types: `alertmanager`, `authentik`, `backups`, `grafana`, `jellyfin`, `nextcloud`, `prometheus`, `ssh_tasks`. `jellyseerr` was absorbed into Jellyfin config (optional `jellyseerr_url`/`jellyseerr_api_key` fields); existing Jellyseerr service instances were migrated at startup. `backups` and `authentik` are new.

### Users → Authentik

The Jellyfin-backed Users page is removed. Authentik is the user-directory source (OIDC auth unchanged). The Authentik service page has a Users tab (directory) and a Messaging tab (compose via the existing SMTP/mail queue).

### Observability split

The cross-service Observability page is removed. Alertmanager/Grafana/Prometheus each have their own service-type tabs. Users who want a cross-service overview build it via widgets on a named dashboard.

### Legacy routes

`/media`, `/files`, `/actions`, `/users`, `/observability`, `/backups` return 404 (no redirects). Bookmarks must be updated.

### Empty state

A fresh install lands on the Main Dashboard with an "Add a service" CTA until services are configured.