Route all prometheus widget queries through Grafana /api/ds/query instead of
direct Prom HTTP. PrometheusConfig: drop base_url, add grafana_url +
datasource_uid; secret grafana_api_key (required). PrometheusWidgetSource →
MetricSource with _gateway_query POST method. normalize_grafana_frames
recovered from 65bae95 + shared _dedup_label helper. Gateway-path status
check. Startup old-config validation. CHANGELOG migration note. All adapter
tests rewritten for POST /api/ds/query + Grafana frames mock. Backend: 331
pytest pass, ruff clean. Frontend: build green (unchanged in S1).
The build endpoint had Depends(get_jellyfin_client) and Depends(get_user_id)
which executed BEFORE the function body. If Jellyfin was unreachable, these
raised HTTPException(503), the function never ran, and the worker was never
started. The frontend mutation had no onError handler, so the failure was
completely silent — the button briefly showed 'Building...' then reverted
to 'Build index' with zero feedback.
Backend fix: removed the Jellyfin dependencies from post_build_index.
The worker subprocess resolves its own Jellyfin connection via
_resolve_jellyfin(service_id) — the endpoint just needs to start the
worker process. The libraries count starts at 0 and gets updated by
the worker once it connects.
Frontend fix: added onError to useBuildIndex that invalidates the status
query (so the UI reflects the non-building state). MediaTab now displays
the build error inline: 'Build failed: <message>' next to the button.
283 backend tests pass (updated build test for new no-dependency flow);
128 frontend tests pass; ruff/eslint clean.
Extract the duplicated _resolve_service_record helper (identical in
routers/monitoring.py and routers/authentik_users.py) into a shared
services/service_resolution.py module. Both routers now import
resolve_service_record from the shared module.
The authentik router previously hardcoded service_type='authentik' in
its local copy; the shared helper takes service_type as a param (same
as monitoring's did).
Tests updated: test_api.py patches now target the correct module paths
(resolve_service_record on the monitoring module where it's imported,
build_service_record on the service_resolution module).
283 backend tests pass; ruff clean.
On a fresh deploy with no Jellyfin service configured yet,
get_jellyfin_client (and get_user_id / get_ssh_client) raised a plain
RuntimeError, which bubbled up as a 500 traceback on every
Jellyfin-dependent route (dashboard counts/libraries/activity, media,
users). Convert those RuntimeErrors to HTTPException(503) with a clear
detail message so FastAPI returns a clean 503 JSON response instead of
a 500, and the frontend can render a not-configured state.
- dependencies.py: get_jellyfin_client (no service / missing creds),
get_user_id (no users discovered), and get_ssh_client (no SSH machine
+ no legacy key path) now raise HTTPException(503, detail=...).
- tests/test_api.py: added
TestDashboard.test_jellyfin_endpoints_return_503_when_not_configured
covering /api/dashboard/counts and /activity.
ruff clean; 240 backend tests pass.
Slice 4 of observability-service-registry. Removes the shared-file
Prometheus bridge; external Prometheus now consumes node-exporter targets
via http_sd_configs against GET /api/monitoring/prometheus-targets.
- services/targets.py: removed write_prometheus_targets() (the file
writer) and its json/Path/get_settings imports; updated module docstring.
build_node_exporter_targets() is unchanged and still powers the HTTP
endpoint.
- main.py: removed the startup write_prometheus_targets call.
- routers/settings.py: removed the _write_prometheus_targets helper and
its three post machine create/update/delete call sites + the now-unused
targets import.
- config.py: removed the prometheus_file_sd_dir field.
- docker-compose.yml / docker-compose.dev.yml: removed the
PROMETHEUS_FILE_SD_DIR backend env var.
- tests: removed TestWritePrometheusTargets + the write_prometheus_targets
import in test_targets.py; rewrote the two TestSettingsMachines tests to
assert machines appear/disappear from /api/monitoring/prometheus-targets
(the surviving HTTP path) instead of the removed file-writer side effect.
ruff clean; 239 backend tests pass.
Slice 2 of observability-service-registry. The monitoring router resolves
observability components from the service registry instead of env vars.
- routers/monitoring.py: removed _alertmanager_client/_webhook_client env
readers + the get_settings import. Added _resolve_service_record(store,
service_type, service_id?) -> ServiceRecord|None (requested instance with
type+enabled checks, else first enabled instance), plus _base_url/_timeout/
_auth_headers (Bearer from api_key)/_status_response helpers.
- /alerts + /alertmanager-status now take service_id? + Depends(store),
resolve an alertmanager service, return graceful not-configured/
unreachable payloads including service_id/name; status down-branches now
include peers:[] + error (fixes prior type drift).
- NEW /grafana-status (probes /api/health) and /prometheus-status (probes
/-/healthy then /api/v1/status/buildinfo) returning
{up,version,service_id,name,error}.
- Webhook receiver is now log-only (dropped the outbound
ALERTMANAGER_WEBHOOK_URL forward).
- tests: rewrote TestAlertmanager + TestAlertmanagerWebhook to mock
_resolve_service_record/requests.get (not-configured via empty registry);
added TestGrafanaStatus/TestPrometheusStatus and a TestResolveServiceRecord
unit class covering service_id match/type-mismatch/disabled and first-
enabled/none-enabled paths.
Orphaned config fields alertmanager_url/alertmanager_webhook_url and the
env-var removal land in Slice 5. ruff clean; 240 backend tests pass.
Reviewed fresh-context (read-only): no blockers.
Manage now connects to existing Grafana/Prometheus/Alertmanager instances
and never deploys its own stack.
- docker-compose.yml / docker-compose.dev.yml: removed prometheus, loki,
alloy, grafana, alertmanager, node-exporter services, the monitoring
network, and observability named volumes; they now ship only backend +
frontend. Dev frontend now joins the web network so the Vite dev proxy
can reach the backend.
- backend: alertmanager_url default is now empty; /api/monitoring/alerts
and /alertmanager-status return graceful "not configured" responses
when ALERTMANAGER_URL is unset. Added not-configured tests.
- docker-compose.observability.yml: kept as the optional standalone
example; header clarifies Manage does not deploy it.
- Removed orphaned combined monitoring/prometheus/prometheus.yml
(standalone stack uses prometheus.standalone.yml).
- Docs (README, REQUIREMENTS decision log, monitoring-logging-design,
observability-runbooks, context.md, MIGRATION_PLAN, frontend/README,
CHANGELOG) updated to the connect-to-existing model.
VITE_GRAFANA_URL / VITE_PROMETHEUS_URL remain as optional frontend
deep-link overrides. .env.example still needs a manual update (safety
policy blocks assistant edits): set ALERTMANAGER_URL empty/optional and
move standalone-only vars out of the root file.
The 2026-06-16/17 observability update externalised metrics to
Prometheus + node_exporter + Grafana, but the legacy Manage-side
SSH-scraping monitor was never removed. It duplicated the new stack,
ran SSH df on every machine every 300s, and fed nothing (its UI was
deleted in e2ad731). This slice decommissions the duplication.
Removed (backend):
- services/monitoring_poller.py (MonitoringPoller) — entire file
- services/monitoring_actions.py (disk_space, run_machine_operation,
poll_machine_snapshot, build_machine_client) — entire file;
run_machine_operation had only 2 callers (the poller + /disk), both gone
- tests/test_monitoring_actions.py
- endpoints: POST /api/monitoring/poller, GET /machines/{id}/actions,
GET /disk (and the now-dead _resolve_machine helper)
- lifespan wiring (main.py), dependency wrapper (dependencies.py),
poller.start()/kick() from machine save (routers/settings.py)
- SettingsStore: monitoring_machine_actions table CREATE + 2 indexes +
record/list/prune_machine_actions methods; DROP TABLE IF EXISTS on
startup cleans existing DBs (user-approved)
- config knobs: monitoring_poll_interval_seconds,
monitoring_poll_initial_delay_seconds, monitoring_action_retention_days
- test_api.py: TestMonitoring._ensure_machine + test_disk
Kept (fits the new model): /machines, /prometheus-targets, /alerts,
/alertmanager-status, /alertmanager-webhook; the disk_usage JOB template
(manual on-demand, not monitoring); node_exporter_* machine fields
(they point Prometheus at the right host).
Gate: backend pytest 173 passed; ruff clean.
Developer
alex