diff --git a/openspec/changes/grafana-metric-gateway/spec.md b/openspec/changes/grafana-metric-gateway/spec.md new file mode 100644 index 0000000..4656ecf --- /dev/null +++ b/openspec/changes/grafana-metric-gateway/spec.md @@ -0,0 +1,305 @@ +# SDD Spec: Grafana Metric Gateway + +**Change:** `grafana-metric-gateway` +**Phase:** spec +**Date:** 2026-07-09 + +This spec defines the acceptance requirements for the change. Requirements use the `GM-` prefix (the `prometheus-charting` canonical domain, created by `prometheus-direct-charting`, uses `SC-`; this change supersedes several of those — see §"Canonical delta intent"). + +## Requirement categories + +1. Gateway transport (config + query path) +2. Frames → series normalization +3. Window presets + step mapping +4. Widget kind behavior (chart / gauge / mean / metric) +5. Status check via gateway +6. Widget rename to neutral `Metric*` +7. Startup validation + migration +8. Test + build greenness + +--- + +## 1. Gateway transport (config + query path) + +### GM-101 — Prometheus service config points at a Grafana gateway + +The `prometheus` service type's config model MUST declare `grafana_url: ServiceBaseUrl` (http(s)-schema-enforced), `datasource_uid: str = "prometheus"`, and `timeout_seconds: int = 10`. The former `base_url` field MUST be removed. The secret schema MUST declare `grafana_api_key` (required, encrypted via Fernet) and MUST NOT carry the old optional `api_key`. + +#### Scenario: Config schema reflects gateway fields + +- GIVEN the `grafana-metric-gateway` change is applied +- WHEN `integrations/prometheus.py` `PrometheusConfig` is inspected +- THEN it declares `grafana_url`, `datasource_uid`, `timeout_seconds` +- AND it does NOT declare `base_url` +- AND `DEFINITION.secret_fields` contains `grafana_api_key` (required=True) +- AND `DEFINITION.secret_fields` does NOT contain `api_key` + +### GM-102 — All metric queries route through Grafana /api/ds/query + +The widget source adapter (renamed `MetricSource`, service-type key stays `"prometheus"`) MUST issue `POST {grafana_url}/api/ds/query` for all four widget kinds (`chart`, `gauge`, `mean`, `metric`). The request body MUST contain a `queries` array with `{datasource: {uid: , type: "prometheus"}, expr: , format: "time_series", intervalMs: , maxDataPoints: , refId: "A"}` plus `from`/`to` time bounds. The request MUST carry `Authorization: Bearer {grafana_api_key}`. The adapter MUST NOT issue any direct `GET /api/v1/query` or `GET /api/v1/query_range` call to a Prometheus URL. + +#### Scenario: Chart fetch uses gateway POST + +- GIVEN a `prometheus` service configured with `grafana_url`, `grafana_api_key`, `datasource_uid` +- WHEN a `chart` widget data fetch is executed (mocked HTTP) +- THEN the adapter issues `POST {grafana_url}/api/ds/query` +- AND the request body contains `queries[0].datasource.uid == datasource_uid` +- AND the request body contains `queries[0].expr == ` +- AND the `Authorization` header is `Bearer {grafana_api_key}` +- AND no `GET /api/v1/query_range` call is made + +#### Scenario: No direct Prom HTTP call + +- GIVEN the change is applied +- WHEN `widgets/sources.py` `MetricSource` is inspected +- THEN no code path constructs a URL matching `/api/v1/query` or `/api/v1/query_range` against a Prometheus `base_url` + +### GM-103 — Gateway errors degrade gracefully + +A Grafana timeout, connection error, HTTP 401/403 (auth), or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional. The error message MUST be specific enough to distinguish auth failure from unreachable-host from datasource-not-found where the Grafana response permits. + +#### Scenario: Auth failure surfaces a readable error + +- GIVEN a `prometheus` service with an invalid `grafana_api_key` +- WHEN a widget data fetch is executed (mocked HTTP returns 401) +- THEN the adapter returns `{ "error": }` +- AND no exception propagates + +#### Scenario: Timeout surfaces a readable error + +- GIVEN a `prometheus` service whose Grafana gateway does not respond within the timeout +- WHEN a widget data fetch is executed +- THEN the adapter returns `{ "error": }` + +--- + +## 2. Frames → series normalization + +### GM-104 — Frames normalizer restored and shared + +A `normalize_grafana_frames(raw) -> list[dict]` helper MUST exist in `widgets/prometheus_range.py` alongside the existing `normalize_prometheus_matrix`. It MUST parse Grafana's `/api/v1/ds/query` response (`results..frames[]` with `data.values` + `schema.fields`) and produce the SAME `{label, points:[{t:int, v:float|null}]}` series shape the frontend chart renderer already consumes. The label-dedup rule (`label (n)` suffix on collision) MUST be shared with `normalize_prometheus_matrix` (extracted to a common inner helper, not duplicated). + +#### Scenario: Frames normalized to series shape + +- GIVEN a sample Grafana `/api/ds/query` response with one frame containing timestamps `[1000, 2000]` and values `[1.5, 2.5]` +- WHEN `normalize_grafana_frames(raw)` is called +- THEN it returns `[{"label": , "points": [{"t": 1000, "v": 1.5}, {"t": 2000, "v": 2.5}]}]` + +#### Scenario: Label fallback chain + +- GIVEN a frame whose value-field has `config.displayName` set +- WHEN normalized +- THEN the label is the displayName +- GIVEN a frame whose value-field has Prometheus `labels` but no displayName +- WHEN normalized +- THEN the label is the sorted `k=v` join (excluding `__`-prefixed keys) +- GIVEN a frame with neither displayName nor labels +- WHEN normalized +- THEN the label is `"value"` + +#### Scenario: Dedup shared with matrix path + +- GIVEN two frames that would produce the same label +- WHEN normalized +- THEN the second gets a `(1)` suffix +- AND the dedup logic is not copy-pasted (a shared helper or shared suffix rule is used by both `normalize_grafana_frames` and `normalize_prometheus_matrix`) + +--- + +## 3. Window presets + step mapping + +### GM-105 — Window presets map to Grafana from/to + intervalMs + +The existing `WINDOW_PRESETS` (`1h`/`6h`/`24h`/`7d` → seconds) and `step_for_window` math MUST be reused to derive the gateway request's `from` (e.g. `"now-1h"`), `to` (`"now"`), and `intervalMs` (`step * 1000`). The resulting point count MUST land in the 100–300 band (same target as the pre-change direct-Prom path). Users do not configure `from`/`to`/`step`/`intervalMs` directly. + +#### Scenario: 1h preset maps correctly + +- GIVEN a `chart` widget with `window: "1h"` +- WHEN the gateway request body is constructed +- THEN `from` resolves to approximately `now - 3600s` +- AND `to` is `"now"` +- AND `intervalMs` is `step_for_window(3600) * 1000` +- AND the expected point count is ~200 + +--- + +## 4. Widget kind behavior (chart / gauge / mean / metric) + +### GM-106 — Chart renders multi-series via gateway + +The `chart` widget MUST query the gateway with a range query (PromQL expr + window preset) and return `{ "series": [...] }` via `normalize_grafana_frames`. The frontend `MetricChartWidget` MUST render all returned series via `LineSeriesChart`, each as its own line with a distinct color. There is no single-series restriction. + +#### Scenario: Multi-series chart through gateway + +- GIVEN a `chart` widget whose PromQL returns two series +- WHEN data is fetched via the gateway (mocked) +- THEN the adapter returns `{ "series": [<2 entries>] }` +- AND the frontend renders two distinct lines + +### GM-107 — Gauge renders instant scalar via gateway + +The `gauge` widget MUST query the gateway with an instant PromQL query (mapped to a Grafana instant datasource query or a range query with a single point) and return `{ "value": float, "warn_at"?, "crit_at"?, "min"?, "max"?, "unit"? }`. If the query returns multiple series, the adapter MUST return `{ "error": str }` (scalar-only). The frontend `MetricGaugeWidget` MUST render threshold bands when `warn_at`/`crit_at` are set; otherwise a single neutral gauge. + +#### Scenario: Gauge scalar-only through gateway + +- GIVEN a `gauge` widget whose PromQL returns one scalar +- WHEN fetched via the gateway +- THEN the adapter returns `{ "value": , ... }` +- GIVEN a `gauge` widget whose PromQL returns multiple series +- WHEN fetched +- THEN the adapter returns `{ "error": }` + +### GM-108 — Mean computes over a gateway range window + +The `mean` widget MUST query the gateway with a range query over the configured window preset, average all non-null point values of the single series client-side, and return `{ "value": float, "unit"? }`. Scalar-only: multiple series → `{ "error": str }`. + +#### Scenario: Mean through gateway + +- GIVEN a `mean` widget with `window: "1h"` and a PromQL returning one series +- WHEN fetched via the gateway +- THEN the adapter returns `{ "value": }` + +### GM-109 — Metric widget returns scalar via gateway + +The instant `metric` widget MUST query the gateway and return `{ "result": }` (same shape as today) sourced via Grafana rather than direct Prom. + +#### Scenario: Metric instant query through gateway + +- GIVEN a `metric` widget with a PromQL expression +- WHEN fetched +- THEN the adapter issues a gateway POST (not a direct Prom GET) +- AND returns the scalar result in the existing `{ "result": ... }` shape + +--- + +## 5. Status check via gateway + +### GM-110 — get_prometheus_status validates the full gateway path + +`get_prometheus_status` MUST run a trivial query (e.g. `up`) through the Grafana gateway. Success → `{ "up": true, "version": , ... }` (version derived from the successful path; if Grafana's response doesn't carry a Prom version, a sentinel like `"ok"` is acceptable). Failure → `{ "up": false, "error": }` distinguishing auth failure, unreachable host, and datasource-not-found where the response permits. + +#### Scenario: Healthy gateway + +- GIVEN a `prometheus` service with valid gateway credentials +- WHEN `get_prometheus_status` is called +- THEN it issues a gateway POST with `expr: "up"` +- AND returns `{ "up": true, ... }` + +#### Scenario: Auth failure in status + +- GIVEN a `prometheus` service with an invalid `grafana_api_key` +- WHEN `get_prometheus_status` is called +- THEN it returns `{ "up": false, "error": }` + +--- + +## 6. Widget rename to neutral Metric* + +### GM-111 — Widget files renamed via git mv + +`PrometheusChartWidget.tsx` → `MetricChartWidget.tsx`, `PrometheusGaugeWidget.tsx` → `MetricGaugeWidget.tsx`, `PrometheusMeanWidget.tsx` → `MetricMeanWidget.tsx`, plus their test files, MUST be renamed via `git mv` (history preserved). The exported component names MUST change to `MetricChartWidget` / `MetricGaugeWidget` / `MetricMeanWidget`. The rendering code (recharts, `LineSeriesChart`, gauge bands) MUST be preserved unchanged. + +#### Scenario: No Prometheus*Widget names remain + +- GIVEN the change is applied +- WHEN `grep -r "PrometheusChartWidget\|PrometheusGaugeWidget\|PrometheusMeanWidget" frontend/src` is run +- THEN it returns no matches +- AND `MetricChartWidget.tsx`, `MetricGaugeWidget.tsx`, `MetricMeanWidget.tsx` exist + +### GM-112 — Registry binds Metric* components + +`integrations/registry.ts` MUST bind the `prometheus` service's `chart`/`gauge`/`mean` widget kinds to the renamed `Metric*` components. The widget KIND names (`chart`/`gauge`/`mean`/`metric`) MUST NOT change (they are persisted in widget instance rows). + +#### Scenario: Registry uses Metric* components + +- GIVEN the change is applied +- WHEN `SERVICE_REGISTRY.prometheus.widgets` is inspected +- THEN the `chart` kind's `component` is `MetricChartWidget` +- AND the kind strings are still `"chart"`, `"gauge"`, `"mean"`, `"metric"` + +--- + +## 7. Startup validation + migration + +### GM-113 — Startup rejects old prometheus config shape + +When the backend starts and a persisted `prometheus` service instance has a `config` dict containing the old `base_url` key (without `grafana_url`), the startup config validation MUST log a clear migration message (naming the service) and either skip the instance gracefully or mark it as misconfigured — it MUST NOT crash startup. + +#### Scenario: Old-shape config logged not crashed + +- GIVEN a persisted `prometheus` service with `config: {"base_url": "http://prom:9090"}` (no `grafana_url`) +- WHEN the backend starts +- THEN startup completes (no crash) +- AND a log message names the service and instructs reconfiguration + +### GM-114 — CHANGELOG documents the migration + +`CHANGELOG.md` `[Unreleased]` MUST include an entry stating that `prometheus` service instances must be reconfigured: replace `base_url` with `grafana_url`, add the `grafana_api_key` secret, and optionally set `datasource_uid`. It MUST be marked **BREAKING**. + +#### Scenario: CHANGELOG entry present + +- GIVEN the change is applied +- WHEN `CHANGELOG.md` is inspected +- THEN an `[Unreleased]` entry mentions `grafana_url`, `grafana_api_key`, `datasource_uid` +- AND it is marked BREAKING + +--- + +## 8. Test + build greenness + +### GM-115 — Backend tests + ruff green + +`PYTHONPATH=src python3 -m pytest -q` and `PYTHONPATH=src python3 -m ruff check src tests` from `backend/` MUST pass. New/updated tests MUST cover: gateway POST assertion (mocked), `normalize_grafana_frames` (label fallback chain, dedup, NaN handling), gauge/mean scalar-only through gateway, status check through gateway, and the startup old-config validation. + +### GM-116 — Frontend build + lint + test green + +`npm run build`, `npm run lint`, and `npx vitest run` from `frontend/` MUST pass (0 errors). The renamed `Metric*` widget tests MUST cover loading, error, and rendered states. + +--- + +## Canonical delta intent + +This is the project's **first non-additive canonical sync**. The `prometheus-charting` canonical domain (`openspec/specs/prometheus-charting/spec.md`) has 27 requirements (SC-101..SC-127). This change MODIFIES the transport-related requirements, PRESERVES the durable-behavior requirements, and ADDS new requirements. The sync phase (`sdd-sync`) MUST author a `## MODIFIED Requirements` delta (not `## ADDED` only). + +### MODIFIED (supersedes a canonical SC-xxx — intent may be preserved or changed) + +| GM-req | Supersedes | What changes | +|---|---|---| +| GM-101 | SC-101 (partial) | Transport: direct Prom `/api/v1/query_range` → Grafana `/api/ds/query`; config `base_url` → `grafana_url` + `datasource_uid` + `grafana_api_key` | +| GM-102 | SC-101 (partial) | No direct Prom HTTP; all queries via gateway POST | +| GM-103 | SC-103 | Error semantics preserved (graceful `{error}`), but transport-specific failures (Grafana 401/403, datasource-not-found) are new | +| GM-104 | SC-102 | Shared helper now holds BOTH `normalize_prometheus_matrix` (kept for qBit) AND `normalize_grafana_frames` (new/restored); dedup shared | +| GM-105 | SC-104 | Step math reused but mapped to `intervalMs`/`maxDataPoints` instead of Prom `step` | +| GM-106 | SC-107 | Multi-series intent preserved; transport changes | +| GM-107 | SC-109, SC-111 | Gauge scalar-only preserved; instant query now via gateway | +| GM-108 | SC-112, SC-114 | Mean over window preserved; range query now via gateway | +| GM-109 | SC-105 (partial) | `metric` kind still bound to `prometheus`; transport changes | +| GM-110 | (new behavior for status) | `get_prometheus_status` now validates gateway path (previously direct Prom health) | +| GM-111 | SC-125 | Widget names change: `Prometheus*Widget` → `Metric*Widget` | +| GM-112 | SC-105 (partial) | Registry still binds kinds to `prometheus`; component refs rename | +| GM-113 | SC-126 (partial) | No auto-migration still holds; startup validation of old shape is new | +| GM-114 | SC-122 | CHANGELOG migration note changes from "delete grafana, recreate prometheus" to "reconfigure prometheus with gateway fields" | +| — | SC-115 | **MODIFIED**: "no grafana refs in backend" is no longer fully true — `normalize_grafana_frames` is a grafana-named function. The criterion becomes "no `grafana` SERVICE TYPE refs" (transport refs allowed). | +| — | SC-116 | **MODIFIED**: same as SC-115 for frontend — grafana transport is allowed; no grafana service type / link widget. | +| — | SC-117 | **PRESERVED** — no `grafana` service type is reintroduced. Still true. | +| — | SC-118 | **PRESERVED** — `get_grafana_status` stays removed; no grafana status surface. | +| — | SC-119 | **PRESERVED** — orphaned widgets still degrade gracefully. | +| — | SC-120 | **PRESERVED** — no grafana-specific tests reintroduced. | +| — | SC-121 | **MODIFIED**: `config.yaml` now references Grafana as the sanctioned transport (not "must not reference grafana"). | + +### PRESERVED (unchanged in canonical — no delta entry needed) + +SC-106 (chart renderer preserved), SC-108 (chart window preset), SC-110 (gauge threshold bands), SC-113 (mean plain PromQL), SC-117 (no grafana service type), SC-118 (no grafana status), SC-119 (orphan degrade), SC-120 (no grafana tests), SC-123 (backend tests pass), SC-124 (frontend build+lint), SC-127 (independent of service-storage-harness). + +### ADDS (new canonical requirements — `## ADDED Requirements` in delta) + +GM-110 (gateway-path status check — new behavior), GM-113 (startup old-config validation — new), and the canonical domain gains an explicit statement that Grafana is the sanctioned transport for the `prometheus` service. + +--- + +## Assumptions (where the proposal was silent and the spec settles) + +1. **Grafana instant-query mapping.** Grafana's `/api/ds/query` is inherently a range query. For the `gauge` (instant) and `metric` (instant) paths, the spec assumes the adapter issues a gateway query with a minimal window (e.g. `from=now-1m, to=now, maxDataPoints=1`) and extracts the single latest point, rather than attempting a separate Grafana instant-query endpoint. This mirrors how the pre-Change-A Grafana path handled chart data and is the pragmatic mapping. If Grafana exposes a cleaner instant path, the design phase may refine this — but the requirement (scalar result via gateway) holds either way. +2. **Version field in status.** Grafana's `/api/ds/query` response does not carry Prometheus build-info. `get_prometheus_status` returns `version: "ok"` (or omits it) on success. A richer version (via Grafana's datasource health endpoint) is a future-phase item, not this change. +3. **`normalize_prometheus_matrix` retention.** The matrix normalizer is kept (not deleted) because `QbittorrentSampleStore.window()` returns data in a matrix-adjacent shape consumed by `QbittorrentSpeedWidget`. Removing it would break the qBit speed widget. Both normalizers coexist. +4. **Old `api_key` secret.** The old optional `api_key` secret on the `prometheus` service is removed from the definition. Persisted rows may still carry it in their encrypted secrets blob; it is inert (the adapter reads `grafana_api_key` only) and gets dropped on the next save.