Files

T

Developer bf8de32815 spec(grafana-metric-gateway): add spec (16 reqs GM-101..116)

Gateway transport: prometheus service config gains grafana_url/api_key/
datasource_uid; all queries via POST /api/ds/query; frames->series restored.
First non-additive canonical sync: MODIFIES 17 SC- requirements (transport
changes, intent preserved), PRESERVES 11, ADDS 3 (gateway status, startup
validation, sanctioned-transport statement). 3 spec assumptions settled where
proposal was silent.

2026-07-09 20:49:45 +00:00

18 KiB

Raw Blame History

SDD Spec: Grafana Metric Gateway

Change: grafana-metric-gateway Phase: spec Date: 2026-07-09

This spec defines the acceptance requirements for the change. Requirements use the GM- prefix (the prometheus-charting canonical domain, created by prometheus-direct-charting, uses SC-; this change supersedes several of those — see §"Canonical delta intent").

1. Gateway transport (config + query path)

GM-101 — Prometheus service config points at a Grafana gateway

The prometheus service type's config model MUST declare grafana_url: ServiceBaseUrl (http(s)-schema-enforced), datasource_uid: str = "prometheus", and timeout_seconds: int = 10. The former base_url field MUST be removed. The secret schema MUST declare grafana_api_key (required, encrypted via Fernet) and MUST NOT carry the old optional api_key.

Scenario: Config schema reflects gateway fields

GIVEN the grafana-metric-gateway change is applied
WHEN integrations/prometheus.py PrometheusConfig is inspected
THEN it declares grafana_url, datasource_uid, timeout_seconds
AND it does NOT declare base_url
AND DEFINITION.secret_fields contains grafana_api_key (required=True)
AND DEFINITION.secret_fields does NOT contain api_key

GM-102 — All metric queries route through Grafana /api/ds/query

The widget source adapter (renamed MetricSource, service-type key stays "prometheus") MUST issue POST {grafana_url}/api/ds/query for all four widget kinds (chart, gauge, mean, metric). The request body MUST contain a queries array with {datasource: {uid: <datasource_uid>, type: "prometheus"}, expr: <promql>, format: "time_series", intervalMs: <step_ms>, maxDataPoints: <pts>, refId: "A"} plus from/to time bounds. The request MUST carry Authorization: Bearer {grafana_api_key}. The adapter MUST NOT issue any direct GET /api/v1/query or GET /api/v1/query_range call to a Prometheus URL.

Scenario: Chart fetch uses gateway POST

GIVEN a prometheus service configured with grafana_url, grafana_api_key, datasource_uid
WHEN a chart widget data fetch is executed (mocked HTTP)
THEN the adapter issues POST {grafana_url}/api/ds/query
AND the request body contains queries[0].datasource.uid == datasource_uid
AND the request body contains queries[0].expr == <promql>
AND the Authorization header is Bearer {grafana_api_key}
AND no GET /api/v1/query_range call is made

Scenario: No direct Prom HTTP call

GIVEN the change is applied
WHEN widgets/sources.py MetricSource is inspected
THEN no code path constructs a URL matching /api/v1/query or /api/v1/query_range against a Prometheus base_url

GM-103 — Gateway errors degrade gracefully

A Grafana timeout, connection error, HTTP 401/403 (auth), or non-2xx response MUST cause the widget data fetch to return { "error": str } (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional. The error message MUST be specific enough to distinguish auth failure from unreachable-host from datasource-not-found where the Grafana response permits.

Scenario: Auth failure surfaces a readable error

GIVEN a prometheus service with an invalid grafana_api_key
WHEN a widget data fetch is executed (mocked HTTP returns 401)
THEN the adapter returns { "error": <str mentioning auth or 401> }
AND no exception propagates

Scenario: Timeout surfaces a readable error

GIVEN a prometheus service whose Grafana gateway does not respond within the timeout
WHEN a widget data fetch is executed
THEN the adapter returns { "error": <str mentioning timeout> }

2. Frames → series normalization

GM-104 — Frames normalizer restored and shared

A normalize_grafana_frames(raw) -> list[dict] helper MUST exist in widgets/prometheus_range.py alongside the existing normalize_prometheus_matrix. It MUST parse Grafana's /api/v1/ds/query response (results.<refId>.frames[] with data.values + schema.fields) and produce the SAME {label, points:[{t:int, v:float|null}]} series shape the frontend chart renderer already consumes. The label-dedup rule (label (n) suffix on collision) MUST be shared with normalize_prometheus_matrix (extracted to a common inner helper, not duplicated).

Scenario: Frames normalized to series shape

GIVEN a sample Grafana /api/ds/query response with one frame containing timestamps [1000, 2000] and values [1.5, 2.5]
WHEN normalize_grafana_frames(raw) is called
THEN it returns [{"label": <str>, "points": [{"t": 1000, "v": 1.5}, {"t": 2000, "v": 2.5}]}]

Scenario: Label fallback chain

GIVEN a frame whose value-field has config.displayName set
WHEN normalized
THEN the label is the displayName
GIVEN a frame whose value-field has Prometheus labels but no displayName
WHEN normalized
THEN the label is the sorted k=v join (excluding __-prefixed keys)
GIVEN a frame with neither displayName nor labels
WHEN normalized
THEN the label is "value"

Scenario: Dedup shared with matrix path

GIVEN two frames that would produce the same label
WHEN normalized
THEN the second gets a (1) suffix
AND the dedup logic is not copy-pasted (a shared helper or shared suffix rule is used by both normalize_grafana_frames and normalize_prometheus_matrix)

3. Window presets + step mapping

GM-105 — Window presets map to Grafana from/to + intervalMs

The existing WINDOW_PRESETS (1h/6h/24h/7d → seconds) and step_for_window math MUST be reused to derive the gateway request's from (e.g. "now-1h"), to ("now"), and intervalMs (step * 1000). The resulting point count MUST land in the 100–300 band (same target as the pre-change direct-Prom path). Users do not configure from/to/step/intervalMs directly.

Scenario: 1h preset maps correctly

GIVEN a chart widget with window: "1h"
WHEN the gateway request body is constructed
THEN from resolves to approximately now - 3600s
AND to is "now"
AND intervalMs is step_for_window(3600) * 1000
AND the expected point count is ~200

GM-106 — Chart renders multi-series via gateway

The chart widget MUST query the gateway with a range query (PromQL expr + window preset) and return { "series": [...] } via normalize_grafana_frames. The frontend MetricChartWidget MUST render all returned series via LineSeriesChart, each as its own line with a distinct color. There is no single-series restriction.

Scenario: Multi-series chart through gateway

GIVEN a chart widget whose PromQL returns two series
WHEN data is fetched via the gateway (mocked)
THEN the adapter returns { "series": [<2 entries>] }
AND the frontend renders two distinct lines

GM-107 — Gauge renders instant scalar via gateway

The gauge widget MUST query the gateway with an instant PromQL query (mapped to a Grafana instant datasource query or a range query with a single point) and return { "value": float, "warn_at"?, "crit_at"?, "min"?, "max"?, "unit"? }. If the query returns multiple series, the adapter MUST return { "error": str } (scalar-only). The frontend MetricGaugeWidget MUST render threshold bands when warn_at/crit_at are set; otherwise a single neutral gauge.

Scenario: Gauge scalar-only through gateway

GIVEN a gauge widget whose PromQL returns one scalar
WHEN fetched via the gateway
THEN the adapter returns { "value": <float>, ... }
GIVEN a gauge widget whose PromQL returns multiple series
WHEN fetched
THEN the adapter returns { "error": <str> }

GM-108 — Mean computes over a gateway range window

The mean widget MUST query the gateway with a range query over the configured window preset, average all non-null point values of the single series client-side, and return { "value": float, "unit"? }. Scalar-only: multiple series → { "error": str }.

Scenario: Mean through gateway

GIVEN a mean widget with window: "1h" and a PromQL returning one series
WHEN fetched via the gateway
THEN the adapter returns { "value": <mean of non-null points> }

The instant metric widget MUST query the gateway and return { "result": <data> } (same shape as today) sourced via Grafana rather than direct Prom.

Scenario: Metric instant query through gateway

GIVEN a metric widget with a PromQL expression
WHEN fetched
THEN the adapter issues a gateway POST (not a direct Prom GET)
AND returns the scalar result in the existing { "result": ... } shape

5. Status check via gateway

GM-110 — get_prometheus_status validates the full gateway path

get_prometheus_status MUST run a trivial query (e.g. up) through the Grafana gateway. Success → { "up": true, "version": <str>, ... } (version derived from the successful path; if Grafana's response doesn't carry a Prom version, a sentinel like "ok" is acceptable). Failure → { "up": false, "error": <specific str> } distinguishing auth failure, unreachable host, and datasource-not-found where the response permits.

Scenario: Healthy gateway

GIVEN a prometheus service with valid gateway credentials
WHEN get_prometheus_status is called
THEN it issues a gateway POST with expr: "up"
AND returns { "up": true, ... }

Scenario: Auth failure in status

GIVEN a prometheus service with an invalid grafana_api_key
WHEN get_prometheus_status is called
THEN it returns { "up": false, "error": <str mentioning auth> }

PrometheusChartWidget.tsx → MetricChartWidget.tsx, PrometheusGaugeWidget.tsx → MetricGaugeWidget.tsx, PrometheusMeanWidget.tsx → MetricMeanWidget.tsx, plus their test files, MUST be renamed via git mv (history preserved). The exported component names MUST change to MetricChartWidget / MetricGaugeWidget / MetricMeanWidget. The rendering code (recharts, LineSeriesChart, gauge bands) MUST be preserved unchanged.

Scenario: No Prometheus*Widget names remain

GIVEN the change is applied
WHEN grep -r "PrometheusChartWidget\|PrometheusGaugeWidget\|PrometheusMeanWidget" frontend/src is run
THEN it returns no matches
AND MetricChartWidget.tsx, MetricGaugeWidget.tsx, MetricMeanWidget.tsx exist

GM-112 — Registry binds Metric* components

integrations/registry.ts MUST bind the prometheus service's chart/gauge/mean widget kinds to the renamed Metric* components. The widget KIND names (chart/gauge/mean/metric) MUST NOT change (they are persisted in widget instance rows).

Scenario: Registry uses Metric* components

GIVEN the change is applied
WHEN SERVICE_REGISTRY.prometheus.widgets is inspected
THEN the chart kind's component is MetricChartWidget
AND the kind strings are still "chart", "gauge", "mean", "metric"

7. Startup validation + migration

GM-113 — Startup rejects old prometheus config shape

When the backend starts and a persisted prometheus service instance has a config dict containing the old base_url key (without grafana_url), the startup config validation MUST log a clear migration message (naming the service) and either skip the instance gracefully or mark it as misconfigured — it MUST NOT crash startup.

Scenario: Old-shape config logged not crashed

GIVEN a persisted prometheus service with config: {"base_url": "http://prom:9090"} (no grafana_url)
WHEN the backend starts
THEN startup completes (no crash)
AND a log message names the service and instructs reconfiguration

GM-114 — CHANGELOG documents the migration

CHANGELOG.md [Unreleased] MUST include an entry stating that prometheus service instances must be reconfigured: replace base_url with grafana_url, add the grafana_api_key secret, and optionally set datasource_uid. It MUST be marked BREAKING.

Scenario: CHANGELOG entry present

GIVEN the change is applied
WHEN CHANGELOG.md is inspected
THEN an [Unreleased] entry mentions grafana_url, grafana_api_key, datasource_uid
AND it is marked BREAKING

8. Test + build greenness

GM-115 — Backend tests + ruff green

PYTHONPATH=src python3 -m pytest -q and PYTHONPATH=src python3 -m ruff check src tests from backend/ MUST pass. New/updated tests MUST cover: gateway POST assertion (mocked), normalize_grafana_frames (label fallback chain, dedup, NaN handling), gauge/mean scalar-only through gateway, status check through gateway, and the startup old-config validation.

GM-116 — Frontend build + lint + test green

npm run build, npm run lint, and npx vitest run from frontend/ MUST pass (0 errors). The renamed Metric* widget tests MUST cover loading, error, and rendered states.

Canonical delta intent

This is the project's first non-additive canonical sync. The prometheus-charting canonical domain (openspec/specs/prometheus-charting/spec.md) has 27 requirements (SC-101..SC-127). This change MODIFIES the transport-related requirements, PRESERVES the durable-behavior requirements, and ADDS new requirements. The sync phase (sdd-sync) MUST author a ## MODIFIED Requirements delta (not ## ADDED only).

MODIFIED (supersedes a canonical SC-xxx — intent may be preserved or changed)

GM-req	Supersedes	What changes
GM-101	SC-101 (partial)	Transport: direct Prom `/api/v1/query_range` → Grafana `/api/ds/query`; config `base_url` → `grafana_url` + `datasource_uid` + `grafana_api_key`
GM-102	SC-101 (partial)	No direct Prom HTTP; all queries via gateway POST
GM-103	SC-103	Error semantics preserved (graceful `{error}`), but transport-specific failures (Grafana 401/403, datasource-not-found) are new
GM-104	SC-102	Shared helper now holds BOTH `normalize_prometheus_matrix` (kept for qBit) AND `normalize_grafana_frames` (new/restored); dedup shared
GM-105	SC-104	Step math reused but mapped to `intervalMs`/`maxDataPoints` instead of Prom `step`
GM-106	SC-107	Multi-series intent preserved; transport changes
GM-107	SC-109, SC-111	Gauge scalar-only preserved; instant query now via gateway
GM-108	SC-112, SC-114	Mean over window preserved; range query now via gateway
GM-109	SC-105 (partial)	`metric` kind still bound to `prometheus`; transport changes
GM-110	(new behavior for status)	`get_prometheus_status` now validates gateway path (previously direct Prom health)
GM-111	SC-125	Widget names change: `PrometheusWidget` → `MetricWidget`
GM-112	SC-105 (partial)	Registry still binds kinds to `prometheus`; component refs rename
GM-113	SC-126 (partial)	No auto-migration still holds; startup validation of old shape is new
GM-114	SC-122	CHANGELOG migration note changes from "delete grafana, recreate prometheus" to "reconfigure prometheus with gateway fields"
—	SC-115	MODIFIED: "no grafana refs in backend" is no longer fully true — `normalize_grafana_frames` is a grafana-named function. The criterion becomes "no `grafana` SERVICE TYPE refs" (transport refs allowed).
—	SC-116	MODIFIED: same as SC-115 for frontend — grafana transport is allowed; no grafana service type / link widget.
—	SC-117	PRESERVED — no `grafana` service type is reintroduced. Still true.
—	SC-118	PRESERVED — `get_grafana_status` stays removed; no grafana status surface.
—	SC-119	PRESERVED — orphaned widgets still degrade gracefully.
—	SC-120	PRESERVED — no grafana-specific tests reintroduced.
—	SC-121	MODIFIED: `config.yaml` now references Grafana as the sanctioned transport (not "must not reference grafana").

PRESERVED (unchanged in canonical — no delta entry needed)

SC-106 (chart renderer preserved), SC-108 (chart window preset), SC-110 (gauge threshold bands), SC-113 (mean plain PromQL), SC-117 (no grafana service type), SC-118 (no grafana status), SC-119 (orphan degrade), SC-120 (no grafana tests), SC-123 (backend tests pass), SC-124 (frontend build+lint), SC-127 (independent of service-storage-harness).

ADDS (new canonical requirements — `## ADDED Requirements` in delta)

GM-110 (gateway-path status check — new behavior), GM-113 (startup old-config validation — new), and the canonical domain gains an explicit statement that Grafana is the sanctioned transport for the prometheus service.

Assumptions (where the proposal was silent and the spec settles)

Grafana instant-query mapping. Grafana's /api/ds/query is inherently a range query. For the gauge (instant) and metric (instant) paths, the spec assumes the adapter issues a gateway query with a minimal window (e.g. from=now-1m, to=now, maxDataPoints=1) and extracts the single latest point, rather than attempting a separate Grafana instant-query endpoint. This mirrors how the pre-Change-A Grafana path handled chart data and is the pragmatic mapping. If Grafana exposes a cleaner instant path, the design phase may refine this — but the requirement (scalar result via gateway) holds either way.
Version field in status. Grafana's /api/ds/query response does not carry Prometheus build-info. get_prometheus_status returns version: "ok" (or omits it) on success. A richer version (via Grafana's datasource health endpoint) is a future-phase item, not this change.
normalize_prometheus_matrix retention. The matrix normalizer is kept (not deleted) because QbittorrentSampleStore.window() returns data in a matrix-adjacent shape consumed by QbittorrentSpeedWidget. Removing it would break the qBit speed widget. Both normalizers coexist.
Old api_key secret. The old optional api_key secret on the prometheus service is removed from the definition. Persisted rows may still carry it in their encrypted secrets blob; it is inert (the adapter reads grafana_api_key only) and gets dropped on the next save.

18 KiB Raw Blame History Unescape Escape