manage/openspec/changes/grafana-metric-gateway/design.md

SDD Design: Grafana Metric Gateway

Change: grafana-metric-gateway Phase: design Date: 2026-07-09

0. Source findings (read before anything else)

The proposal and spec were written against a mental model. Reading actual source surfaced deviations the design must account for. Trust source, not assumptions.

Spec/proposal claim	Actual source reality	Design impact
Spec assumption #3: "normalize_prometheus_matrix retained because QbittorrentSampleStore.window() returns data in a matrix-adjacent shape consumed by QbittorrentSpeedWidget"	WRONG. QbittorrentWidgetSource.fetch (sources.py:395–425) builds {series} inline — {"label": "download", "points": [{"t": s["ts"] * 1000, "v": s["dl_speed"]}, ...]}. It NEVER calls normalize_prometheus_matrix. After this change removes all direct-Prom paths, normalize_prometheus_matrix becomes dead code.	Design recommends KEEPING normalize_prometheus_matrix (harmless, future-proof for direct_url future-phase) but flagging it as currently-unused. Do NOT delete — removing a tested helper adds risk for zero gain.
Proposal: "rename PrometheusWidgetSource → MetricSource"	Only 3 widgets are renamed (Chart/Gauge/Mean). The 4th kind, metric, uses PrometheusMetricWidget — which is NOT renamed per proposal §3 ("MetricChartWidget / MetricGaugeWidget / MetricMeanWidget").	PrometheusMetricWidget stays as-is. The internal MetricSource class handles all 4 kinds; the frontend component name for metric is unchanged.
monitoring.py helpers: _base_url, _auth_headers, _timeout	These read service.config.get("base_url") and service.secrets.get("api_key") — both will change (config: grafana_url; secret: grafana_api_key).	get_prometheus_status is rewritten to use gateway fields directly, not the shared _base_url/_auth_headers helpers (those still serve other service types). A local helper in the status function reads grafana_url/grafana_api_key/datasource_uid.
Proposal §5.1: "secret schema: replace any existing secret with grafana_api_key"	Current prometheus definition has SecretField(key="api_key", label="API key", helper="Optional bearer token") — it's optional, not required.	Replaced by SecretField(key="grafana_api_key", label="Grafana API key", required=True, helper="Service account token or API key for the Grafana gateway"). Old api_key is inert in persisted rows (dropped on next save per spec assumption #4).
PrometheusMetricWidget.tsx has complex formatting logic (formatPrometheusValue)	~63 lines, parses {resultType, result} vector/matrix shapes. It calls the backend metric kind which currently returns {"result": payload.get("data", {})} from _instant_query.	After this change, _instant_query becomes _gateway_query (POST /api/ds/query), and the response is normalized via normalize_grafana_frames then returned in a compatible shape. The frontend component stays unchanged; only the transport + response normalization changes. See §2.4 for the exact shape mapping.

No proposal/spec scope change is required — the intent (route through Grafana) still holds. The findings above refine implementation details.

---

1. Architecture overview

This change cuts the direct-Prometheus HTTP path and routes all metric queries through Grafana's /api/ds/query datasource proxy. Grafana becomes the transport; Prometheus remains the logical service type.

 BEFORE (prometheus-direct-charting)          AFTER (grafana-metric-gateway)
 ─────                                        ─────
 WidgetData fetch                             WidgetData fetch
   └► PrometheusWidgetSource.fetch              └► MetricSource.fetch
        └► GET prom:9090/api/v1/query_range          └► POST grafana:3000/api/ds/query
        └► GET prom:9090/api/v1/query                      {queries:[{datasource:{uid, type},
        └► normalize_prometheus_matrix()                     expr, intervalMs, maxDataPoints}],
                                                             from, to}
                                                        └► normalize_grafana_frames()
                                                        → {series} / {value} / {result}

Config changes from {base_url, timeout_seconds} + optional api_key to {grafana_url, datasource_uid, timeout_seconds} + required grafana_api_key. No frontend rendering change.

---

2. Backend design

2.1 PrometheusConfig model (GM-101)

File: backend/src/media_library_viewer_api/integrations/prometheus.py

class PrometheusConfig(ServiceConfigBase):
    """Non-secret Prometheus-via-Grafana gateway config."""
    grafana_url: ServiceBaseUrl
    datasource_uid: str = "prometheus"
    timeout_seconds: int = 10

Secret fields:

secret_fields=[
    SecretField(
        key="grafana_api_key",
        label="Grafana API key",
        required=True,
        helper="Service account token or API key for the Grafana gateway",
    ),
],

Widget config models (PrometheusMetricWidgetConfig, PrometheusChartWidgetConfig, PrometheusGaugeWidgetConfig, PrometheusMeanWidgetConfig) are UNCHANGED — they hold promql, window, warn_at, etc., which are transport-agnostic.

Note on naming: the widget config model classes keep Prometheus* names (they are Pydantic models internal to the definition module, not user-facing components). Only the frontend components rename to Metric*. This is intentional: the service IS Prometheus (the PromQL dialect, the datasource); Grafana is just how Manage reaches it.

2.2 normalize_grafana_frames helper (GM-104)

File: backend/src/media_library_viewer_api/widgets/prometheus_range.py (added alongside existing helpers)

Recovered from git commit 65bae95 (GrafanaWidgetSource._fetch_chart), refactored into a standalone function that shares the label-dedup rule with normalize_prometheus_matrix:

def normalize_grafana_frames(raw: dict[str, Any]) -> list[dict[str, Any]]:
    """Turn a Grafana /api/ds/query response into the {label, points} series shape.

    Parses results.<refId>.frames[] where each frame has:
      - data.values: [[timestamps...], [values...]]
      - schema.fields: [{name, labels?, config?: {displayName?}}, ...]

    Label rule (same as normalize_prometheus_matrix, shared via _dedup_label):
    1. Prefer config.displayName (explicitly set in Grafana).
    2. Else use Prometheus metric labels (sorted k=v, excluding __-prefixed).
    3. Else fall back to the field name, or "value".
    4. Dedup collisions with " (n)" suffix.
    """

Shared dedup extraction: The label-dedup logic (seen dict + (n) suffix) currently exists inline in both normalize_prometheus_matrix and the recovered _fetch_chart. The design extracts it into a private _dedup_label(label: str, seen: dict[str, int]) -> str helper used by both normalizers. This prevents duplication (GM-104 "dedup logic is not copy-pasted").

def _dedup_label(label: str, seen: dict[str, int]) -> str:
    """Apply ' (n)' suffix on collision. Mutates and reads from seen dict."""
    if label in seen:
        seen[label] += 1
        return f"{label} ({seen[label]})"
    seen[label] = 0
    return label

Both normalize_prometheus_matrix and normalize_grafana_frames call _dedup_label instead of inlining the logic.

2.3 MetricSource class — gateway query transport (GM-102)

File: backend/src/media_library_viewer_api/widgets/sources.py

PrometheusWidgetSource is renamed to MetricSource. The SERVICE_ADAPTERS dict key stays "prometheus" (GM-112 — kind strings unchanged).

The core new method is _gateway_query, which replaces both _range_query and _instant_query:

class MetricSource:
    """Run PromQL queries through a Grafana gateway (/api/ds/query)."""

    async def fetch(self, service, widget_kind, config):
        # dispatch chart/gauge/mean/metric → _fetch_chart/_fetch_gauge/_fetch_mean/_fetch_metric
        # each calls _gateway_query with appropriate window/maxDataPoints

    async def _gateway_query(
        self, grafana_url: str, api_key: str, datasource_uid: str,
        timeout: int, promql: str,
        window_seconds: int | None = None,
        max_data_points: int = 200,
    ) -> dict[str, Any]:
        """POST {grafana_url}/api/ds/query; return normalized result.

        - window_seconds=None → instant query (from=now-1m, to=now, maxDataPoints=1)
        - window_seconds=<N> → range query (from=now-Ns, to=now, step derived)
        """

Request body construction:

step = step_for_window(window_seconds) if window_seconds else 15
interval_ms = step * 1000

body = {
    "queries": [{
        "datasource": {"uid": datasource_uid, "type": "prometheus"},
        "expr": promql,
        "format": "time_series",
        "intervalMs": interval_ms,
        "maxDataPoints": 1 if window_seconds is None else max_data_points,
        "refId": "A",
    }],
    "from": f"now-{window_seconds or 60}s" if window_seconds else "now-1m",
    "to": "now",
}

Headers: {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}.

HTTP: requests.post(f"{grafana_url}/api/ds/query", json=body, headers=headers, timeout=timeout) wrapped in asyncio.wait_for(asyncio.to_thread(...), timeout=timeout).

Error handling (GM-103): catch asyncio.TimeoutError → {"error": "Grafana query timed out"}; catch requests.RequestException → {"error": f"Grafana query failed: {exc}"}. The outer fetch method's try/except catches everything else → {"error": ...}. Never raises.

Per-kind mapping (§2.3.1–§2.3.4):

2.3.1 _fetch_chart (GM-106)

async def _fetch_chart(self, grafana_url, api_key, datasource_uid, timeout, config):
    promql = config.get("promql")
    if not promql:
        return {"error": "promql is required"}
    window = WINDOW_PRESETS.get(config.get("window", "1h"), WINDOW_PRESETS["1h"])
    raw = await self._gateway_query(grafana_url, api_key, datasource_uid, timeout, promql, window_seconds=window)
    if "error" in raw:
        return raw
    return {"series": normalize_grafana_frames(raw)}

The _gateway_query return value on success is the raw Grafana JSON response dict. normalize_grafana_frames parses results.A.frames[] into [{label, points}].

2.3.2 _fetch_gauge (GM-107)

Instant query via window_seconds=None (minimal window, maxDataPoints=1). The last point of the single frame is the scalar:

async def _fetch_gauge(self, grafana_url, api_key, datasource_uid, timeout, config):
    raw = await self._gateway_query(grafana_url, api_key, datasource_uid, timeout,
                                     config.get("promql") or "", window_seconds=None)
    if "error" in raw:
        return raw
    series = normalize_grafana_frames(raw)
    if len(series) != 1:
        return {"error": "Gauge requires a single-series query; refine your PromQL"}
    points = series[0]["points"]
    if not points:
        return {"error": "Gauge query returned no scalar value"}
    value = points[-1]["v"]  # latest point
    if value is None:
        return {"error": "Gauge query returned no scalar value"}
    return {
        "value": value,
        "warn_at": config.get("warn_at"),
        "crit_at": config.get("crit_at"),
        "min": config.get("min"),
        "max": config.get("max"),
        "unit": config.get("unit"),
    }

2.3.3 _fetch_mean (GM-108)

Range query over the window preset; average all non-null points of the single series:

async def _fetch_mean(self, grafana_url, api_key, datasource_uid, timeout, config):
    promql = config.get("promql")
    if not promql:
        return {"error": "promql is required"}
    window = WINDOW_PRESETS.get(config.get("window", "1h"), WINDOW_PRESETS["1h"])
    raw = await self._gateway_query(grafana_url, api_key, datasource_uid, timeout, promql, window_seconds=window)
    if "error" in raw:
        return raw
    series = normalize_grafana_frames(raw)
    if len(series) != 1:
        return {"error": "Mean requires a single-series query; refine your PromQL"}
    nums = [p["v"] for p in series[0]["points"] if p["v"] is not None]
    if not nums:
        return {"error": "Mean query returned no numeric samples in the window"}
    return {"value": sum(nums) / len(nums), "unit": config.get("unit")}

2.3.4 _fetch_metric (GM-109)

Instant query; return result in a shape compatible with PrometheusMetricWidget's expectations. The frontend component currently reads data.result.result (a vector/matrix). The gateway response normalized via normalize_grafana_frames produces [{label, points}] — the frontend component must handle this shape. Design decision: return the normalized series as-is in {"result": series} so the frontend component adapts to read data.result as an array of {label, points} (matching the chart shape). This is a small frontend adaptation in the component (see §3.3). If the frontend change proves too large for Slice 2, the fallback is to return the raw Grafana response structure and let the frontend parse it — but the normalized shape is preferred for consistency.

2.4 get_prometheus_status via gateway (GM-110)

File: backend/src/media_library_viewer_api/routers/monitoring.py

The current get_prometheus_status does two direct-Prom HTTP calls (/-/healthy + /api/v1/status/buildinfo). These are replaced by a single gateway query:

def get_prometheus_status(service_id, store):
    service = resolve_service_record(store, "prometheus", service_id)
    if service is None:
        return _status_response(None, error="no_service_configured")

    grafana_url = str(service.config.get("grafana_url") or "").rstrip("/")
    api_key = str(service.secrets.get("grafana_api_key") or "")
    datasource_uid = str(service.config.get("datasource_uid") or "prometheus")
    timeout = int(service.config.get("timeout_seconds") or 10)

    if not grafana_url or not api_key:
        return _status_response(service, error="gateway_not_configured")

    try:
        body = {  # trivial 'up' query
            "queries": [{"datasource": {"uid": datasource_uid, "type": "prometheus"},
                         "expr": "up", "format": "time_series",
                         "intervalMs": 15000, "maxDataPoints": 1, "refId": "A"}],
            "from": "now-1m", "to": "now",
        }
        resp = requests.post(f"{grafana_url}/api/ds/query", json=body,
                             headers={"Authorization": f"Bearer {api_key}",
                                       "Content-Type": "application/json"},
                             timeout=timeout)
        resp.raise_for_status()
    except requests.HTTPError as exc:
        status_code = exc.response.status_code if exc.response else 0
        if status_code in (401, 403):
            return _status_response(service, error="auth_failed")
        return _status_response(service, error="gateway_error")
    except requests.RequestException:
        return _status_response(service, error="prometheus_unreachable")

    return _status_response(service, version="ok")

Error mapping: 401/403 → "auth_failed"; connection error → "prometheus_unreachable"; other non-2xx → "gateway_error". Success → version="ok" (spec assumption #2: Grafana /api/ds/query doesn't carry Prom build-info).

The shared _base_url/_auth_headers/_timeout helpers in monitoring.py are NOT modified — they still serve get_alertmanager_status which reads base_url/api_key from alertmanager services. The Prom status function reads gateway fields directly.

2.5 Startup validation for old config shape (GM-113)

File: backend/src/media_library_viewer_api/main.py (lifespan, or a small validation helper)

The existing validate_auth_settings(settings) runs on startup. A complementary check iterates persisted prometheus service instances and detects old-shape config:

def _validate_prometheus_gateway_config(store: SettingsStore) -> None:
    """Warn (not crash) about old-shape prometheus services that need migration."""
    for service in store.list_services("prometheus"):
        config = service.get("config") or {}
        if "base_url" in config and "grafana_url" not in config:
            logger.warning(
                "Prometheus service '%s' (id=%s) uses the old 'base_url' config shape. "
                "Reconfigure with grafana_url + grafana_api_key (see CHANGELOG).",
                service.get("name"), service.get("id"),
            )

Called from lifespan after ensure_defaults(). This logs a warning but does NOT crash, disable, or delete the service (GM-113: "MUST NOT crash startup"). The service's widget fetches will fail at runtime with a gateway error (since grafana_url is absent) — that's the expected degraded state until the operator reconfigures.

2.6 Imports cleanup in sources.py

The PrometheusWidgetSource → MetricSource rename changes the class referenced in SERVICE_ADAPTERS:

SERVICE_ADAPTERS: dict[str, WidgetSource] = {
    "prometheus": MetricSource(),   # was PrometheusWidgetSource()
    ...
}

The import block gains normalize_grafana_frames:

from media_library_viewer_api.widgets.prometheus_range import (
    WINDOW_PRESETS,
    normalize_grafana_frames,
    normalize_prometheus_matrix,   # kept (currently dead code after this change; future-proof)
    step_for_window,
)

---

3. Frontend design

3.1 Widget rename via git mv (GM-111)

Old path	New path
frontend/src/widgets/PrometheusChartWidget.tsx	frontend/src/widgets/MetricChartWidget.tsx
frontend/src/widgets/PrometheusGaugeWidget.tsx	frontend/src/widgets/MetricGaugeWidget.tsx
frontend/src/widgets/PrometheusMeanWidget.tsx	frontend/src/widgets/MetricMeanWidget.tsx
frontend/src/widgets/__tests__/PrometheusChartWidget.test.tsx	frontend/src/widgets/__tests__/MetricChartWidget.test.tsx
frontend/src/widgets/__tests__/PrometheusGaugeWidget.test.tsx	frontend/src/widgets/__tests__/MetricGaugeWidget.test.tsx
frontend/src/widgets/__tests__/PrometheusMeanWidget.test.tsx	frontend/src/widgets/__tests__/MetricMeanWidget.test.tsx

NOT renamed: PrometheusMetricWidget.tsx (the metric kind widget). The proposal only names Chart/Gauge/Mean for the Metric* rename. PrometheusMetricWidget keeps its name.

Per-file changes inside the renamed files:

• Exported function name: PrometheusChartWidget → MetricChartWidget (and Gauge/Mean equivalents).
• All rendering code (recharts, LineSeriesChart, gauge bands, formatValue, formatMean) is preserved unchanged.
• The test files update their import + the component reference.

3.2 Registry binding update (GM-112)

File: frontend/src/integrations/registry.ts

Imports change:

import { MetricChartWidget } from "../widgets/MetricChartWidget";
import { MetricGaugeWidget } from "../widgets/MetricGaugeWidget";
import { MetricMeanWidget } from "../widgets/MetricMeanWidget";
// PrometheusMetricWidget import stays

Widget KIND strings stay "chart", "gauge", "mean", "metric" (persisted in widget instance rows — must not change).

Component refs change:

component: MetricChartWidget,   // was PrometheusChartWidget (chart kind)
component: MetricGaugeWidget,   // was PrometheusGaugeWidget (gauge kind)
component: MetricMeanWidget,    // was PrometheusMeanWidget (mean kind)
// metric kind: PrometheusMetricWidget (unchanged)

3.3 Barrel export update

File: frontend/src/widgets/index.ts

export { MetricChartWidget } from "./MetricChartWidget";
export { MetricGaugeWidget } from "./MetricGaugeWidget";
export { MetricMeanWidget } from "./MetricMeanWidget";
// PrometheusMetricWidget export stays

3.4 PrometheusMetricWidget adaptation (if needed)

PrometheusMetricWidget.tsx currently reads data.result as a Prometheus {resultType, result: [{metric, value}]} shape. After the transport change, the backend returns the normalized Grafana series for the metric kind too. Two options:

• Option A (preferred): Backend _fetch_metric returns {"result": normalize_grafana_frames(raw)} (array of {label, points}), and the frontend component is adapted to render from this shape. Small change — the component's rendering logic switches from parsing Prom vector to rendering the last point of each series.
• Option B (fallback): Backend _fetch_metric returns the raw Grafana response in a Prom-compatible shape, minimizing frontend change.

Design recommends Option A for shape consistency. The component's formatting logic (formatPrometheusValue) is reused for the scalar value extraction.

---

4. Test changes

4.1 Backend tests

File: backend/tests/test_widgets.py

• Existing chart/gauge/mean adapter tests assert GET /api/v1/query_range — rewritten to assert POST /api/ds/query. The mock response changes from Prom matrix shape to Grafana frames shape. Assertions on the {series} / {value} output shape stay the same.
• The ServiceRecord mock in tests changes from {config: {base_url, timeout_seconds}, secrets: {api_key}} to {config: {grafana_url, datasource_uid, timeout_seconds}, secrets: {grafana_api_key}}.

File: backend/tests/test_prometheus_range.py

• New tests for normalize_grafana_frames: label fallback chain (displayName → labels → "value"), dedup, NaN handling, empty frames.
• Existing normalize_prometheus_matrix tests stay (dead code but still tested).

File: backend/tests/test_api.py

• get_prometheus_status test: assert POST /api/ds/query instead of GET /-/healthy + GET /api/v1/status/buildinfo.

File: backend/tests/test_services.py

• Prometheus service config test: assert grafana_url field (not base_url); assert grafana_api_key secret field.

4.2 Frontend tests

• Renamed test files follow the git mv and assert the new component names. Test cases (loading/error/rendered) are preserved.
• If PrometheusMetricWidget is adapted (§3.4 Option A), its test updates the mock data shape.

---

5. Slice plan

Slice 1 — Backend: config + adapter + frames normalizer + status + tests (~300–380 lines)

• integrations/prometheus.py: PrometheusConfig change (drop base_url, add grafana_url/datasource_uid; secret grafana_api_key).
• widgets/prometheus_range.py: add normalize_grafana_frames + _dedup_label shared helper.
• widgets/sources.py: rename PrometheusWidgetSource → MetricSource; add _gateway_query; rewrite _fetch_chart/_fetch_gauge/_fetch_mean/_fetch_metric; update SERVICE_ADAPTERS.
• routers/monitoring.py: rewrite get_prometheus_status for gateway path.
• main.py: add _validate_prometheus_gateway_config startup check.
• CHANGELOG.md: migration note.
• Tests: update all affected backend tests (test_widgets, test_prometheus_range, test_api, test_services).

Slice 2 — Frontend: widget renames + registry + barrel (~200–280 lines)

• git mv 3 component files + 3 test files (6 renames).
• Rename exported functions inside each file.
• registry.ts: update imports + component refs.
• widgets/index.ts: update barrel exports.
• If needed: adapt PrometheusMetricWidget.tsx for normalized series shape.
• Tests: verify renamed tests pass.

Each slice ≤400 lines. S1 → S2 order. S1 is independently shippable (backend works, frontend just has stale names until S2).

---

6. Key design decisions summary

#	Decision	Rationale
1	normalize_grafana_frames recovered from 65bae95, not rewritten	Known-good code; avoids re-deriving the Grafana frames schema
2	Shared _dedup_label extracted from both normalizers	GM-104 "dedup not copy-pasted"; DRY
3	_gateway_query is the single transport method (replaces _range_query + _instant_query)	One POST path for all kinds; window_seconds=None signals instant
4	Instant-query mapping: from=now-1m, to=now, maxDataPoints=1 (spec assumption #1)	Grafana /api/ds/query has no native instant endpoint; minimal-window is pragmatic
5	normalize_prometheus_matrix kept (dead code after this change)	Harmless; future-proof for direct_url future-phase; removing tested code adds risk
6	PrometheusMetricWidget NOT renamed (only Chart/Gauge/Mean → Metric*)	Proposal only names 3 widgets; metric kind stays PrometheusMetricWidget
7	Status check returns version="ok" (spec assumption #2)	Grafana /api/ds/query lacks Prom build-info; richer version is future-phase
8	Startup validation logs warning, doesn't crash (GM-113)	Operator needs to migrate at their pace; degraded widgets show gateway error

---

7. Risk assessment

Risk	Likelihood	Impact	Mitigation
Grafana /api/ds/query response shape varies across versions	Medium	Medium	Recovered normalizer handles shapes defensively; add a test fixture from current Grafana
Old-shape services cause confusing widget errors	Medium	Low	Startup warning + gateway error message names the service
normalize_prometheus_matrix dead code confuses future maintainers	Low	Low	Inline comment explains why it's kept; it's in a tested helper module
PrometheusMetricWidget shape mismatch after transport change	Medium	Medium	§3.4 Option A adaptation; if too large, Option B fallback (raw response)
Canonical sync MODIFIED delta mismatches SC- IDs	Low	Medium	Spec §"Canonical delta intent" maps every GM→SC; sync phase references it

---

8. File change inventory

Backend (Slice 1)

File	Change
integrations/prometheus.py	PrometheusConfig: drop base_url, add grafana_url/datasource_uid; secret grafana_api_key
widgets/prometheus_range.py	Add normalize_grafana_frames + _dedup_label shared helper
widgets/sources.py	Rename class → MetricSource; add _gateway_query; rewrite 4 _fetch_* methods
routers/monitoring.py	Rewrite get_prometheus_status for gateway path
main.py	Add _validate_prometheus_gateway_config startup check
CHANGELOG.md	BREAKING migration note
tests/test_widgets.py	Update chart/gauge/mean tests for gateway POST + frames mock
tests/test_prometheus_range.py	Add normalize_grafana_frames tests
tests/test_api.py	Update get_prometheus_status test
tests/test_services.py	Update prometheus config schema test

Frontend (Slice 2)

File	Change
widgets/PrometheusChartWidget.tsx → MetricChartWidget.tsx	git mv + rename export
widgets/PrometheusGaugeWidget.tsx → MetricGaugeWidget.tsx	git mv + rename export
widgets/PrometheusMeanWidget.tsx → MetricMeanWidget.tsx	git mv + rename export
widgets/__tests__/Prometheus*Widget.test.tsx → Metric*Widget.test.tsx	git mv (3 files) + update imports
widgets/index.ts	Update 3 barrel exports
integrations/registry.ts	Update 3 imports + 3 component refs
widgets/PrometheusMetricWidget.tsx	Adapt for normalized series shape (if §3.4 Option A)