manage/openspec/changes/grafana-metric-gateway/proposal.md

SDD Proposal: Grafana Metric Gateway

Change: grafana-metric-gateway Phase: proposal Date: 2026-07-09

1. Problem / Why Now

New network constraint surfaced after prometheus-direct-charting shipped (2026-07-08): the Prometheus instance is not directly reachable from the Manage backend (firewalled / different network / no auth proxy). Grafana is the only reachable surface and already holds an authenticated, datasource-aware API. The direct-Prom path added by prometheus-direct-charting therefore cannot work in production — every chart/gauge/mean/metric widget call fails at the network layer.

Rather than expose Prometheus directly (infra change outside Manage's control), route metric queries through Grafana's existing /api/ds/query datasource proxy. Grafana becomes the transport; Prometheus remains the logical source.

This is a partial revert of prometheus-direct-charting, justified by new information that was not on the table when direct-Prom was chosen. It is not a fourth flip of preference — the network reality forces it.

2. Target Users and Situations

• Primary users: Operators whose Prometheus sits behind a firewall or on an unreachable network, with Grafana as the only authenticated entry point.
• Workflow moments: identical to today — open a dashboard, see metric charts/gauges/means. The only change is that the data flows Grafana→Prom instead of Manage→Prom directly.
• Urgency: High. The widgets added in Change A are currently non-functional in this network topology.

3. Product Outcome

After this change:

• Operators configure a prometheus service whose connection fields point at a Grafana gateway (URL + API key + datasource UID), not at Prometheus directly.
• The chart / gauge / mean / metric widget kinds behave exactly as today (same config, same rendering, same {series} shape) — only the transport changes.
• Widgets are renamed to neutral Metric* names (MetricChartWidget, MetricGaugeWidget, MetricMeanWidget) so the component names survive future source changes and don't mislead (the widgets talk to Grafana now, not Prometheus directly).
• The prometheus service type remains the sole metric surface — no grafana service type is reintroduced.

4. Scope Boundaries and Non-Goals

In scope

• prometheus service config schema changes: drop the direct-Prom base_url; add grafana_url (ServiceBaseUrl), datasource_uid (default "prometheus"), keep timeout_seconds.
• prometheus service secret schema changes: add grafana_api_key (encrypted, required).
• MetricSource adapter (rename of PrometheusWidgetSource): the chart / gauge / mean / metric dispatch paths issue POST {grafana_url}/api/ds/query with the API key + datasource UID + PromQL expr, and normalize Grafana's frames response into the existing {series} / {value} shapes.
• Frames→series normalization restored (recovered from git history at commit 65bae95, the pre-Change-A GrafanaWidgetSource._fetch_chart), refactored into the existing widgets/prometheus_range.py helper module alongside normalize_prometheus_matrix.
• Widget rename: PrometheusChartWidget → MetricChartWidget, PrometheusGaugeWidget → MetricGaugeWidget, PrometheusMeanWidget → MetricMeanWidget (via git mv, preserving history). Bindings in registry.ts move to the new names; the prometheus service's widget kinds stay chart/gauge/mean/metric (unchanged).
• Status check update: get_prometheus_status now runs a trivial query (up) through the Grafana gateway rather than hitting Prom /api/v1/status/buildinfo directly. This actually validates the full path (Grafana up + datasource reachable + Prom responding) — a stronger signal than the pre-change check.
• service-credential-tester proposal update: the Prom test routine in openspec/changes/service-credential-tester/proposal.md must be updated to test the Grafana gateway path (POST /api/ds/query with up), not direct Prom. Dependency, not a blocker.
• CHANGELOG migration note: existing prometheus service instances must be reconfigured (replace base_url with grafana_url + add grafana_api_key secret + optionally datasource_uid).

Non-goals (explicitly out of scope)

• Reintroducing the grafana service type. Grafana is the transport, not a first-class service. No grafana registry entry, no GrafanaLinkWidget, no LinksTab, no get_grafana_status. (Those were intentionally removed; this change does not restore them.)
• Grafana deep-link widgets (dashboard UID + panel ID links). Out of scope; the user explicitly wanted only the query gateway.
• A pluggable "data source middleware" framework. Per-widget-kind direct wiring only; no adapter registry abstraction. (Same discipline as Change A.)
• Touching the qBittorrent widgets, the storage harness, or MediaIndex. Unaffected.
• Renaming the prometheus service type itself. It stays prometheus (per operator decision — "Prometheus is the concept, Grafana is how I reach it").
• Exposing Prometheus directly as an alternative. Direct-Prom is removed, not kept-as-fallback. If a future topology can reach Prom directly, that's a separate change.
• Grafana status / health as a separate surface. Grafana reachability is folded into the Prom status check (it validates the gateway path).

5. High-Level Approach

5.1 Backend

1. integrations/prometheus.py — update PrometheusConfig:

   • Remove base_url.
   • Add grafana_url: ServiceBaseUrl (reuses the existing schema-enforced http(s) validator).
   • Add datasource_uid: str = "prometheus".
   • Keep timeout_seconds: int = 10.
   • Secret schema: replace any existing secret with grafana_api_key (required, encrypted).

2. widgets/prometheus_range.py — add normalize_grafana_frames(raw) -> list[series] alongside the existing normalize_prometheus_matrix. Recovered from 65bae95, refactored to share the label-dedup rule with the matrix normalizer.

3. widgets/sources.py — rename PrometheusWidgetSource → MetricSource (internal class; service-type key stays "prometheus" in SERVICE_ADAPTERS). Each of _fetch_chart, _fetch_gauge, _fetch_mean, and the instant-metric path switches from /api/v1/query[_range] to POST {grafana_url}/api/ds/query with body:

   {"queries": [{"datasource": {"uid": "<datasource_uid>", "type": "prometheus"},
                 "expr": "<promql>", "format": "time_series",
                 "intervalMs": <step_ms>, "maxDataPoints": <pts>, "refId": "A"}],
    "from": "<window_start>", "to": "now"}
   

   Then normalize via normalize_grafana_frames. Window presets (1h/6h/24h/7d) and step derivation (Change A's step_for_window) map to Grafana's from/intervalMs.

4. routers/monitoring.py — get_prometheus_status runs up through the gateway; success → {up: true, version: <grafana-prom-datasource-implication>}; failure → specific message (auth, unreachable, datasource not found).

5. Tests — update existing tests: the chart/gauge/mean adapter tests now assert a POST /api/ds/query call (mocked) instead of a direct Prom GET. Reuse the same {series} / {value} assertions.

5.2 Frontend

1. git mv widget files (history-preserving): PrometheusChartWidget.tsx → MetricChartWidget.tsx, PrometheusGaugeWidget.tsx → MetricGaugeWidget.tsx, PrometheusMeanWidget.tsx → MetricMeanWidget.tsx, plus their test files. Rename the exported components. The recharts/LineSeriesChart/gauge-band rendering is preserved unchanged.
2. integrations/registry.ts — the prometheus service binding keeps widget kinds chart/gauge/mean/metric but points at the renamed Metric* components. No user-visible change.
3. Service config form (ServiceConfigFields) — automatically reflects the new schema (grafana_url, datasource_uid) since it renders from the backend-provided config_schema. No bespoke UI work beyond relabeling the secret field.
4. Types — PrometheusStatus type stays; possibly gains a gateway field noting the path. Minimal change.

5.3 Canonical spec impact — first non-additive sync

The prometheus-charting canonical domain (openspec/specs/prometheus-charting/spec.md) was created by Change A with requirements stating direct Prom querying (SC-101, SC-102, SC-104, etc.). This change MODIFIES those requirements: the data source becomes Grafana's gateway, not direct Prom. This is the first non-additive canonical sync in the project — sdd-sync will need a ## MODIFIED Requirements delta rather than ## ADDED. Flagged for the sync phase; the requirements' intent (multi-series chart, gauge bands, mean over window) is unchanged, only the mechanism (transport) changes.

6. Success Criteria / Acceptance Criteria

1. A prometheus service configured with Grafana gateway fields (URL + API key + datasource UID) successfully powers chart/gauge/mean/metric widgets — no direct Prom network call is made.
2. The chart widget renders multi-series line charts with the same look as today (recharts via LineSeriesChart).
3. The gauge widget renders threshold bands; the mean widget renders the windowed mean — both sourced via Grafana.
4. The instant metric widget returns a scalar via Grafana.
5. get_prometheus_status validates the full gateway path; a broken Grafana or unreachable datasource produces a specific error message.
6. Widgets are renamed to MetricChartWidget / MetricGaugeWidget / MetricMeanWidget; no Prometheus*Widget names remain in frontend source; tests renamed consistently.
7. The prometheus-charting canonical domain's transport-related requirements are MODIFIED to reflect the Grafana gateway; durable requirements (chart/gauge/mean behavior) are preserved.
8. The service-credential-tester proposal's Prom test routine is updated to test the gateway path.
9. CHANGELOG documents the operator migration (reconfigure prometheus services with Grafana gateway fields).
10. pytest, npm run build, npm run lint, npm run test all green.

7. Risks and Mitigations

Risk	Mitigation
Reintroducing complexity that Change A removed. The frames→series normalization was deleted for a reason.	Recover the known-good implementation from 65bae95 rather than rewriting; refactor into prometheus_range.py so matrix + frames normalizers share the label-dedup rule.
Grafana /api/ds/query response shape varies across Grafana versions.	The recovered normalizer already handles the frames schema defensively (displayName → labels → field name fallback). Add a test fixture from a current Grafana version.
Existing prometheus service instances break silently (their base_url config no longer exists).	SC-9 CHANGELOG note; startup config validation rejects the old shape with a clear migration message.
First non-additive canonical sync — sdd-sync MODIFIED delta could mismatch.	Author the delta carefully; match requirement IDs exactly; verify with sdd-status before archive.
Credential-tester proposal dependency.	Update it in the same change (or sequence this change before the credential tester's apply).
Operators with a directly-reachable Prom lose that option.	Explicit non-goal; documented. If needed later, add a direct_url optional field as a separate change.
Review budget (>400 lines).	Likely 2 slices: (S1) backend config + adapter + frames normalizer + tests; (S2) frontend widget renames + registry + status check.

8. Resolved Questions (no question round needed)

• Q1 — Keep prometheus service type? Yes. It remains the user-facing logical service; only its config/secret schema and adapter internals change.
• Q2 — Restore grafana service type? No. Grafana is the transport, not a service. No deep-link widgets, no status surface.
• Q3 — Widget naming? Neutral Metric* (MetricChartWidget / MetricGaugeWidget / MetricMeanWidget). Decouples widget identity from transport; survives future source changes.
• Q4 — Direct Prom path kept as fallback? No. Removed. If a future topology needs it, separate change.
• Q5 — Status check approach? Run up through the gateway; validates the full path in one call.

9. Future Phases

1. Optional direct_url fallback on the prometheus service for topologies where Prom IS directly reachable (skip the Grafana hop).
2. Multi-datasource support — one Grafana gateway, many datasources (Loki, InfluxDB) selectable per widget.
3. Datasource health breakdown — status check distinguishes "Grafana down" vs "Prom datasource down" via Grafana's /api/datasources/health.

---

Notes for downstream phases

• Dependency: the service-credential-tester proposal must be updated when this change's design is firm (its Prom test routine changes from direct Prom to gateway query).
• Canonical first: this is the project's first non-additive canonical sync (prometheus-charting MODIFIED requirements). The sync phase should be deliberate.
• Recovery source: the frames→series normalizer is known-good at git commit 65bae95 (backend/src/media_library_viewer_api/widgets/sources.py, _fetch_chart). Restore from there, don't rewrite.
• Honest scope note: this partially reverts prometheus-direct-charting (2026-07-08). That change's direct-Prom path and Grafana removal are undone; its gauge/mean/LineSeriesChart/rendering work is preserved. The reversal is justified by the new network constraint (Q1, §1).