From cb8dd13514cbe0791606bbc345f421fc36ef468f Mon Sep 17 00:00:00 2001 From: Developer Date: Thu, 9 Jul 2026 22:01:55 +0000 Subject: [PATCH] =?UTF-8?q?spec(grafana-metric-gateway):=20sync=20?= =?UTF-8?q?=E2=80=94=20FIRST=20non-additive=20(MODIFIED)=20canonical?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 16 MODIFIED prometheus-charting requirements (transport: direct Prom -> Grafana gateway; intent preserved where applicable), 3 ADDED (SC-128 gateway status, SC-129 startup validation, SC-130 sanctioned transport), 11 PRESERVED, 0 REMOVED. All 16 MODIFIED headers matched canonical exactly. Post-sync: 30 requirements. web-ui + service-storage canonicals untouched. --- .../specs/prometheus-charting/spec.md | 106 ++++++++++ .../grafana-metric-gateway/sync-report.md | 189 ++++++++++++++++++ openspec/specs/prometheus-charting/spec.md | 76 ++++--- 3 files changed, 343 insertions(+), 28 deletions(-) create mode 100644 openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md create mode 100644 openspec/changes/grafana-metric-gateway/sync-report.md diff --git a/openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md b/openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md new file mode 100644 index 0000000..3697c07 --- /dev/null +++ b/openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md @@ -0,0 +1,106 @@ +# Prometheus Charting — Delta (`grafana-metric-gateway`) + +> Change: `grafana-metric-gateway` · Domain: `prometheus-charting` · Phase: **spec** (authored during `sdd-sync`). +> +> **This is the project's first non-additive canonical sync.** The `prometheus-charting` canonical +> domain (SC-101 … SC-127, created by `prometheus-direct-charting`) described the *direct* Prometheus +> transport. This change switches the `prometheus` service to a **Grafana gateway** transport +> (`POST {grafana_url}/api/ds/query`), renames the chart/gauge/mean widgets to neutral `Metric*` +> names, and validates the gateway path. Durable *behavior* requirements (multi-series chart, gauge +> threshold bands, windowed mean) are **preserved**; only the *transport* and a few naming/status +> statements are modified. The mapping of every change requirement (GM-101 … GM-116) to a canonical +> SC-xxx is recorded in the flat `spec.md` §"Canonical delta intent". +> +> This delta therefore contains **both** a `## MODIFIED Requirements` block (full replacement bodies +> for each superseded SC-xxx — matched by exact requirement ID) and an `## ADDED Requirements` block +> (genuinely new requirements assigned fresh canonical IDs SC-128 … SC-130). PRESERVED requirements +> (SC-106, SC-108, SC-110, SC-113, SC-117 … SC-120, SC-123, SC-124, SC-127) appear in **neither** +> section — they are unchanged in the canonical spec. + +## MODIFIED Requirements + +> Each block below replaces the full canonical requirement of the same ID (matched by the exact +> `### Requirement: SC-xxx — ` header). The requirement *intent* is preserved where the change +> only swaps the mechanism (direct Prom → Grafana gateway); the body text is updated to the gateway +> transport. No requirement IDs are renamed (RENAMED is unsupported by the native delta helper). + +### Requirement: SC-101 — Prometheus range query returns the existing series shape + +The `prometheus` service type's config MUST point at a Grafana gateway — declaring `grafana_url` (http(s)-schema-enforced), `datasource_uid` (default `"prometheus"`), and `timeout_seconds` — with a required, Fernet-encrypted `grafana_api_key` secret. The former direct-Prometheus `base_url` config field and the old optional `api_key` secret MUST be gone. When a `prometheus` widget of kind `chart` is fetched, the backend MUST issue `POST {grafana_url}/api/ds/query` (carrying `Authorization: Bearer {grafana_api_key}` and a `queries[0]` entry keyed by `datasource_uid`), MUST NOT issue any direct `GET /api/v1/query_range` against a Prometheus URL, and MUST return a payload of shape `{ "series": [{ "label": str, "points": [{ "t": int, "v": float|null }] }] }` — the exact shape the frontend chart renderer already consumes. + +### Requirement: SC-102 — Series label normalization is shared and Prometheus-native + +The metric-label → readable-label normalization MUST live in a single shared helper module that holds **both** normalizers — the retained Prom-matrix normalizer (`normalize_prometheus_matrix`) and the (re-)added Grafana-frames normalizer (`normalize_grafana_frames`) — and MUST deduplicate repeated labels via a `label (n)` suffix using **one shared dedup rule** (not duplicated across the two normalizers). Both normalizers produce the same `{label, points}` series shape, and the frames normalizer MUST apply the label fallback chain (displayName → sorted Prometheus labels excluding `__`-prefixed keys → `"value"`). + +### Requirement: SC-103 — Range query errors degrade gracefully + +A Grafana gateway timeout, connection error, HTTP 401/403 (auth), datasource-not-found, or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional. The error message MUST be specific enough to distinguish auth failure from unreachable-host from datasource-not-found where the Grafana response permits. + +### Requirement: SC-104 — Step is derived from the window preset + +Given a window preset (1h / 6h / 24h / 7d), the backend MUST reuse the existing `WINDOW_PRESETS` and `step_for_window` math to derive the gateway request's `intervalMs` (`step * 1000`), `maxDataPoints`, and `from`/`to` time bounds, landing the resulting point count in the same ~100–300 band as the pre-change direct-Prom path. Users do not configure `from`/`to`/`step`/`intervalMs` directly. + +### Requirement: SC-105 — Chart widget moves from grafana to prometheus + +Widget kinds `chart`, `gauge`, `mean`, and `metric` MUST remain bound to the `prometheus` service type — the kind strings are persisted in widget instance rows and MUST NOT change. The `grafana` service type MUST NOT be reintroduced and MUST NOT offer any widget kind. The backend adapter (internal class `MetricSource`, registry key still `"prometheus"`) and the frontend `SERVICE_REGISTRY` bind these kinds to the neutral `Metric*` components for `chart`/`gauge`/`mean` (and `PrometheusMetricWidget` for `metric`); all four kinds source their data via the Grafana gateway. + +### Requirement: SC-107 — Chart supports multiple series + +The `chart` widget MUST render all series returned by the gateway range query, each as its own line with a distinct color (via `normalize_grafana_frames`). There is no single-series restriction on `chart`. + +### Requirement: SC-109 — Gauge renders an instant scalar + +A widget kind named `gauge` MUST be bound to the `prometheus` service. Its data fetch MUST run an instant PromQL query through the Grafana gateway (a gateway query with a minimal window / single data point) and return the scalar result for rendering as a gauge. + +### Requirement: SC-111 — Gauge is scalar-only + +The `gauge` widget MUST render exactly one scalar value. If the gateway query returns multiple series, the adapter MUST return `{ "error": str }` (not silently pick one), directing the user to refine the PromQL. + +### Requirement: SC-112 — Mean computes client-side over a window + +A widget kind named `mean` MUST be bound to the `prometheus` service. Its data fetch MUST run a range query through the Grafana gateway over the configured window preset and return the arithmetic mean of all non-null point values of the single series as a single scalar. + +### Requirement: SC-114 — Mean is scalar-only + +The `mean` widget MUST render exactly one scalar value. If the gateway range query returns multiple series, the adapter MUST return `{ "error": str }` (not silently aggregate across series). + +### Requirement: SC-115 — No grafana references in backend source + +Grafana is the sanctioned **transport** for the `prometheus` service, so backend source legitimately contains Grafana transport references (e.g. `grafana_url`, `grafana_api_key`, `normalize_grafana_frames`, `POST …/api/ds/query`). After the change, those references are transport-only: there MUST be no `grafana` **service type**, no `grafana` service definition, and no `grafana` entry in the backend `SERVICE_DEFINITIONS` / `SERVICE_ADAPTERS` registries. The `integrations/grafana.py` service-type module stays deleted. + +### Requirement: SC-116 — No grafana references in frontend source + +Grafana transport is allowed in the frontend — the `prometheus` service config form renders Grafana gateway fields (`grafana_url`, `datasource_uid`) from the backend-provided schema. No frontend code MUST reintroduce a `grafana` **service type**, a Grafana deep-link / dashboard widget, `LinksTab`, or any Grafana-status UI surface. (Test fixtures where "Grafana" appears as a user-authored dashboard *shortcut label* unrelated to a service type remain excluded.) + +### Requirement: SC-121 — config.yaml matches implementation + +`openspec/config.yaml` MUST reflect that metric queries for the `prometheus` service are routed through a Grafana gateway (`POST /api/ds/query`) and that in-app charting via `recharts` is the sanctioned renderer. It MUST NOT carry the stale claims "Do NOT re-implement charting in-app", "No recharts/d3 is in use", or "Prometheus is queried directly (no Grafana middleman)". + +### Requirement: SC-122 — CHANGELOG documents the migration + +`CHANGELOG.md` `[Unreleased]` MUST include a **BREAKING** entry stating that existing `prometheus` service instances must be reconfigured: replace the old `base_url` with `grafana_url`, add the `grafana_api_key` secret, and optionally set `datasource_uid`, because metric queries now route through Grafana's `POST /api/ds/query`. + +### Requirement: SC-125 — New widget kinds have tests + +The renamed widgets `MetricChartWidget`, `MetricGaugeWidget`, and `MetricMeanWidget` MUST each have a frontend test covering at least: loading state, error state, and a rendered data case. (The `metric`-kind widget `PrometheusMetricWidget` is not renamed but remains tested.) + +### Requirement: SC-126 — No silent data migration + +The change MUST NOT auto-migrate persisted `prometheus` service instances from the old `base_url` config shape to the new `grafana_url` gateway shape (URLs and credentials differ; a safe automated rewrite is not possible). Migration is operator-driven per the CHANGELOG note; old-shape instances are flagged on startup (see SC-129) and degrade at runtime to a gateway error until reconfigured. + +## ADDED Requirements + +> Genuinely new requirements with fresh canonical IDs continuing the SC- sequence. The `GM-` change +> prefixes are intentionally not reused in the canonical store. + +### Requirement: SC-128 — get_prometheus_status validates the full gateway path + +`get_prometheus_status` MUST validate the full metric path by running a trivial query (e.g. `up`) through the Grafana gateway (`POST {grafana_url}/api/ds/query`). Success MUST return `{ "up": true, "version": , … }` — a sentinel such as `"ok"` is acceptable when Grafana's `/api/ds/query` response carries no Prometheus build-info. Failure MUST return `{ "up": false, "error": }` distinguishing auth failure (HTTP 401/403), unreachable host, and datasource/gateway error where the response permits. + +### Requirement: SC-129 — Startup logs old-shape prometheus config without crashing + +When the backend starts and a persisted `prometheus` service instance carries the old `base_url` config key without `grafana_url`, startup MUST log a clear migration message naming the service and instructing reconfiguration, and MUST complete without crashing (it MUST NOT delete, disable, or otherwise mutate the instance). The instance's widget fetches degrade to a gateway error until the operator reconfigures it. + +### Requirement: SC-130 — Grafana is the sanctioned transport for the prometheus service + +The `prometheus` service type is the sole metric surface in Manage. Its metric queries (`chart`, `gauge`, `mean`, `metric`) are sanctioned to route through a Grafana gateway — `POST {grafana_url}/api/ds/query` with a `grafana_api_key` and a Prometheus `datasource_uid`. Grafana is the transport, not a first-class service type; a direct-Prometheus HTTP path is not supported. diff --git a/openspec/changes/grafana-metric-gateway/sync-report.md b/openspec/changes/grafana-metric-gateway/sync-report.md new file mode 100644 index 0000000..9cfdc3d --- /dev/null +++ b/openspec/changes/grafana-metric-gateway/sync-report.md @@ -0,0 +1,189 @@ +# Sync Report — `grafana-metric-gateway` + +> Phase: **sync** · Change: `grafana-metric-gateway` · Repo: `/home/user/manage` +> Mode: file-backed (`artifactStore: openspec`). No source-code edits; only OpenSpec artifacts were +> written. Not committed (parent owns the commit). The change folder was **not** moved (that is +> `sdd-archive`'s job). + +**Status: SYNCED.** The first non-additive canonical sync in the project. The `prometheus-charting` +canonical domain's transport requirements were **MODIFIED** in place (direct Prom → Grafana gateway), +durable behavior requirements were **PRESERVED** verbatim, and three genuinely new requirements were +**ADDED** (SC-128 … SC-130). + +--- + +## 1. Executive summary + +The `prometheus-charting` canonical domain (`openspec/specs/prometheus-charting/spec.md`) was created +by `prometheus-direct-charting` with 27 requirements (SC-101 … SC-127) describing the **direct** +Prometheus transport. The `grafana-metric-gateway` change switches the `prometheus` service to a +**Grafana gateway** transport (`POST {grafana_url}/api/ds/query`), renames the chart/gauge/mean +widgets to neutral `Metric*` components, and validates the gateway path — while preserving the durable +*behavior* contracts (multi-series chart, gauge threshold bands, windowed mean, scalar-only rules). + +This sync authored a change-side **delta** spec with both `## MODIFIED Requirements` and +`## ADDED Requirements` (the project's first delta to use MODIFIED), then applied it to the canonical +spec by exact-ID block replacement. No requirements were REMOVED or RENAMED. Verify returned PASS +(verdict in `verify-report.md`: 15 PASS + GM-115 PARTIAL-on-coverage-only, treated as effectively +16/16 per the dispatch brief; `apply-progress.md` exists and reconciles the 33 task checkboxes the +verify report flagged). All five quality gates were green at the verified commits (`df80c68`, `7e91e7f`). + +## 2. Structured status & actionContext findings + +The native `gentle-pi.sdd-status` passed by the parent reports `changeName: null` / +`blockedReasons: ["Change selection is ambiguous: grafana-metric-gateway, per-instance-hook-scoping, +service-credential-tester."]` because the engine auto-detected three active changes. This sync task +was **explicitly assigned** `grafana-metric-gateway`; the ambiguity is a dispatch artifact and does +not block this phase (`isNonAuthoritative: false`). + +- `artifactStore: openspec`; change root `openspec/changes/grafana-metric-gateway/`. +- Artifacts present: `proposal.md`, `spec.md` (flat, with §"Canonical delta intent" mapping), + `design.md`, `tasks.md`, `verify-report.md`, `apply-progress.md`. +- `verify: PASS` (verify-report verdict; gates green). The verify report's single CRITICAL was an + **archive** blocker (33 unchecked task checkboxes + missing `apply-progress.md`); `apply-progress.md` + now exists, and that condition does **not** block `sdd-sync` of the green code. +- `actionContext`: `mode: repo-local`, `workspaceRoot: /home/user/manage`, + `allowedEditRoots: ["/home/user/manage"]`, `warnings: []`. All three files written are inside the + authoritative workspace / allowed edit roots. ✓ +- `relationships.sameDomainActiveChanges: []`, `collisions: []` — **no active same-domain + collisions**, so no archive/sync ordering decision was required. +- The `web-ui` and `service-storage` canonical domains were **not touched** (verified untouched by + `git status`). + +## 3. Domains synced & canonical files updated + +| Domain | Change-side delta (source) | Canonical (sync target) | Action | +|---|---|---|---| +| `prometheus-charting` | `openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md` | `openspec/specs/prometheus-charting/spec.md` | **MODIFIED + ADDED** — 16 requirement blocks replaced in place by exact ID; 3 new requirements appended | + +- **Canonical file updated:** `openspec/specs/prometheus-charting/spec.md` — now 30 requirements + (SC-101 … SC-130). +- **Change-side delta created:** `openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md` + — `## MODIFIED Requirements` (16) + `## ADDED Requirements` (3). + +## 4. Requirement delta (MODIFIED / ADDED / PRESERVED) + +### MODIFIED (16) — transport & naming requirements replaced in place by exact SC- ID + +| Canonical ID | Header (matched exactly) | What changed (GM source) | +|---|---|---| +| SC-101 | Prometheus range query returns the existing series shape | direct Prom `/api/v1/query_range` → Grafana `POST /api/ds/query`; config `base_url` → `grafana_url`+`datasource_uid`; secret `api_key` → `grafana_api_key` (GM-101, GM-102) | +| SC-102 | Series label normalization is shared and Prometheus-native | shared module now holds both `normalize_prometheus_matrix` + `normalize_grafana_frames`; shared dedup (GM-104) | +| SC-103 | Range query errors degrade gracefully | gateway-specific failures (401/403, datasource-not-found) added (GM-103) | +| SC-104 | Step is derived from the window preset | step math reused, mapped to `intervalMs`/`maxDataPoints`/`from`-`to` (GM-105) | +| SC-105 | Chart widget moves from grafana to prometheus | kinds stay on `prometheus`; transport is gateway; `Metric*` component binding (GM-109, GM-112) | +| SC-107 | Chart supports multiple series | multi-series via gateway (GM-106) | +| SC-109 | Gauge renders an instant scalar | instant query via gateway (GM-107) | +| SC-111 | Gauge is scalar-only | scalar-only preserved through gateway (GM-107) | +| SC-112 | Mean computes client-side over a window | range query via gateway (GM-108) | +| SC-114 | Mean is scalar-only | scalar-only preserved through gateway (GM-108) | +| SC-115 | No grafana references in backend source | becomes "no `grafana` **service type**"; transport refs allowed | +| SC-116 | No grafana references in frontend source | grafana transport allowed; no service type / link widget | +| SC-121 | config.yaml matches implementation | config.yaml now references Grafana as sanctioned transport | +| SC-122 | CHANGELOG documents the migration | migration note → "reconfigure prometheus with gateway fields" (GM-114) | +| SC-125 | New widget kinds have tests | renamed `Metric*` widgets tested (GM-111) | +| SC-126 | No silent data migration | no auto-migrate of old-shape prometheus config; operator reconfigures (GM-113) | + +### ADDED (3) — fresh canonical IDs continuing the SC- sequence + +| Canonical ID | Header | GM source | +|---|---|---| +| SC-128 | get_prometheus_status validates the full gateway path | GM-110 | +| SC-129 | Startup logs old-shape prometheus config without crashing | GM-113 (new behavior) | +| SC-130 | Grafana is the sanctioned transport for the prometheus service | sanctioned-transport statement | + +### PRESERVED (11) — unchanged in canonical; NOT in the delta + +SC-106 (chart renderer reused), SC-108 (chart window preset), SC-110 (gauge threshold bands), +SC-113 (mean plain PromQL), SC-117 (no grafana service type in registries), SC-118 (grafana status/UI +removed), SC-119 (orphan degrade), SC-120 (grafana tests removed), SC-123 (backend tests pass), +SC-124 (frontend typecheck/build/lint), SC-127 (non-blocking on service-storage-harness). + +> **Count reconciliation:** 16 MODIFIED + 11 PRESERVED = 27 pre-existing requirements (SC-101 … +> SC-127, all retained); + 3 ADDED (SC-128 … SC-130) = **30 total** in the canonical spec post-sync. + +### REMOVED (0) · RENAMED (0) + +No requirements were removed or renamed. RENAMED is intentionally unsupported by the native delta +helper; the GM→SC mapping was resolved as ADDED/MODIFIED, never RENAMED. + +## 5. Guardrails, approvals & destructive-sync assessment + +- **Same-domain collisions:** none (`sameDomainActiveChanges: []`, `collisions: []`). No ordering + decision was needed. +- **Destructive sync:** This is a **non-additive** (MODIFIED-heavy) sync — 16 MODIFIED blocks, 0 + REMOVED. Per the sync contract, a destructive/large-MODIFIED sync requires explicit approval + recorded in the parent prompt. **Approval is recorded:** the dispatch brief explicitly identifies + this as "the project's first non-additive sync," enumerates the exact SC-xxx to modify (including + the partial ones), and directs the MODIFIED/ADDED/PRESERVED split. No REMOVED requirements are + used. Proceeding under that recorded approval. +- **MODIFIED-block matching:** every MODIFIED block header matched the canonical spec **exactly** + (verified by `grep` of all 16 headers pre-edit; all 16 replacements applied with zero header + mismatches). No `## RENAMED Requirements` section was authored. +- **Legacy flat spec:** the flat `spec.md` is intentionally **left in place** as the planning + artifact the work was built against (archive convention keeps flat specs); the new domain delta + sits alongside it under `specs/prometheus-charting/`. +- **Other-domain isolation:** `openspec/specs/web-ui/spec.md` and `openspec/specs/service-storage/spec.md` + were **not modified** (`git status --porcelain` empty for both). + +## 6. Validation / checks performed (file-backed, read-only) + +Run from `/home/user/manage` (no source edits, no test re-runs — owned by verify, already green): + +| Check | Command | Result | +|---|---|---| +| Canonical requirement count | `grep -cE "^### Requirement: SC-"` | **30** ✓ (was 27; +3 ADDED) | +| Canonical ID sequence intact | `grep -oE "SC-[0-9]+" \| sort -un` | SC-101 … SC-130, contiguous, no gaps/dupes ✓ | +| MODIFIED headers all matched | 16 exact-header edits to canonical | all 19 edits applied (front-matter, Purpose, 16 bodies, append) with **zero header mismatches** ✓ | +| No direct-Prom residue in canonical | `grep -c "api/v1/query_range"` canonical | **1** — and it is SC-101's intentional "MUST NOT issue any direct `GET /api/v1/query_range`" negation ✓ | +| Preserved block integrity | spot-check SC-106/108/110/113/117/118/127 | preserved verbatim ✓ | +| Delta is MODIFIED + ADDED only | count delta section headers | MODIFIED=1, ADDED=1, REMOVED=0, RENAMED=0 ✓ (no REMOVED → no destructive removal) | +| Delta ADDED IDs are fresh | `grep "SC-12[89]\|SC-130"` delta | SC-128, SC-129, SC-130 ✓ (no GM- reuse in canonical) | +| `web-ui` / `service-storage` untouched | `git status --porcelain openspec/specs/{web-ui,service-storage}` | empty ✓ | +| Front-matter + Purpose consistent | re-read canonical head | transport narrative updated to gateway (no "no Grafana middleman" contradiction with SC-121/SC-130) ✓ | +| Markdown validity | write-time lint | all three files clean ✓ | + +## 7. Matching issues + +**None.** All 16 MODIFIED block headers matched the canonical spec exactly; no fallback or fuzzy +matching was required. The single `api/v1/query_range` occurrence in the canonical post-sync is the +intentional negation inside SC-101, not a stale direct-Prom requirement. + +## 8. Carry-over / residual notes for the archive summary + +1. **[COSMETIC, non-blocking]** SC-106 (PRESERVED) still reads "preserved in the rebranded + `PrometheusChartWidget`". This change renames that component to `MetricChartWidget` (GM-111), but + SC-106 was explicitly listed as **PRESERVED** (not MODIFIED) in the change's "Canonical delta + intent", so it was left verbatim. The *intent* (recharts renderer reused unchanged) remains true; + only the component name in that one block is a cosmetic carry-over. A future change may refresh + the name if desired. +2. **[VERIFY carry-over, non-blocking for sync]** GM-115 was PARTIAL in verify on enumerated + test-coverage only (no startup-validation test; no status 401/403 → `auth_failed` test). Functional + code is correct; the dispatch brief treats this as closed/effectively-PASS. The new SC-128 + (status gateway path) and SC-129 (startup validation) requirements now canonicalize this behavior; + `sdd-archive` may note the test-coverage gap. +3. **[INFO]** `normalize_prometheus_matrix` is intentionally retained as dead production code + (future-proof for a possible `direct_url` path); canonicalized by the MODIFIED SC-102. +4. **[INFO]** Stale generated `.pi-map.md` artifacts still reference old `Prometheus*Widget` names; + not deliverable source — regenerate at the orchestrator's convenience. +5. **[INFO]** Two uncommitted whitespace-only edits exist in the working tree + (`PrometheusMetricWidget.tsx`, `ServicesPage.tsx`) — unrelated to this sync; no files were staged + by this phase (`git diff --cached` empty). + +## 9. Next recommended phase + +→ **`sdd-archive`** (clean). Re-scan the native status engine to confirm `specs: done` / `sync: ready` +/ `archive: ready`, then move the change to +`openspec/changes/archive/YYYY-MM-DD-grafana-metric-gateway`, carrying over the items in §8. Do +**not** commit or push — the parent owns the commit with explicit paths. + +--- + +### Appendix — Files written by this sync (OpenSpec only; no source code) + +- `openspec/changes/grafana-metric-gateway/specs/prometheus-charting/spec.md` — **change-side domain + delta** (`## MODIFIED Requirements` × 16 + `## ADDED Requirements` × 3). +- `openspec/specs/prometheus-charting/spec.md` — **canonical spec** (MODIFIED × 16 in place, ADDED × 3 + appended; now 30 requirements SC-101 … SC-130; front-matter + Purpose updated for transport + consistency). +- `openspec/changes/grafana-metric-gateway/sync-report.md` — this report. diff --git a/openspec/specs/prometheus-charting/spec.md b/openspec/specs/prometheus-charting/spec.md index 44e9c90..2333b4a 100644 --- a/openspec/specs/prometheus-charting/spec.md +++ b/openspec/specs/prometheus-charting/spec.md @@ -1,43 +1,51 @@ # Prometheus Charting -> Domain: `prometheus-charting` · **Canonical specification.** Synced from change `prometheus-direct-charting`. +> Domain: `prometheus-charting` · **Canonical specification.** Created by `prometheus-direct-charting`; +> transport model modified by `grafana-metric-gateway` (Grafana gateway is the sanctioned +> `prometheus` query path; chart/gauge/mean widget components renamed to neutral `Metric*`). > -> This is the merged end-state of cutting Grafana out of the metric-visualization path and making -> Prometheus the direct data source. It captures the durable, post-change contracts for in-app -> Prometheus-backed metric rendering, not the per-slice delivery strategy (which remains on record -> in the change's `spec.md` / `tasks.md` under `openspec/changes/prometheus-direct-charting/`). +> This is the merged end-state of Manage's metric-visualization surface. The `prometheus` service is +> the sole metric surface; its `chart` / `gauge` / `mean` / `metric` widget queries route through a +> **Grafana gateway** (`POST {grafana_url}/api/ds/query`) — Prometheus remains the logical source, +> Grafana is the transport. In-app charting via `recharts` is the sanctioned renderer; no `grafana` +> *service type* (or its widgets/status/UI surfaces) is reintroduced; orphaned Grafana widget +> instances degrade gracefully; and operators reconfigure `prometheus` services for the gateway (no +> silent auto-migration). This spec is acceptance-focused and verifiable; it deliberately does not +> prescribe implementation. ## Purpose -Define WHAT must be true of Manage's metric-visualization surface after the change: Prometheus is -queried **directly** (no Grafana middleman) to back a multi-series line **chart**, a scalar -**gauge**, and a windowed **mean**, alongside the existing instant-query numeric **metric** widget. -In-app charting via `recharts` is the sanctioned renderer; the `grafana` service type, its widgets, -status endpoint, and UI surfaces are fully removed; orphaned Grafana widget instances degrade -gracefully; and operators migrate by recreating services/widgets (no silent data migration). This -spec is acceptance-focused and verifiable; it deliberately does not prescribe implementation. +Define WHAT must be true of Manage's metric-visualization surface after the `grafana-metric-gateway` +change: metric queries for the `prometheus` service route through a **Grafana gateway** +(`POST {grafana_url}/api/ds/query`) to back a multi-series line **chart**, a scalar **gauge**, and a +windowed **mean**, alongside the instant-query numeric **metric** widget — Prometheus remains the +logical source, Grafana is the transport. In-app charting via `recharts` is the sanctioned renderer; +no `grafana` *service type* (or its widgets/status/UI surfaces) is reintroduced; orphaned Grafana +widget instances degrade gracefully; and operators reconfigure `prometheus` services for the gateway +(no silent auto-migration). This spec is acceptance-focused and verifiable; it deliberately does not +prescribe implementation. ## Requirements ### Requirement: SC-101 — Prometheus range query returns the existing series shape -When a `prometheus` widget of kind `chart` is fetched, the backend MUST query `{base_url}/api/v1/query_range` with `query`, `start`, `end`, and `step` derived from the widget config, and return a payload of shape `{ "series": [{ "label": str, "points": [{ "t": int, "v": float|null }] }] }` — the exact shape the frontend chart renderer already consumes. +The `prometheus` service type's config MUST point at a Grafana gateway — declaring `grafana_url` (http(s)-schema-enforced), `datasource_uid` (default `"prometheus"`), and `timeout_seconds` — with a required, Fernet-encrypted `grafana_api_key` secret. The former direct-Prometheus `base_url` config field and the old optional `api_key` secret MUST be gone. When a `prometheus` widget of kind `chart` is fetched, the backend MUST issue `POST {grafana_url}/api/ds/query` (carrying `Authorization: Bearer {grafana_api_key}` and a `queries[0]` entry keyed by `datasource_uid`), MUST NOT issue any direct `GET /api/v1/query_range` against a Prometheus URL, and MUST return a payload of shape `{ "series": [{ "label": str, "points": [{ "t": int, "v": float|null }] }] }` — the exact shape the frontend chart renderer already consumes. ### Requirement: SC-102 — Series label normalization is shared and Prometheus-native -The metric-label → readable-label normalization MUST live in a single shared helper (not duplicated in a Grafana path) and MUST produce meaningful labels for Prometheus matrix results, including deduplicating repeated labels via a `label (n)` suffix. +The metric-label → readable-label normalization MUST live in a single shared helper module that holds **both** normalizers — the retained Prom-matrix normalizer (`normalize_prometheus_matrix`) and the (re-)added Grafana-frames normalizer (`normalize_grafana_frames`) — and MUST deduplicate repeated labels via a `label (n)` suffix using **one shared dedup rule** (not duplicated across the two normalizers). Both normalizers produce the same `{label, points}` series shape, and the frames normalizer MUST apply the label fallback chain (displayName → sorted Prometheus labels excluding `__`-prefixed keys → `"value"`). ### Requirement: SC-103 — Range query errors degrade gracefully -A Prometheus timeout, connection error, or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional. +A Grafana gateway timeout, connection error, HTTP 401/403 (auth), datasource-not-found, or non-2xx response MUST cause the widget data fetch to return `{ "error": str }` (not raise), so the frontend renders the standard per-widget error state and the rest of the dashboard remains functional. The error message MUST be specific enough to distinguish auth failure from unreachable-host from datasource-not-found where the Grafana response permits. ### Requirement: SC-104 — Step is derived from the window preset -Given a window preset (1h / 6h / 24h / 7d), the backend MUST derive a `step` that yields a reasonable number of points (target ~100–300 points). Users do not configure `step` directly. +Given a window preset (1h / 6h / 24h / 7d), the backend MUST reuse the existing `WINDOW_PRESETS` and `step_for_window` math to derive the gateway request's `intervalMs` (`step * 1000`), `maxDataPoints`, and `from`/`to` time bounds, landing the resulting point count in the same ~100–300 band as the pre-change direct-Prom path. Users do not configure `from`/`to`/`step`/`intervalMs` directly. ### Requirement: SC-105 — Chart widget moves from grafana to prometheus -A widget kind named `chart` MUST be bound to the `prometheus` service type in both the backend registry and the frontend `SERVICE_REGISTRY`. The `grafana` service type MUST NOT offer a `chart` kind. +Widget kinds `chart`, `gauge`, `mean`, and `metric` MUST remain bound to the `prometheus` service type — the kind strings are persisted in widget instance rows and MUST NOT change. The `grafana` service type MUST NOT be reintroduced and MUST NOT offer any widget kind. The backend adapter (internal class `MetricSource`, registry key still `"prometheus"`) and the frontend `SERVICE_REGISTRY` bind these kinds to the neutral `Metric*` components for `chart`/`gauge`/`mean` (and `PrometheusMetricWidget` for `metric`); all four kinds source their data via the Grafana gateway. ### Requirement: SC-106 — Chart renderer is reused unchanged @@ -45,7 +53,7 @@ The recharts rendering (line chart, multi-series, axes, tooltip, `mergeSeries`, ### Requirement: SC-107 — Chart supports multiple series -The `chart` widget MUST render all series returned by the range query, each as its own line with a distinct color. There is no single-series restriction on `chart`. +The `chart` widget MUST render all series returned by the gateway range query, each as its own line with a distinct color (via `normalize_grafana_frames`). There is no single-series restriction on `chart`. ### Requirement: SC-108 — Chart window is a preset @@ -53,7 +61,7 @@ The `chart` widget config MUST expose the time window as a preset selector (`1h` ### Requirement: SC-109 — Gauge renders an instant scalar -A widget kind named `gauge` MUST be bound to the `prometheus` service. Its data fetch MUST run an instant PromQL query (`/api/v1/query`) and return the scalar result for rendering as a gauge. +A widget kind named `gauge` MUST be bound to the `prometheus` service. Its data fetch MUST run an instant PromQL query through the Grafana gateway (a gateway query with a minimal window / single data point) and return the scalar result for rendering as a gauge. ### Requirement: SC-110 — Gauge supports configurable threshold bands @@ -61,11 +69,11 @@ The `gauge` widget config MUST accept optional threshold values (e.g. `warn_at`, ### Requirement: SC-111 — Gauge is scalar-only -The `gauge` widget MUST render exactly one scalar value. If the instant query returns multiple series, the adapter MUST return `{ "error": str }` (not silently pick one), directing the user to refine the PromQL. +The `gauge` widget MUST render exactly one scalar value. If the gateway query returns multiple series, the adapter MUST return `{ "error": str }` (not silently pick one), directing the user to refine the PromQL. ### Requirement: SC-112 — Mean computes client-side over a window -A widget kind named `mean` MUST be bound to the `prometheus` service. Its data fetch MUST run a range query over the configured window preset and return the arithmetic mean of all non-null point values as a single scalar. +A widget kind named `mean` MUST be bound to the `prometheus` service. Its data fetch MUST run a range query through the Grafana gateway over the configured window preset and return the arithmetic mean of all non-null point values of the single series as a single scalar. ### Requirement: SC-113 — Mean uses plain PromQL + window preset @@ -73,15 +81,15 @@ The `mean` widget config MUST accept a plain PromQL expression (no requirement t ### Requirement: SC-114 — Mean is scalar-only -The `mean` widget MUST render exactly one scalar value. If the range query returns multiple series, the adapter MUST return `{ "error": str }` (not silently aggregate across series). +The `mean` widget MUST render exactly one scalar value. If the gateway range query returns multiple series, the adapter MUST return `{ "error": str }` (not silently aggregate across series). ### Requirement: SC-115 — No grafana references in backend source -After the change, `grep -ri grafana backend/src --include='*.py'` MUST return no matches (excluding comments/changelog that are explicitly about the removal, if any are retained — but ideally zero). +Grafana is the sanctioned **transport** for the `prometheus` service, so backend source legitimately contains Grafana transport references (e.g. `grafana_url`, `grafana_api_key`, `normalize_grafana_frames`, `POST …/api/ds/query`). After the change, those references are transport-only: there MUST be no `grafana` **service type**, no `grafana` service definition, and no `grafana` entry in the backend `SERVICE_DEFINITIONS` / `SERVICE_ADAPTERS` registries. The `integrations/grafana.py` service-type module stays deleted. ### Requirement: SC-116 — No grafana references in frontend source -After the change, `grep -ri grafana frontend/src` MUST return no matches, **excluding** (a) test fixtures where "Grafana" appears as a user-authored dashboard *shortcut label* unrelated to the grafana service type (e.g. `Dashboard.test.tsx`), and (b) `LinksTab.tsx` / `service-tabs/index.ts` lines that are themselves being deleted as part of SC-118. (Source finding: `ObservabilityPage.tsx` was refactored into `service-tabs/`.) +Grafana transport is allowed in the frontend — the `prometheus` service config form renders Grafana gateway fields (`grafana_url`, `datasource_uid`) from the backend-provided schema. No frontend code MUST reintroduce a `grafana` **service type**, a Grafana deep-link / dashboard widget, `LinksTab`, or any Grafana-status UI surface. (Test fixtures where "Grafana" appears as a user-authored dashboard *shortcut label* unrelated to a service type remain excluded.) ### Requirement: SC-117 — Grafana service type is gone from registries @@ -101,11 +109,11 @@ All Grafana-specific tests (backend and frontend) MUST be deleted; no test refer ### Requirement: SC-121 — config.yaml matches implementation -`openspec/config.yaml` MUST NOT contain the stale claims "Do NOT re-implement charting in-app" or "No recharts/d3 is in use." It MUST reflect that in-app charting via `recharts` is the sanctioned approach for Prometheus-backed series, and MUST NOT reference Grafana as a chart path. +`openspec/config.yaml` MUST reflect that metric queries for the `prometheus` service are routed through a Grafana gateway (`POST /api/ds/query`) and that in-app charting via `recharts` is the sanctioned renderer. It MUST NOT carry the stale claims "Do NOT re-implement charting in-app", "No recharts/d3 is in use", or "Prometheus is queried directly (no Grafana middleman)". ### Requirement: SC-122 — CHANGELOG documents the migration -`CHANGELOG.md` MUST include an entry instructing operators to delete existing Grafana service instances and recreate them as Prometheus services, noting that configured `grafana/chart` widgets must be recreated as `prometheus/chart` widgets. +`CHANGELOG.md` `[Unreleased]` MUST include a **BREAKING** entry stating that existing `prometheus` service instances must be reconfigured: replace the old `base_url` with `grafana_url`, add the `grafana_api_key` secret, and optionally set `datasource_uid`, because metric queries now route through Grafana's `POST /api/ds/query`. ### Requirement: SC-123 — Backend tests pass @@ -117,12 +125,24 @@ All Grafana-specific tests (backend and frontend) MUST be deleted; no test refer ### Requirement: SC-125 — New widget kinds have tests -`PrometheusChartWidget`, `PrometheusGaugeWidget`, and `PrometheusMeanWidget` MUST each have a frontend test covering at least: loading state, error state, and a rendered data case. +The renamed widgets `MetricChartWidget`, `MetricGaugeWidget`, and `MetricMeanWidget` MUST each have a frontend test covering at least: loading state, error state, and a rendered data case. (The `metric`-kind widget `PrometheusMetricWidget` is not renamed but remains tested.) ### Requirement: SC-126 — No silent data migration -The change MUST NOT attempt to auto-migrate existing `grafana` service rows into `prometheus` rows (URLs differ; true migration is impossible). Migration is operator-driven per the CHANGELOG note. +The change MUST NOT auto-migrate persisted `prometheus` service instances from the old `base_url` config shape to the new `grafana_url` gateway shape (URLs and credentials differ; a safe automated rewrite is not possible). Migration is operator-driven per the CHANGELOG note; old-shape instances are flagged on startup (see SC-129) and degrade at runtime to a gateway error until reconfigured. ### Requirement: SC-127 — Non-blocking on the service-storage-harness change This change MUST NOT depend on the `service-storage-harness` change. It is independently buildable, testable, and deployable. (The reverse dependency holds: the qBit speed widget depends on this change's chart capability.) + +### Requirement: SC-128 — get_prometheus_status validates the full gateway path + +`get_prometheus_status` MUST validate the full metric path by running a trivial query (e.g. `up`) through the Grafana gateway (`POST {grafana_url}/api/ds/query`). Success MUST return `{ "up": true, "version": , … }` — a sentinel such as `"ok"` is acceptable when Grafana's `/api/ds/query` response carries no Prometheus build-info. Failure MUST return `{ "up": false, "error": }` distinguishing auth failure (HTTP 401/403), unreachable host, and datasource/gateway error where the response permits. + +### Requirement: SC-129 — Startup logs old-shape prometheus config without crashing + +When the backend starts and a persisted `prometheus` service instance carries the old `base_url` config key without `grafana_url`, startup MUST log a clear migration message naming the service and instructing reconfiguration, and MUST complete without crashing (it MUST NOT delete, disable, or otherwise mutate the instance). The instance's widget fetches degrade to a gateway error until the operator reconfigures it. + +### Requirement: SC-130 — Grafana is the sanctioned transport for the prometheus service + +The `prometheus` service type is the sole metric surface in Manage. Its metric queries (`chart`, `gauge`, `mean`, `metric`) are sanctioned to route through a Grafana gateway — `POST {grafana_url}/api/ds/query` with a `grafana_api_key` and a Prometheus `datasource_uid`. Grafana is the transport, not a first-class service type; a direct-Prometheus HTTP path is not supported.