manage/openspec/specs/service-credential-testing/spec.md

Service Credential Testing

Domain: service-credential-testing · Canonical specification. Synced from change service-credential-tester.

This is the merged end-state of adding a per-service-type credential tester for the service add/edit dialog. It captures the durable, post-change contracts for the POST /api/services/test endpoint, the closed test_callable dispatch registry, the per-type connection-test routines, the no-persistence + no-secret-logging guarantees, and the frontend test UI (Test button + result pill + gated confirm with a "Save anyway" override) — not the per-slice delivery strategy (which remains on record in the change's spec.md / tasks.md under openspec/changes/service-credential-tester/).

Purpose

Define WHAT must be true of Manage's service credential tester after the change: a backend endpoint (POST /api/services/test) that accepts the unsaved ServiceInstanceInput form values (never a stored service id), validates them first (reusing _validate_input, 422 on malformed config before any network probe), dispatches through a closed compile-time test_callable registry on ServiceDefinition, runs a minimal read-only probe per service type with a short timeout, and returns a structured { ok, detail, evidence } result without persisting any row or secret. Each remote service type (qBittorrent, Prometheus via the Grafana gateway, Alertmanager, Jellyfin, Authentik, ssh_tasks, Nextcloud) carries its own test_connection routine alongside its integration module; backups has no remote test (test_callable = None) and returns an explicit no-test ok.

On the frontend, a shared ServiceTestPanel (Test credentials button + inline result pill) is wired into both the create dialog (CreateServiceDialog in ServicesPage.tsx) and the service edit surface (ServiceConfigEditor in Settings.tsx); the Create/Save confirm buttons are gated on a passing test, with a "Save anyway" override for pre-configuring an offline service, and editing any connectivity field clears the stale result. This spec is acceptance-focused and verifiable; it deliberately does not prescribe implementation.

Requirements

Requirement: CT-101 — Test endpoint accepts unsaved service input and returns a structured result

The backend MUST expose POST /api/services/test accepting a ServiceInstanceInput body (service_type, name, config, secrets, enabled) — NOT a service id, because the tester operates on unsaved form values. On success the endpoint MUST return HTTP 200 with a body of shape { "ok": bool, "detail": str, "evidence": str | null }. The endpoint MUST be auth-gated identically to every other /api/services/* endpoint (JWT/API-key).

Scenario: successful connection returns ok with evidence

• GIVEN a valid ServiceInstanceInput for a reachable service
• WHEN POST /api/services/test is called with that body
• THEN the response is HTTP 200 with { "ok": true, "detail": <human summary>, "evidence": <proof string> }

Scenario: failed connection returns ok=false with actionable detail

• GIVEN a valid input whose target is unreachable or rejects auth
• WHEN the endpoint is called
• THEN the response is HTTP 200 with { "ok": false, "detail": <specific failure message>, "evidence": null }

Requirement: CT-102 — Validation runs before the test and rejects malformed config with 422

The endpoint MUST reuse the existing _validate_input helper (from routers/services.py) to validate service_type, config (via the definition's config_model), and secret-key names BEFORE any network probe. Malformed input MUST raise HTTP 422 with the same detail format as POST /api/services/instances, so the existing validation-error surfacing (the #1 fix in ServicesPage.tsx) covers the test endpoint too.

Scenario: schema-less URL rejected before any network call

• GIVEN a qbittorrent input whose base_url lacks the http:// schema
• WHEN the endpoint is called
• THEN the response is HTTP 422 with a detail naming the schema requirement
• AND no network call is made to qBittorrent

Scenario: unknown service type rejected

• GIVEN an input with service_type = "nonexistent"
• WHEN the endpoint is called
• THEN the response is HTTP 422 with "Unknown service type"

Requirement: CT-103 — Closed per-type dispatch mirrors SERVICE_DEFINITIONS

The endpoint MUST dispatch to a per-type test routine via the service definition (a new optional test_callable field on ServiceDefinition, defaulting to None). The dispatch is closed and compile-time — no runtime plugin loading, no arbitrary callable execution. A service type whose definition has test_callable = None (e.g. backups, which has no remote connection) MUST return { "ok": true, "detail": "No connection test for this service type", "evidence": null } without any network call.

Scenario: backups type returns ok without a network probe

• GIVEN a backups service input
• WHEN the endpoint is called
• THEN the response is { "ok": true, "detail": "No connection test for this service type", "evidence": null }
• AND no network request is issued

Scenario: unknown widget kind is not involved

• GIVEN a prometheus input
• WHEN the endpoint is called
• THEN the dispatch keys off service_type only (not widget_kind); the test is per-service-instance, not per-widget

Requirement: CT-104 — qBittorrent test logs in and probes maindata; surfaces auth failures specifically

The qbittorrent test routine MUST construct a QbittorrentClient from the config + decrypted secrets, call login (POST /api/v2/auth/login), then probe GET /api/v2/sync/maindata (or /app/version). On success it MUST return { "ok": true, "evidence": <qBittorrent version or "connected"> }. When qBittorrent's login returns anything other than "Ok." (e.g. the literal "Fails."), the routine MUST return { "ok": false, "detail": "Authentication failed — qBittorrent rejected the credentials." } — surfacing the specific auth failure at the UI layer, not just in backend logs.

Scenario: correct credentials succeed with version evidence

• GIVEN a qBittorrent instance reachable at the configured URL with valid username/password
• WHEN the test routine runs
• THEN it returns { "ok": true, "detail": "Connected to qBittorrent.", "evidence": <version> }

Scenario: wrong password surfaces a specific auth message

• GIVEN a qBittorrent instance that returns "Fails." on login
• WHEN the test routine runs
• THEN it returns { "ok": false, "detail": "Authentication failed — qBittorrent rejected the credentials.", "evidence": null }

Requirement: CT-105 — Prometheus test queries the Grafana gateway with expr "up"

The prometheus test routine MUST issue POST {grafana_url}/api/ds/query carrying Authorization: Bearer {grafana_api_key} and a queries[0] entry keyed by the configured datasource_uid with expr: "up". There MUST be no direct Prometheus HTTP call (the prometheus service sources through Grafana per grafana-metric-gateway). On success it returns { "ok": true, "evidence": "Gateway reachable; datasource responded." }. On HTTP 401/403 it returns an auth-specific message; on connection error it returns an unreachable message.

Scenario: gateway reachable returns ok

• GIVEN a prometheus service with valid Grafana gateway credentials
• WHEN the test routine runs
• THEN it returns { "ok": true, "detail": "Grafana gateway reachable.", "evidence": "Gateway reachable; datasource responded." }

Scenario: wrong API key surfaces auth failure

• GIVEN a prometheus service whose grafana_api_key is invalid
• WHEN the test routine runs
• THEN it returns { "ok": false, "detail": <message mentioning authentication/authorization>, "evidence": null }

Requirement: CT-106 — Alertmanager test probes /api/v2/alerts or /api/v2/status

The alertmanager test routine MUST issue an authenticated GET (using the optional api_key secret when set) to /api/v2/alerts (or /api/v2/status). On success it returns { "ok": true, "evidence": <cluster version> }. On failure it translates the error per the shared helper.

Requirement: CT-107 — Jellyfin test calls /Users via JellyfinClient

The jellyfin test routine MUST construct a JellyfinClient from config + secrets and call .users(). On success it returns { "ok": true, "evidence": "<N> users" }. On failure it translates the error.

Requirement: CT-108 — Authentik test probes its directory endpoint

The authentik test routine MUST issue a lightweight authenticated GET against the Authentik directory endpoint (the same one the existing Authentik users flow uses). On success it returns { "ok": true, "evidence": <instance slug or "connected"> }. On failure it translates the error.

Requirement: CT-109 — ssh_tasks test reuses the test_machine_ssh connect flow

The ssh_tasks test routine MUST reuse the existing SSH connection logic (construct an RemoteSSHClient from the service config + SSH key, call .connect(), translate banner/auth errors per the established test_machine_ssh patterns). On success it returns { "ok": true, "evidence": "Connected to <host>:<port>" }. The known-host recording behavior (first successful connect records the host key) is preserved.

Requirement: CT-110 — Nextcloud test probes /status.php

The nextcloud test routine MUST issue an unauthenticated GET to {base_url}/status.php. On success it returns { "ok": true, "evidence": <Nextcloud version> }. On failure it translates the error.

Requirement: CT-111 — Backups type has no remote test

The backups service type has no remote connection (its data is internal). Its definition MUST set test_callable = None, and the endpoint returns { "ok": true, "detail": "No connection test for this service type" } (CT-103).

Requirement: CT-112 — The test endpoint MUST NOT persist any service row or secret

The test endpoint MUST NOT call store.upsert_service, store.update_setting, or any other persistence method. It accepts plaintext secrets in the request body (over TLS, identical to the existing create endpoint), probes the target, and discards the secrets. A test MUST verify this by calling POST /api/services/test and asserting the service count in the store is unchanged afterward.

Scenario: calling /test does not create a row

• GIVEN an empty service store
• WHEN POST /api/services/test is called with a valid qbittorrent input
• THEN the store's service count remains 0
• AND no new row exists for that service type

Requirement: CT-113 — Secrets are never logged

The test endpoint MUST log at INFO level only test requested type=%s ok=%s — never the request body, config, or secrets. The log call MUST pass through sanitize_log_extra (the existing helper) so any accidental inclusion of secret-named keys is scrubbed.

Scenario: logs contain no secret values

• GIVEN a test call carrying secrets: { "password": "hunter2" }
• WHEN the endpoint runs
• THEN no log line contains "hunter2"
• AND the INFO log line reads test requested type=qbittorrent ok=false (or similar), with no secret values

Requirement: CT-114 — useTestServiceInstance mutation hook + testServiceInstance API client function

frontend/src/api/services.ts MUST gain testServiceInstance(input: ServiceInstanceInput): Promise<TestResult> posting to /api/services/test. frontend/src/hooks/useServices.ts MUST gain useTestServiceInstance() returning a TanStack mutation whose mutateAsync resolves to { ok, detail, evidence }.

Requirement: CT-115 — Test credentials button in CreateServiceDialog and the ServicePage edit dialog

Both the create dialog (CreateServiceDialog in ServicesPage.tsx) and the edit dialog on ServicePage.tsx MUST render a "Test credentials" button below the config/secret form fields. Clicking it fires the useTestServiceInstance mutation with the current draft values. While pending, the button shows a "Testing…" state and is disabled.

Requirement: CT-116 — Result pill renders both states with evidence or detail

After a test completes, the dialog MUST render an inline status pill:

• On success (ok: true): a green pill reading ✓ Connected plus the evidence string.
• On failure (ok: false): a red/destructive pill reading ✗ <detail>.

The pill is rendered in the same dialog, below the Test button, above the footer.

Requirement: CT-117 — Create/Save confirm is gated on a passed test with a "Save anyway" override

The Create (in the add dialog) and Save (in the edit dialog) confirm buttons MUST be disabled by default until testPassed is true. A "Save anyway" checkbox MUST appear that, when checked, re-enables the confirm button — allowing an operator to pre-configure a service that is not yet online. The default state of the checkbox is unchecked (gating is on).

Requirement: CT-118 — Editing a connectivity field clears the test result

When the user edits any field that affects connectivity (the URL/base_url config field, or any secret field), the dialog MUST clear the previous testPassed state and the result pill, requiring a fresh test before the confirm button is re-enabled (unless "Save anyway" is checked). This prevents a stale green result from masking a typo correction.

Requirement: CT-119 — Backend tests cover routines, dispatch, validation, and no-persistence

pytest from backend/ MUST pass, including new tests for:

• Each type's test_connection routine (mocked client) returning the expected {ok, detail, evidence} shape for both success and at least one failure case.
• The POST /api/services/test endpoint dispatching correctly per type.
• Validation running before the test (422 on malformed config, no network call).
• No-persistence: calling /test leaves the store unchanged.

Requirement: CT-120 — Frontend tests cover button, pill, gating, and field-clear

npm run test (vitest) from frontend/ MUST pass, including new tests for:

• The Test button fires the mutation.
• The result pill renders both the green (ok) and red (failed) states.
• The Create/Save confirm button is disabled until the test passes (and re-enabled by "Save anyway").
• Editing a connectivity field clears testPassed.

Requirement: CT-121 — Build and lint stay green

npm run build (tsc -b + vite build) and npm run lint from frontend/, and ruff check src tests from backend/, MUST pass with 0 errors (pre-existing warnings acceptable).