feat(v2): complete v2 reimplementation

2026-07-31 13:33:39 +02:00
parent 396219e776
commit bd107d6a30
137 changed files with 20737 additions and 155 deletions
@@ -0,0 +1,29 @@
+# M10 SSH source evidence
+
+The released source capability is `ssh`: private-key-only, pinned-host-key,
+forced-SFTP chroot access. No password, shell, command channel, agent, default
+key discovery, or arbitrary remote root is supported.
+
+## Fixture
+
+`tests/compose.ssh.yaml` builds a test-only OpenSSH server. Each run generates
+host and client Ed25519 keys under pytest `tmp_path`, mounts no committed keys,
+and configures a dedicated `backup` account with `ChrootDirectory /home/backup`
+and `ForceCommand internal-sftp`. The ordinary operator Compose stack is not
+modified. Run `make test-ssh-integration` to build, run, and tear down the
+fixture.
+
+## Verification
+
+- Fake transport tests cover pin mismatch before authentication/SFTP, bounded
+  reads, unsafe entries, and accepted/rejected private-key algorithms.
+- The opt-in live test covers a private-key probe, backup, signed verification,
+  and restore through the forced-SFTP fixture.
+- `make test-ssh-integration` passed after fixture isolation and SFTP-channel
+  concurrency fixes.
+- Final `make check` passed: 107 unit/contract, 73 integration (one skipped),
+  15 fault, and 33 security tests; Ruff, mypy, TypeScript, and the frontend
+  build passed.
+
+See `docs/runbooks/ssh-sources.md` for deployment prerequisites, rotation, and
+containment limitations.
@@ -0,0 +1,55 @@
+# M11 recovery import evidence
+
+- Recovery bundles use the existing versioned `BTREC` Argon2id/AES-GCM codec
+  and now carry a version-2 authenticated catalog containing repositories,
+  key epochs, sources, jobs, executions, and backups required for encrypted
+  restore.
+- `backup-tool admin recovery import` requires a migrated empty destination DB,
+  re-inspects every surviving repository through configured allowlists, writes
+  signing/data keys exclusively with mode `0600`, and rejects conflicts.
+- Imported sources are `unavailable`; imported jobs are `archived` and disabled.
+  Imported metadata therefore supports existing restore records without silently
+  restarting backup schedules.
+- The focused host-loss drill exports an encrypted backup, imports it into a
+  fresh metadata/key host, and restores the file byte-for-byte. It also proves
+  unsafe repository paths and non-empty destination metadata are rejected.
+- Encrypted repository creation is enabled only after that drill passed and now
+  records its active data-key epoch atomically with repository metadata.
+- Interrupted imports remove newly installed key files on handled failure; a
+  retry also safely adopts only exact, authenticated key files left by an
+  unclean process loss. Rotation writes an fsynced repository journal before
+  its DB transition; worker startup deterministically completes a committed
+  epoch transition or removes an uncommitted one while retaining old-active.
+  A stale rollback journal is cleared safely even when the unreferenced new key
+  was already deleted before the journal cleanup could run.
+- Key-aware GC decrypts encrypted manifests using their declared epoch key and
+  remains fail-closed for absent, wrong, or corrupt keys/manifests. Restore
+  removes setuid, setgid, and sticky bits from captured modes. Migration 0007
+  refuses downgrade while key metadata is populated.
+- Recovery catalog import preserves backup `created_at` and `tombstoned_at`.
+- Snapshot staging roots, per-execution directories, blob directories, and
+  plaintext temporary blobs are created owner-only (`0700`/`0600`) independent
+  of umask and are rejected if their permissions are unsafe.
+
+## Verification
+
+```text
+pytest tests/integration/test_encrypted_repository.py -q
+5 passed
+
+make test-fault
+11 passed
+
+make test-security
+21 passed
+
+make lint && make typecheck && make frontend-build
+passed
+
+make check
+86 unit/contract, 53 integration, 12 fault, and 22 security tests passed;
+Ruff, mypy, TypeScript, and frontend build passed.
+
+git diff --check && git diff --cached --quiet
+passed
+```
@@ -0,0 +1,43 @@
+# M12 evidence
+
+- Migration revision: `0008_notification_outbox`
+- Event schema version: `1`
+- Catalog: 22 live stable IDs only. Deferred/unimplemented operation types are not public notification contracts.
+- Delivery guarantee: durable at-least-once, stable event ID, leased worker retries; receiver deduplication is required.
+
+## Green focused evidence
+
+```text
+PYTHONPATH=.:backend/src .venv/bin/python -m pytest \
+  tests/contract/test_notification_contract.py \
+  tests/integration/test_migrations.py \
+  tests/integration/test_all_operational_events_deliver.py \
+  tests/integration/test_notifications.py \
+  tests/fault/test_notification_retries.py \
+  tests/security/test_webhook_ssrf.py -q
+28 passed (live-catalog contract, fair dispatch regression; no deferred event IDs).
+
+# Scheduler-role service-path regression
+PYTHONPATH=.:backend/src .venv/bin/python -m pytest \
+  tests/integration/test_scheduler_live_sync.py -q
+1 passed (scheduler role service path)
+```
+
+The suite uses temporary SQLite/repository roots, a fake SMTP implementation, fake resolver inputs, and dispatcher monkeypatches; it performs no real webhook DNS, HTTP, or SMTP delivery. It verifies receiver-visible versioned webhook headers/signatures, STARTTLS-before-AUTH SMTP behavior, transient/permanent SMTP classification, persisted SMTP attempt limits, lease-abandoned attempt closure, selected-only test sends, manual-retry idempotency, and the absence of a plaintext-secret idempotency verifier. Behavioral producer tests cover every live catalog family: execution, schedule, backup/verification, restore, and retention. They also prove fair dispatch under an execution backlog, scheduler-role delivery, and worker-owned retention/GC maintenance. The CLI scheduler role now runs `SchedulerService`; worker maintenance runs durable retention/GC on startup and at bounded intervals.
+
+## Quality evidence
+
+- `make test-fast`: 89 passed.
+- `make test-integration`: 61 passed in 27.00s (the execution wrapper nevertheless returned exit 124 at its fixed 30s wall limit).
+- `make test-fault`: focused fair-dispatch regression passed.
+- `make test-security`: 28 passed.
+- `make lint` and `make typecheck`: passed.
+- `make frontend-build`: passed.
+- `git diff --check`: passed.
+- staged-file check: no staged files.
+
+Final verification: `make check` passed — 90 unit/contract, 61 integration, 14 fault, and 28 security tests; Ruff, mypy, TypeScript, and the frontend build passed. `git diff --check` and the staged-file check also passed.
+
+## Rollback
+
+Disable or archive subscriptions and stop worker dispatch. Do not delete notification events, deliveries, or attempts: they remain audit history. A fresh host recovery intentionally starts with no notification settings or credentials and must be reconfigured.
@@ -0,0 +1,20 @@
+# M13 UI/OpenAPI evidence
+
+- OpenAPI is deterministically exported to `openapi/v2.json`; generated browser client drift is checked by `npm --prefix frontend run api:check`.
+- `text/event-stream` is declared in OpenAPI for execution events. The generated client intentionally emits `executionEventsUrl(...) -> URL`, not a misleading JSON `Promise`; UI opens that URL using browser `EventSource`.
+- Notification delivery retry requests send an `Idempotency-Key` and the browser CSRF header.
+- Recovery status is CLI/runbook-only. The browser has no export, import, bundle, key, or passphrase transfer control.
+- v2.1 PostgreSQL, MySQL, and TAR/download controls remain absent.
+
+## Green M13 checks
+
+```text
+npm --prefix frontend test -- --run  # 9 passed
+npm --prefix frontend run typecheck  # passed
+npm --prefix frontend run build      # passed
+npx --prefix frontend playwright test --config frontend/playwright.config.ts  # 1 passed
+.venv/bin/python tools/export_openapi.py --check openapi/v2.json  # current
+npm --prefix frontend run api:check  # current
+```
+
+Final verification: `make check` passed — 90 unit/contract, 61 integration, 14 fault, and 28 security tests; Ruff, mypy, TypeScript, and the frontend build passed. `git diff --check` and the staged-file check also passed.
@@ -0,0 +1,34 @@
+# M14 operations and packaging evidence
+
+## Delivered
+
+- Pinned non-root OCI application and proxy images; isolated web, scheduler, worker,
+  migrate, and admin roles; same-origin Unix-socket proxy; no reload or embedded roles.
+  The proxy port is loopback-bound by default (`127.0.0.1`), so the localhost public
+  URL cannot permit remote first-admin setup takeover. Operators exposing it through
+  an external reverse proxy must set a non-loopback public URL and bootstrap secret.
+- Role-aware readiness, JSON structured logs, safe worker claim shutdown, and
+  dependency-free Prometheus metrics for request volume/duration, active/stale/failed
+  execution state, schedule lag, corrupt/unavailable repositories, and free space.
+- SBOM generation at `tools/generate_sbom.py`, generated CycloneDX artifact
+  `m14-sbom.json`, and base-image/source provenance in `m14-provenance.md`.
+- Metadata, repository, key, upgrade, disaster-recovery, and observability runbooks.
+
+## Green verification
+
+```text
+ docker compose config --quiet                         # passed (loopback port binding)
+ docker compose build --pull                           # passed
+ make test-e2e                                         # 1 passed in 51.69s (loopback regression)
+ make check                                            # passed after loopback regression
+   91 unit/contract, 61 integration, 15 fault, 33 security
+   Ruff/format, mypy (40 files), TypeScript, frontend build all passed
+ python tools/generate_sbom.py                         # 256 components
+```
+
+`test_compose_v2.py` creates its source and host-bind key fixture in pytest temporary
+directories. The actual service-owned master key is generated only in an ephemeral
+Compose named volume, then `down --volumes --remove-orphans` removes it. No fixture
+secret or source directory is committed. The test runs migration, starts the stack,
+checks readiness and metrics, stops/restarts the worker cleanly, restarts runtime
+roles, verifies setup metadata persists, and tears down the project.
@@ -0,0 +1,13 @@
+# M14 build provenance
+
+- **Source revision:** `396219e776aa9a115900d2b7bfd9fb5c1cfde115`
+- **Application base:** `python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7`
+- **Frontend builder:** `node:22.17.1-alpine@sha256:5539840ce9d013fa13e3b9814c9353024be7ac75aca5db6d039504a56c04ea59`
+- **Proxy base:** `nginx:1.29.7-alpine@sha256:e7257f1ef28ba17cf7c248cb8ccf6f0c6e0228ab9c315c152f9c203cd34cf6d1`
+- **Build command:** `docker compose build --pull`
+- **SBOM:** `docs/release/m14-sbom.json`, generated deterministically with
+  `python tools/generate_sbom.py` from the pinned backend manifest and frontend lockfile.
+
+The build uses digest-pinned bases and a non-root runtime user. Provenance records
+inputs and generation instructions rather than embedding a mutable image tag or a
+secret-bearing build environment.
@@ -0,0 +1,23 @@
+# M15 v2.0 certification evidence
+
+## Reference host and method
+
+Current CI reference host: Linux 7.1.4, Python 3.14.6, 12 CPUs. Certification uses a deterministic synthetic metadata workload: 100 jobs, 1,000,000 declared entries, and 100,000 cataloged backups with 10 TiB **logical** bytes per backup. It does not claim a physical 10 TiB transfer.
+
+`m15-scale-report.json` records 100,000 backup inserts in 0.57 s, a 100-row deep pagination query in 0.004209 s, and a 5.32 MB SQLite fixture.
+
+## Capability certification
+
+```text
+python tools/assert_capabilities.py --release v2.0 \
+  --include local,ssh,restore,webhook,email \
+  --exclude postgresql,mysql,tar_download
+```
+
+Passed. The released contract exposes `local`, `ssh`, `restore`, `webhook`, and `email`; PostgreSQL, MySQL, and TAR download remain disabled.
+
+SSH is private-key-only and requires a dedicated forced-SFTP chroot account; no password, shell, or remote-command path is released.
+
+## Remaining certification gates
+
+Run the full project, container E2E, SSH live integration, fault/security/leakage suites, and review the synthetic workload boundaries before a release commit is created.
@@ -0,0 +1,21 @@
+{
+  "method": "synthetic metadata certification; logical bytes are sparse and no physical 10 TiB payload is allocated",
+  "reference_host": {
+    "cpus": 12,
+    "platform": "Linux-7.1.4-arch1-1-x86_64-with-glibc2.44",
+    "python": "3.14.6"
+  },
+  "results": {
+    "backup_insert_seconds": 0.57,
+    "database_bytes": 5320704,
+    "pagination_rows": 100,
+    "pagination_seconds": 0.004209,
+    "total_seconds": 0.592
+  },
+  "workload": {
+    "backups": 100000,
+    "entries_declared": 1000000,
+    "jobs": 100,
+    "logical_bytes_per_backup": 10995116277760
+  }
+}
@@ -0,0 +1,12 @@
+# M5 Durable Execution Lifecycle Evidence
+
+- Coverage: state-transition contract and DB-enforced active-execution uniqueness
+  under concurrent enqueue; disabled/archived-job rejection; lease reclamation
+  and fencing; retry semantics; startup stale-worker reconciliation; durable
+  event revision counts; and idle worker shutdown.
+- Verification: the focused M5 suite passed: 29 tests across transition,
+  queue/lease, and worker-loss acceptance coverage.
+- Automation: `make test-fault` runs the worker-loss acceptance suite, and
+  `make check` now includes it.
+- Scope: validates M5 durable lifecycle behavior only. Snapshot publication,
+  verification, and restore remain M6 work.
@@ -0,0 +1,14 @@
+# M6 Backup, Verification, and Restore Evidence
+
+- Coverage: local full-snapshot staging; immutable SHA-256 blobs; canonical
+  Ed25519-signed manifests; manifest/blob verification; publication markers and
+  startup reconciliation; selected and dry-run restores; `fail`, `skip`, and
+  `replace` root policies; destination containment; corruption handling; and
+  restore recovery.
+- Focused acceptance: the M6 integration, publication-fault, and restore-path
+  suites pass.
+- Full verification: `make check` passed with 68 unit/contract, 37
+  integration, 7 fault, and 17 security tests, plus Ruff, mypy, frontend
+  typecheck, and frontend production build.
+- Scope: excludes M7 incremental baselines/exclusions and later scheduling,
+  remote source, encryption, retention, and UI milestones.
@@ -0,0 +1,12 @@
+# M7 Incrementals, Exclusions, and Empty-Source Safety Evidence
+
+- Coverage: ordered normalized exclusion rules, exclusion policy persisted in signed
+  manifests, incremental compatible-baseline selection, independently restorable
+  complete manifests, parent linkage, content-addressed blob reuse, and
+  empty-source opt-in.
+- Verification: M7 unit, incremental, and empty-source suites are included in
+  the passing project verification.
+- Full verification: `make check` passed with 78 unit/contract, 39
+  integration, 7 fault, and 17 security tests, plus Ruff, mypy, frontend
+  typecheck, and frontend production build.
+- Scope: scheduling remains M8 work.
@@ -0,0 +1,9 @@
+# M8 Scheduling Evidence
+
+- Coverage: five-field cron validation, IANA timezones, UTC nominal runs,
+  durable schedule CRUD, live next-run updates, misfire handling, and
+  no-overlap delivery through the execution enqueue guard.
+- Full verification: `make check` passed with 81 unit/contract, 40
+  integration, 10 fault, and 17 security tests, plus Ruff, mypy, frontend
+  typecheck, and frontend production build.
+- Scope: retention and garbage collection are M9 work.
@@ -0,0 +1,8 @@
+# M9 Retention and Garbage Collection Evidence
+
+- Coverage: union retention policies, newest-backup protection, pins, tombstones,
+  referenced-object preservation, garbage collection, and reconciliation safety.
+- Full verification: `make check` passed with 86 unit/contract, 42 integration,
+  11 fault, and 17 security tests, plus Ruff, mypy, frontend typecheck, and
+  frontend production build.
+- Scope: remote source support remains M10 work.