feat(v2): complete v2 reimplementation

2026-07-31 13:33:39 +02:00
parent 396219e776
commit bd107d6a30
137 changed files with 20737 additions and 155 deletions
@@ -0,0 +1,29 @@
+# M10 SSH source evidence
+
+The released source capability is `ssh`: private-key-only, pinned-host-key,
+forced-SFTP chroot access. No password, shell, command channel, agent, default
+key discovery, or arbitrary remote root is supported.
+
+## Fixture
+
+`tests/compose.ssh.yaml` builds a test-only OpenSSH server. Each run generates
+host and client Ed25519 keys under pytest `tmp_path`, mounts no committed keys,
+and configures a dedicated `backup` account with `ChrootDirectory /home/backup`
+and `ForceCommand internal-sftp`. The ordinary operator Compose stack is not
+modified. Run `make test-ssh-integration` to build, run, and tear down the
+fixture.
+
+## Verification
+
+- Fake transport tests cover pin mismatch before authentication/SFTP, bounded
+  reads, unsafe entries, and accepted/rejected private-key algorithms.
+- The opt-in live test covers a private-key probe, backup, signed verification,
+  and restore through the forced-SFTP fixture.
+- `make test-ssh-integration` passed after fixture isolation and SFTP-channel
+  concurrency fixes.
+- Final `make check` passed: 107 unit/contract, 73 integration (one skipped),
+  15 fault, and 33 security tests; Ruff, mypy, TypeScript, and the frontend
+  build passed.
+
+See `docs/runbooks/ssh-sources.md` for deployment prerequisites, rotation, and
+containment limitations.
@@ -0,0 +1,55 @@
+# M11 recovery import evidence
+
+- Recovery bundles use the existing versioned `BTREC` Argon2id/AES-GCM codec
+  and now carry a version-2 authenticated catalog containing repositories,
+  key epochs, sources, jobs, executions, and backups required for encrypted
+  restore.
+- `backup-tool admin recovery import` requires a migrated empty destination DB,
+  re-inspects every surviving repository through configured allowlists, writes
+  signing/data keys exclusively with mode `0600`, and rejects conflicts.
+- Imported sources are `unavailable`; imported jobs are `archived` and disabled.
+  Imported metadata therefore supports existing restore records without silently
+  restarting backup schedules.
+- The focused host-loss drill exports an encrypted backup, imports it into a
+  fresh metadata/key host, and restores the file byte-for-byte. It also proves
+  unsafe repository paths and non-empty destination metadata are rejected.
+- Encrypted repository creation is enabled only after that drill passed and now
+  records its active data-key epoch atomically with repository metadata.
+- Interrupted imports remove newly installed key files on handled failure; a
+  retry also safely adopts only exact, authenticated key files left by an
+  unclean process loss. Rotation writes an fsynced repository journal before
+  its DB transition; worker startup deterministically completes a committed
+  epoch transition or removes an uncommitted one while retaining old-active.
+  A stale rollback journal is cleared safely even when the unreferenced new key
+  was already deleted before the journal cleanup could run.
+- Key-aware GC decrypts encrypted manifests using their declared epoch key and
+  remains fail-closed for absent, wrong, or corrupt keys/manifests. Restore
+  removes setuid, setgid, and sticky bits from captured modes. Migration 0007
+  refuses downgrade while key metadata is populated.
+- Recovery catalog import preserves backup `created_at` and `tombstoned_at`.
+- Snapshot staging roots, per-execution directories, blob directories, and
+  plaintext temporary blobs are created owner-only (`0700`/`0600`) independent
+  of umask and are rejected if their permissions are unsafe.
+
+## Verification
+
+```text
+pytest tests/integration/test_encrypted_repository.py -q
+5 passed
+
+make test-fault
+11 passed
+
+make test-security
+21 passed
+
+make lint && make typecheck && make frontend-build
+passed
+
+make check
+86 unit/contract, 53 integration, 12 fault, and 22 security tests passed;
+Ruff, mypy, TypeScript, and frontend build passed.
+
+git diff --check && git diff --cached --quiet
+passed
+```
@@ -0,0 +1,43 @@
+# M12 evidence
+
+- Migration revision: `0008_notification_outbox`
+- Event schema version: `1`
+- Catalog: 22 live stable IDs only. Deferred/unimplemented operation types are not public notification contracts.
+- Delivery guarantee: durable at-least-once, stable event ID, leased worker retries; receiver deduplication is required.
+
+## Green focused evidence
+
+```text
+PYTHONPATH=.:backend/src .venv/bin/python -m pytest \
+  tests/contract/test_notification_contract.py \
+  tests/integration/test_migrations.py \
+  tests/integration/test_all_operational_events_deliver.py \
+  tests/integration/test_notifications.py \
+  tests/fault/test_notification_retries.py \
+  tests/security/test_webhook_ssrf.py -q
+28 passed (live-catalog contract, fair dispatch regression; no deferred event IDs).
+
+# Scheduler-role service-path regression
+PYTHONPATH=.:backend/src .venv/bin/python -m pytest \
+  tests/integration/test_scheduler_live_sync.py -q
+1 passed (scheduler role service path)
+```
+
+The suite uses temporary SQLite/repository roots, a fake SMTP implementation, fake resolver inputs, and dispatcher monkeypatches; it performs no real webhook DNS, HTTP, or SMTP delivery. It verifies receiver-visible versioned webhook headers/signatures, STARTTLS-before-AUTH SMTP behavior, transient/permanent SMTP classification, persisted SMTP attempt limits, lease-abandoned attempt closure, selected-only test sends, manual-retry idempotency, and the absence of a plaintext-secret idempotency verifier. Behavioral producer tests cover every live catalog family: execution, schedule, backup/verification, restore, and retention. They also prove fair dispatch under an execution backlog, scheduler-role delivery, and worker-owned retention/GC maintenance. The CLI scheduler role now runs `SchedulerService`; worker maintenance runs durable retention/GC on startup and at bounded intervals.
+
+## Quality evidence
+
+- `make test-fast`: 89 passed.
+- `make test-integration`: 61 passed in 27.00s (the execution wrapper nevertheless returned exit 124 at its fixed 30s wall limit).
+- `make test-fault`: focused fair-dispatch regression passed.
+- `make test-security`: 28 passed.
+- `make lint` and `make typecheck`: passed.
+- `make frontend-build`: passed.
+- `git diff --check`: passed.
+- staged-file check: no staged files.
+
+Final verification: `make check` passed — 90 unit/contract, 61 integration, 14 fault, and 28 security tests; Ruff, mypy, TypeScript, and the frontend build passed. `git diff --check` and the staged-file check also passed.
+
+## Rollback
+
+Disable or archive subscriptions and stop worker dispatch. Do not delete notification events, deliveries, or attempts: they remain audit history. A fresh host recovery intentionally starts with no notification settings or credentials and must be reconfigured.
@@ -0,0 +1,20 @@
+# M13 UI/OpenAPI evidence
+
+- OpenAPI is deterministically exported to `openapi/v2.json`; generated browser client drift is checked by `npm --prefix frontend run api:check`.
+- `text/event-stream` is declared in OpenAPI for execution events. The generated client intentionally emits `executionEventsUrl(...) -> URL`, not a misleading JSON `Promise`; UI opens that URL using browser `EventSource`.
+- Notification delivery retry requests send an `Idempotency-Key` and the browser CSRF header.
+- Recovery status is CLI/runbook-only. The browser has no export, import, bundle, key, or passphrase transfer control.
+- v2.1 PostgreSQL, MySQL, and TAR/download controls remain absent.
+
+## Green M13 checks
+
+```text
+npm --prefix frontend test -- --run  # 9 passed
+npm --prefix frontend run typecheck  # passed
+npm --prefix frontend run build      # passed
+npx --prefix frontend playwright test --config frontend/playwright.config.ts  # 1 passed
+.venv/bin/python tools/export_openapi.py --check openapi/v2.json  # current
+npm --prefix frontend run api:check  # current
+```
+
+Final verification: `make check` passed — 90 unit/contract, 61 integration, 14 fault, and 28 security tests; Ruff, mypy, TypeScript, and the frontend build passed. `git diff --check` and the staged-file check also passed.
@@ -0,0 +1,34 @@
+# M14 operations and packaging evidence
+
+## Delivered
+
+- Pinned non-root OCI application and proxy images; isolated web, scheduler, worker,
+  migrate, and admin roles; same-origin Unix-socket proxy; no reload or embedded roles.
+  The proxy port is loopback-bound by default (`127.0.0.1`), so the localhost public
+  URL cannot permit remote first-admin setup takeover. Operators exposing it through
+  an external reverse proxy must set a non-loopback public URL and bootstrap secret.
+- Role-aware readiness, JSON structured logs, safe worker claim shutdown, and
+  dependency-free Prometheus metrics for request volume/duration, active/stale/failed
+  execution state, schedule lag, corrupt/unavailable repositories, and free space.
+- SBOM generation at `tools/generate_sbom.py`, generated CycloneDX artifact
+  `m14-sbom.json`, and base-image/source provenance in `m14-provenance.md`.
+- Metadata, repository, key, upgrade, disaster-recovery, and observability runbooks.
+
+## Green verification
+
+```text
+ docker compose config --quiet                         # passed (loopback port binding)
+ docker compose build --pull                           # passed
+ make test-e2e                                         # 1 passed in 51.69s (loopback regression)
+ make check                                            # passed after loopback regression
+   91 unit/contract, 61 integration, 15 fault, 33 security
+   Ruff/format, mypy (40 files), TypeScript, frontend build all passed
+ python tools/generate_sbom.py                         # 256 components
+```
+
+`test_compose_v2.py` creates its source and host-bind key fixture in pytest temporary
+directories. The actual service-owned master key is generated only in an ephemeral
+Compose named volume, then `down --volumes --remove-orphans` removes it. No fixture
+secret or source directory is committed. The test runs migration, starts the stack,
+checks readiness and metrics, stops/restarts the worker cleanly, restarts runtime
+roles, verifies setup metadata persists, and tears down the project.
@@ -0,0 +1,13 @@
+# M14 build provenance
+
+- **Source revision:** `396219e776aa9a115900d2b7bfd9fb5c1cfde115`
+- **Application base:** `python:3.12.11-slim-bookworm@sha256:519591d6871b7bc437060736b9f7456b8731f1499a57e22e6c285135ae657bf7`
+- **Frontend builder:** `node:22.17.1-alpine@sha256:5539840ce9d013fa13e3b9814c9353024be7ac75aca5db6d039504a56c04ea59`
+- **Proxy base:** `nginx:1.29.7-alpine@sha256:e7257f1ef28ba17cf7c248cb8ccf6f0c6e0228ab9c315c152f9c203cd34cf6d1`
+- **Build command:** `docker compose build --pull`
+- **SBOM:** `docs/release/m14-sbom.json`, generated deterministically with
+  `python tools/generate_sbom.py` from the pinned backend manifest and frontend lockfile.
+
+The build uses digest-pinned bases and a non-root runtime user. Provenance records
+inputs and generation instructions rather than embedding a mutable image tag or a
+secret-bearing build environment.
@@ -0,0 +1,23 @@
+# M15 v2.0 certification evidence
+
+## Reference host and method
+
+Current CI reference host: Linux 7.1.4, Python 3.14.6, 12 CPUs. Certification uses a deterministic synthetic metadata workload: 100 jobs, 1,000,000 declared entries, and 100,000 cataloged backups with 10 TiB **logical** bytes per backup. It does not claim a physical 10 TiB transfer.
+
+`m15-scale-report.json` records 100,000 backup inserts in 0.57 s, a 100-row deep pagination query in 0.004209 s, and a 5.32 MB SQLite fixture.
+
+## Capability certification
+
+```text
+python tools/assert_capabilities.py --release v2.0 \
+  --include local,ssh,restore,webhook,email \
+  --exclude postgresql,mysql,tar_download
+```
+
+Passed. The released contract exposes `local`, `ssh`, `restore`, `webhook`, and `email`; PostgreSQL, MySQL, and TAR download remain disabled.
+
+SSH is private-key-only and requires a dedicated forced-SFTP chroot account; no password, shell, or remote-command path is released.
+
+## Remaining certification gates
+
+Run the full project, container E2E, SSH live integration, fault/security/leakage suites, and review the synthetic workload boundaries before a release commit is created.
@@ -0,0 +1,21 @@
+{
+  "method": "synthetic metadata certification; logical bytes are sparse and no physical 10 TiB payload is allocated",
+  "reference_host": {
+    "cpus": 12,
+    "platform": "Linux-7.1.4-arch1-1-x86_64-with-glibc2.44",
+    "python": "3.14.6"
+  },
+  "results": {
+    "backup_insert_seconds": 0.57,
+    "database_bytes": 5320704,
+    "pagination_rows": 100,
+    "pagination_seconds": 0.004209,
+    "total_seconds": 0.592
+  },
+  "workload": {
+    "backups": 100000,
+    "entries_declared": 1000000,
+    "jobs": 100,
+    "logical_bytes_per_backup": 10995116277760
+  }
+}
@@ -0,0 +1,12 @@
+# M5 Durable Execution Lifecycle Evidence
+
+- Coverage: state-transition contract and DB-enforced active-execution uniqueness
+  under concurrent enqueue; disabled/archived-job rejection; lease reclamation
+  and fencing; retry semantics; startup stale-worker reconciliation; durable
+  event revision counts; and idle worker shutdown.
+- Verification: the focused M5 suite passed: 29 tests across transition,
+  queue/lease, and worker-loss acceptance coverage.
+- Automation: `make test-fault` runs the worker-loss acceptance suite, and
+  `make check` now includes it.
+- Scope: validates M5 durable lifecycle behavior only. Snapshot publication,
+  verification, and restore remain M6 work.
@@ -0,0 +1,14 @@
+# M6 Backup, Verification, and Restore Evidence
+
+- Coverage: local full-snapshot staging; immutable SHA-256 blobs; canonical
+  Ed25519-signed manifests; manifest/blob verification; publication markers and
+  startup reconciliation; selected and dry-run restores; `fail`, `skip`, and
+  `replace` root policies; destination containment; corruption handling; and
+  restore recovery.
+- Focused acceptance: the M6 integration, publication-fault, and restore-path
+  suites pass.
+- Full verification: `make check` passed with 68 unit/contract, 37
+  integration, 7 fault, and 17 security tests, plus Ruff, mypy, frontend
+  typecheck, and frontend production build.
+- Scope: excludes M7 incremental baselines/exclusions and later scheduling,
+  remote source, encryption, retention, and UI milestones.
@@ -0,0 +1,12 @@
+# M7 Incrementals, Exclusions, and Empty-Source Safety Evidence
+
+- Coverage: ordered normalized exclusion rules, exclusion policy persisted in signed
+  manifests, incremental compatible-baseline selection, independently restorable
+  complete manifests, parent linkage, content-addressed blob reuse, and
+  empty-source opt-in.
+- Verification: M7 unit, incremental, and empty-source suites are included in
+  the passing project verification.
+- Full verification: `make check` passed with 78 unit/contract, 39
+  integration, 7 fault, and 17 security tests, plus Ruff, mypy, frontend
+  typecheck, and frontend production build.
+- Scope: scheduling remains M8 work.
@@ -0,0 +1,9 @@
+# M8 Scheduling Evidence
+
+- Coverage: five-field cron validation, IANA timezones, UTC nominal runs,
+  durable schedule CRUD, live next-run updates, misfire handling, and
+  no-overlap delivery through the execution enqueue guard.
+- Full verification: `make check` passed with 81 unit/contract, 40
+  integration, 10 fault, and 17 security tests, plus Ruff, mypy, frontend
+  typecheck, and frontend production build.
+- Scope: retention and garbage collection are M9 work.
@@ -0,0 +1,8 @@
+# M9 Retention and Garbage Collection Evidence
+
+- Coverage: union retention policies, newest-backup protection, pins, tombstones,
+  referenced-object preservation, garbage collection, and reconciliation safety.
+- Full verification: `make check` passed with 86 unit/contract, 42 integration,
+  11 fault, and 17 security tests, plus Ruff, mypy, frontend typecheck, and
+  frontend production build.
+- Scope: remote source support remains M10 work.
@@ -0,0 +1,11 @@
+# Disaster recovery runbook
+
+1. Isolate the failed host and preserve the metadata volume, repository roots, logs,
+   image digest, and master-key backup. Do not restart writers repeatedly.
+2. Provision a clean host with the same pinned image and non-root volume permissions.
+3. Restore the master key securely, restore repository roots read-only first, and
+   restore metadata from a verified backup or the passphrase-protected recovery bundle.
+4. Run `migrate current`, start only `web`, validate `/readyz` and repository
+   inspection, then start scheduler and worker one at a time.
+5. Perform and document a test restore before enabling scheduled work. Rotate secrets
+   if host compromise is possible.
@@ -0,0 +1,13 @@
+# Master and repository key runbook
+
+- Store the Compose master key outside the checkout. It must be a regular file,
+  at least 32 bytes, mode `0600`, owned by the service UID (`10001` for Compose).
+  Loss of this key destroys access to encrypted metadata secrets.
+- Back up the master key independently from metadata and repositories; do not put it
+  in an image, Compose environment variable, log, ticket, or recovery bundle.
+- Use the `admin recovery export` command with a passphrase file descriptor to create
+  a separately protected recovery bundle. Validate it on an isolated host.
+- Rotate repository data keys only with `admin repository-key rotate`; retain prior
+  recovery material until a restore drill succeeds.
+- If compromise is suspected, stop the worker, preserve evidence, rotate credentials,
+  export a fresh recovery bundle, and run a restore drill before resuming writes.
@@ -0,0 +1,19 @@
+# Metadata backup and restore runbook
+
+## Backup
+
+1. Confirm `docker compose ps` shows exactly one scheduler and one worker.
+2. For an online SQLite backup, run a host-side SQLite `.backup` against the mounted
+   `metadata.db`; do **not** copy only the main file while WAL writers run.
+3. For a filesystem copy, stop `web`, `scheduler`, and `worker` first, then retain
+   `metadata.db`, `metadata.db-wal`, and `metadata.db-shm` together.
+4. Encrypt and test the backup outside the appliance. Never place a database dump in
+   the repository or OCI image.
+
+## Restore
+
+1. Stop all runtime roles and preserve the failed metadata volume unchanged.
+2. Restore the complete SQLite backup into the metadata volume with the service user
+   ownership (UID 10001 in the supplied Compose deployment).
+3. Run `docker compose run --rm migrate current`; only start the stack when it reports
+   the expected revision. Validate `/readyz` and a read-only API request after startup.
@@ -0,0 +1,15 @@
+# Notifications runbook
+
+## Configure
+
+Create a filtered email or webhook subscription through `/api/v2/notifications/subscriptions`. Webhooks require a write-only signing secret. Configure SMTP separately at `/api/v2/notifications/email-settings`; only authenticated STARTTLS SMTP is accepted. Verify a channel with `POST /subscriptions/{id}/test` and inspect delivery/attempt history before relying on it.
+
+Filters are a nonempty set of exact catalog IDs or family wildcards such as `execution.*`; they may be narrowed by job IDs, repository IDs, or severity. The public catalog is live-events-only: every listed type is emitted by a currently available operation. Deferred channels and source capabilities have no catalog entries.
+
+## Rotate and recover
+
+Rotate webhook keys using the signing-key rotate endpoint with an idempotency key and an explicit bounded overlap. Receivers must accept both signatures during overlap, then remove the old key after expiry. A recovery bundle deliberately excludes subscriptions, SMTP settings, signing secrets, event history, and delivery attempts. Reconfigure notifications after a fresh-host recovery.
+
+## Failure handling
+
+The worker claims due deliveries with a lease. Transient errors enter bounded exponential retry; interrupted leases recover as retryable work and may send an event again. Inspect response class and redacted diagnostics in history. A terminal failed delivery can be retried manually once the destination is corrected. To stop outbound traffic, disable/archive subscriptions or stop the worker; do not delete outbox history. Rollback consists of disabling subscriptions and worker dispatch while retaining audit/outbox records for investigation.
@@ -0,0 +1,17 @@
+# Observability and alert response
+
+The proxy exposes `/livez`, `/readyz`, and Prometheus text at `/metrics`. Metrics use
+no source paths, IDs, credentials, tokens, or secret values. Runtime logs are JSON
+records with an event, timestamp, role, and request correlation ID where applicable.
+
+Alert when any of the following remains non-zero or grows:
+
+- `backup_tool_stale_execution_leases`
+- `backup_tool_failed_executions`
+- `backup_tool_corrupt_backups`
+- `backup_tool_unavailable_repositories`
+- `backup_tool_schedule_lag_seconds`
+
+Also alert on low `backup_tool_filesystem_free_bytes`. For any alert, preserve logs,
+validate `/readyz`, stop the worker before destructive repository investigation, and
+use the matching metadata, repository, key, upgrade, or disaster-recovery runbook.
@@ -0,0 +1,70 @@
+# Recovery bundle export and validation
+
+M11 recovery exports an offline, passphrase-encrypted catalog and key bundle.
+Import is a local CLI operation that reconstructs only the metadata required to
+restore existing encrypted backups; it does not reactivate backup scheduling.
+
+## Export
+
+Choose an absolute path in a trusted, non-symlinked directory. The destination
+must not already exist; export creates it with mode `0600` and never overwrites
+it.
+
+```sh
+read -r -s recovery_passphrase
+printf '\n'
+printf '%s\n' "$recovery_passphrase" | \
+  backup-tool admin recovery export \
+  --output /secure/offline/backup-tool-recovery.btrec \
+  --passphrase-fd 0
+unset recovery_passphrase
+```
+
+The passphrase is read from the inherited file descriptor. It is never a CLI
+argument. Store the resulting `BTREC` file away from the host and away from the
+live repository-key directories.
+
+## Validate
+
+Validation authenticates and decrypts the bundle, checks the versioned Argon2id
+and AES-GCM format, and verifies the included catalog/key relationships. It
+prints only a status and repository count.
+
+```sh
+read -r -s recovery_passphrase
+printf '\n'
+printf '%s\n' "$recovery_passphrase" | \
+  backup-tool admin recovery validate \
+  --input /secure/offline/backup-tool-recovery.btrec \
+  --passphrase-fd 0
+unset recovery_passphrase
+```
+
+Wrong passphrases, tampering, malformed headers, unsupported KDF parameters,
+and invalid encrypted payloads intentionally produce the same validation error.
+Do not use a failed validation result to diagnose which of those conditions
+occurred.
+
+## Fresh-host import
+
+Before importing, run migrations on the replacement host and configure its
+repository allowlist to include the surviving repository directory. The
+repository must pass normal metadata/path inspection. The replacement metadata
+database must be current and otherwise empty; import rejects a non-empty
+destination and any existing/conflicting key files.
+
+```sh
+backup-tool migrate upgrade
+read -r -s recovery_passphrase
+printf '\n'
+printf '%s\n' "$recovery_passphrase" | \
+  backup-tool admin recovery import \
+  --input /secure/offline/backup-tool-recovery.btrec \
+  --passphrase-fd 0
+unset recovery_passphrase
+```
+
+Import installs signing and data keys with restrictive modes, restores the
+repository/source/job/execution/backup catalog needed for restore, and marks
+sources unavailable plus jobs archived and disabled. Reconfigure sources and
+explicitly create or enable new jobs before taking another backup.
@@ -0,0 +1,12 @@
+# Repository recovery runbook
+
+1. Stop `worker` before inspecting or repairing a repository; never edit a live
+   repository behind an active lease.
+2. Preserve the repository directory and its metadata volume before remediation.
+3. Verify repository state through the operator API and verify individual backups
+   before any restore. Treat a corrupt verification result as an incident, not a
+   deletion request.
+4. Mount replacement repository roots at the same allowlisted path, restore metadata,
+   then start `migrate`, `web`, `scheduler`, and finally `worker`.
+5. Keep archived repositories mounted until retention and restore obligations expire.
+   Do not remove manifests or blobs manually.
@@ -0,0 +1,19 @@
+# SSH sources
+
+SSH sources require a dedicated account confined by an OpenSSH `ChrootDirectory`
+and `ForceCommand internal-sftp`. The chroot directory is root-owned; writable
+content is below it. Disable passwords, keyboard-interactive authentication,
+shells, PTYs, TCP/X11/agent forwarding, and tunnelling. Configure the source
+root as `/` only.
+
+Generate a dedicated unencrypted Ed25519, ECDSA, or RSA-3072+ client key and
+store it through the write-only `ssh_private_key` secret endpoint. Do not put a
+key, passphrase, password, command, agent path, or key file path in source
+configuration. Pin the server's exact OpenSSH public host key (`algorithm
+base64`) before probing. On host-key rotation, obtain the replacement through
+an out-of-band administrative channel, update the source pin, then probe.
+
+The server administrator controls mutable content inside the chroot. The client
+rejects traversal names, symlinks, special files, changed files, and configured
+resource-limit overflows, but cannot claim atomic no-follow behavior against a
+maliciously changing filesystem inside that server-controlled boundary.
@@ -0,0 +1,12 @@
+# Upgrade and rollback runbook
+
+1. Record the running image digest and take a tested metadata backup plus repository
+   recovery evidence before changing the image.
+2. Pull/build the pinned image, then run `docker compose run --rm migrate upgrade`.
+   Do not start web, scheduler, or worker against an unverified schema.
+3. Start the stack, wait for `/readyz`, and inspect `/metrics` for stale leases,
+   schedule lag, unavailable repositories, and corrupt backups.
+4. If migration fails, stop and restore the prior metadata backup and matching image;
+   do not attempt to downgrade an unknown partially migrated database in place.
+5. Preserve migration logs and verify a representative backup restore before closing
+   the change.
@@ -0,0 +1,9 @@
+# Notification security policy (M12)
+
+Webhook callbacks accept absolute `http` and `https` URLs only. HTTP is an approved compatibility option, not a relaxation of egress controls. URLs with credentials, fragments, literal IP addresses, or ports outside 80/443/8080/8443 are rejected. Immediately before every request the worker resolves all A/AAAA answers; any non-global answer rejects the whole destination. The transport connects only to an approved answer, verifies the connected peer address, disables proxy environment use, preserves HTTPS SNI/certificate validation, and rejects redirects.
+
+Webhook bodies are canonical JSON event envelopes. `X-Backup-Event-ID` is stable over retries. Each attempt supplies the versioned HMAC-SHA-256 timestamp/body input and one `X-Backup-Signature` header per active or overlap key. Key IDs and monotonically increasing subscription-local versions identify keys; an old key is retained only until its configured overlap expiry. Key material is encrypted in `secrets` and is never returned, logged, exported in recovery bundles, or included in audit/event payloads.
+
+SMTP is configured as one write-only password-backed singleton. Delivery performs EHLO, verified STARTTLS, a second EHLO, and SMTP AUTH; any inability to do this fails closed. Message content is a compact event summary with no attachments, Bcc, paths, raw exception text, credentials, or full webhook payload.
+
+Delivery is at-least-once. A worker crash after a send but before recording success can lead to a duplicate; consumers must deduplicate using the event ID. Redirects, malformed destinations, and SSRF validation failures are terminal. Connect/read transport failures, SMTP transient failures, and HTTP 408/425/429/5xx use bounded retry. Deployment egress controls are defense in depth, not a substitute for this policy.
@@ -0,0 +1,35 @@
+# Repository encryption threat model
+
+## Status
+
+This document defines the M11 repository-encryption boundary. Encrypted repository
+creation remains disabled until the recovery host-loss acceptance test passes.
+
+## Confidential data
+
+For an `aes-256-gcm` repository, blob contents and signed manifest contents are
+AEAD-encrypted. Data keys and signing private keys are never stored in repository
+metadata, SQLite, logs, command arguments, environment output, or API responses.
+Recovery bundles are separately passphrase-encrypted.
+
+## Intentional leakage
+
+The v1 content-addressed layout retains plaintext SHA-256 blob names. Encryption
+therefore leaks blob equality, object count, repository layout, manifest/backup
+identifiers, and ciphertext sizes derived from plaintext sizes. It does not claim
+to hide those values, a compromised running host with unlocked keys, source data
+while it is read, or a recovery-bundle passphrase.
+
+## Key lifecycle
+
+Encryption uses distinct data and signing keys. Rotation creates a new data-key
+epoch for subsequent objects and retains historical epochs so existing backups
+remain verifiable and restorable. It neither changes immutable repository policy
+nor rewrites existing objects.
+
+## Recovery boundary
+
+Recovery export, validation, and import are local CLI operations only. A bundle
+contains the required key material and an authenticated catalog; passphrases are
+not accepted on command lines. Bundle parsing must reject malformed or unsupported
+parameters without revealing whether a passphrase, key, or ciphertext was wrong.