backup-tool/docs/superpowers/plans/2026-07-27-backup-tool-v2-implementation.md

# Backup Tool v2 Implementation Plan

**Status:** Ready for execution after branch kickoff  
**Date:** 2026-07-27  
**Design authority:** [`../specs/2026-07-27-backup-tool-reimplementation-design.md`](../specs/2026-07-27-backup-tool-reimplementation-design.md)  
**Replaces:** [`2026-05-11-backup-tool-implementation.md`](./2026-05-11-backup-tool-implementation.md) for all v2 work

## 1. Reason for Existence

This plan converts the decision-closed design into dependency-ordered, test-first, independently reviewable milestones. It prevents framework-first rebuilding: repository format, publication safety, restore, and crash recovery become executable contracts before scheduling, optimization, UI breadth, or v2.1 adapters.

A milestone is complete only when its named behavior tests pass, its failure path is proved, its design decisions and acceptance invariants are traceable, and the working tree contains no unrelated changes.

## 2. Scope and Guardrails

- V2 is a clean rewrite. It MUST NOT contain `/api/v1`, legacy DB readers, importers, converters, timestamp-directory parsers, or v1 payload access.
- Preserve v1 only as a Git tag/branch for historical reference before deleting executable v1 code. No v1 file or test is adapted into a compatibility layer.
- V2.0 includes local and SSH/SFTP sources, local repositories, restore, optional repository encryption, recovery bundles, webhooks, and email.
- PostgreSQL, MySQL, and generated TAR download are v2.1-only capabilities and MUST remain absent from v2.0 OpenAPI, UI, image, and capability metadata.
- Certified topology is one node, SQLite/WAL, one scheduler process, one worker process, and one or more web processes. Web processes MUST perform no backup, restore, scheduler, or repository I/O.
- Every task follows red → green → refactor. Preserve failing-then-passing command output in PR/CI evidence.
- Do not start a dependent milestone until its dependency gate is green. Do not split an atomic set from §6 across releases.

## 3. Target Project Layout

```text
backend/
  pyproject.toml
  alembic/versions/
  src/backup_tool/
    api/{app,deps,errors,pagination,etag,idempotency,sse,routers/,schemas/}
    adapters/{base,local,sftp,postgres,mysql}.py
    db/{engine,models,repositories}.py
    domain/{manifest,paths,exclusions,transitions,cron,retention}.py
    execution/{enqueue,claims,leases,runner,retry,cancellation,progress}.py
    notifications/{events,dispatcher,webhook,email,retry}.py
    observability/{logging,metrics,health}.py
    repository/{layout,blob_store,manifest_store,staging,publisher,verifier,restore,reconcile,gc,encryption}.py
    security/{auth,secrets,redaction,repository_crypto,recovery_bundle,ssrf}.py
    services/{sources,jobs,repositories,backups,restores}.py
    scheduler/service.py
    {cli,config,ids,clock}.py
frontend/src/
  api/generated/
  app/
  pages/
  components/
contracts/repository/v1/
openapi/v2.json
tests/{unit,contract,integration,fault,security,e2e,performance,fixtures}/
tools/{export_openapi,forbidden_v1_scan,leakage_scan,check_traceability,run_scale_certification}.py
docs/{runbooks,security,release}/
```

Module rules: domain code imports no FastAPI, SQLAlchemy, filesystem, Paramiko, SMTP, or HTTP client; adapters and repository code implement ports; services own use cases; routers translate HTTP only; ORM and API schemas never act as domain models.

## 4. Dependency Spine

```text
M0 contracts
 └─ M1 runtime/DB ─ M2 auth/API
        └─ M3 repository ─ M4 sources/jobs ─ M5 execution
             └─ M6 full backup/verify/restore
                  ├─ M7 incremental/exclusions
                  ├─ M8 scheduling
                  ├─ M9 retention/GC/reconcile
                  ├─ M10 SSH/SFTP
                  └─ M11 encryption/recovery
                       └─ M12 notifications ─ M13 UI/API ─ M14 operations ─ M15 v2.0
M15 ─ M16 PostgreSQL ─ M17 MySQL ─ M18 TAR/download ─ v2.1
```

M7-M11 MAY be developed on separate branches after M6, but merge in dependency order and never use concurrent writers in one worktree.

## 5. Milestones

### M0 — Freeze protocol and test foundation

**Depends on:** approved design.  
**Deliver:** lock Python/Node/tool versions; create package/test layout and `Makefile`; commit canonical repository/manifest JSON schemas, golden valid/invalid fixtures, normalized-path vectors, state-transition table, error-code catalog, capability schema, and deterministic fault-point names. Tag/archive v1, then remove `backend/app`, `backend/backup`, old tests, and handwritten `frontend/src/api/client.ts`.  
**Verify:** `.venv/bin/python tools/forbidden_v1_scan.py . && .venv/bin/python -m pytest tests/contract/test_repository_format.py tests/contract/test_no_v1.py -q` (run `make setup` first on a clean checkout)
**Commit:** `chore(v2): establish protocol and test foundation`

### M1 — Runtime, UUIDv7, SQLite/WAL, and Alembic baseline

**Depends on:** M0.  
**Deliver:** typed config; canonical source/repository/restore allowlists; key-file owner/mode checks; UUIDv7 IDs; UTC clock port; role CLIs; SQLite foreign keys/WAL/busy timeout; complete explicit schema, constraints, indexes, and first hand-reviewed migration. Startup checks migration state and never calls `create_all`.  
**Verify:** `cd backend && python -m alembic upgrade head && python -m alembic downgrade base && python -m alembic upgrade head && python -m pytest ../tests/unit/test_config.py ../tests/integration/test_migrations.py -q`  
**Commit:** `feat(v2): add validated runtime and metadata store`

### M2 — Admin setup, authentication, secrets, audit, and API conventions

**Depends on:** M1.  
**Deliver:** guarded one-admin setup; Argon2id; secure session/CSRF; hashed scoped API tokens; envelope-encrypted write-only secrets; request IDs; append-only audit; RFC 9457 problems; stable cursor pagination/sort; ETags; request-digest idempotency; truthful `/livez` and `/readyz`.  
**Verify:** `python -m pytest tests/unit/test_redaction.py tests/contract/test_api_conventions.py tests/integration/test_auth.py tests/security/test_auth.py -q && python tools/leakage_scan.py`  
**Commit:** `feat(v2): secure identity secrets and API conventions`

### M3 — Initialize and inspect repositories

**Depends on:** M2.  
**Deliver:** allowlisted local repository init; `repository.json`; immutable compression/encryption choice; canonical JSON; plaintext SHA-256 blob identity; capacity probe; minimum free bytes/percent; safe partial-init cleanup; path/symlink containment. Encryption interfaces exist, but encrypted creation remains capability-disabled until M11.  
**Verify:** `python -m pytest tests/contract/test_repository_format.py tests/integration/test_repository_init.py tests/security/test_repository_paths.py -q`  
**Commit:** `feat(v2): initialize immutable content repositories`

### M4 — Typed local sources and repository-targeted jobs

**Depends on:** M3.  
**Deliver:** adapter protocol; normalized POSIX entries; bounded local streaming; symlink/cross-filesystem/special-file policy; source probes; typed source/job CRUD; immutable source kind; archive semantics; repository IDs instead of destination strings; enabled, mode, exclusions, retention, and `allow_empty`.  
**Verify:** `python -m pytest tests/unit/test_paths.py tests/unit/test_exclusions.py tests/integration/test_local_source_api.py tests/security/test_source_containment.py -q`  
**Commit:** `feat(v2): add local sources and repository jobs`

### M5 — Durable execution lifecycle

**Depends on:** M4.  
**Deliver:** monotonic states; `202` execution resource; `409` active execution; DB-enforced one nonterminal execution/job; disabled-job check inside enqueue; leases, heartbeats, retries using one execution ID, attempts, cancellation, reason codes, redacted progress/logs, SSE with polling fallback, graceful shutdown, stale-worker fencing, startup reconciliation.  
**Verify:** `python -m pytest tests/unit/test_execution_transitions.py tests/integration/test_queue_leases.py tests/fault/test_worker_loss.py -q`  
**Commit:** `feat(v2): add durable leased execution lifecycle`

### M6 — Atomic full backup, verification, and restore

**Depends on:** M5.  
**Deliver:** unique staging; capacity preflight; adapter-only reads; chunked hashing; immutable blob install; canonical complete manifest; fsync/atomic rename where supported; object verification; metadata visibility only after durable publish; reconciliation marker; metadata/full verification; durable dry-run/selected restore; `fail|skip|replace`; destination containment; result manifest; integrity states.  
**Verify:** `python -m pytest tests/integration/test_full_backup_restore.py tests/fault/test_publication_crashes.py tests/security/test_restore_paths.py -q`  
**Commit:** `feat(v2): publish verified snapshots and durable restores`

### M7 — Exclusions, independent incrementals, and empty-source safety

**Depends on:** M6.  
**Deliver:** versioned gitignore semantics shared by every adapter; compatible-baseline selection; requested/effective mode; `baseline_missing`; blob reuse; declared metadata-trust rules; independently restorable complete manifests; empty-source opt-in and vanished-source confirmation.  
**Verify:** `python -m pytest tests/unit/test_exclusions.py tests/integration/test_incremental.py tests/integration/test_empty_source.py -q`  
**Commit:** `feat(v2): add safe deduplicated snapshots`

### M8 — Deterministic durable scheduling

**Depends on:** M6.  
**Deliver:** one schedule/job; semantic five-field cron; IANA timezone; next run/last result; `(schedule_id, nominal_utc)` uniqueness; immediate create/update/delete synchronization; 15-minute grace; coalesce/no-overlap; exact DST gap/fold behavior; disabled job/schedule rejection; same enqueue service as manual execution.  
**Verify:** `python -m pytest tests/unit/test_cron_dst.py tests/integration/test_scheduler_live_sync.py tests/fault/test_schedule_delivery.py -q`  
**Commit:** `feat(v2): schedule idempotent timezone-aware executions`

### M9 — Retention, tombstones, reconciliation, and safe GC

**Depends on:** M6 and M7.  
**Deliver:** union policy for keep-last/days/daily/weekly/monthly/pins; newest protection; preview/apply; archive without payload deletion; tombstone/audit; grace plus second reference scan; `deletion_failed`; confirmed purge/report; bidirectional reconciliation; unknown-object quarantine; shared-blob safety.  
**Verify:** `python -m pytest tests/unit/test_retention.py tests/integration/test_gc.py tests/fault/test_deletion_failure.py tests/integration/test_reconcile.py -q`  
**Commit:** `feat(v2): add retention reconciliation and safe GC`

### M10 — Strict SSH/SFTP parity

**Depends on:** M6 and M7.  
**Deliver:** typed remote config; write-only auth; pinned known-host key; fail-closed probe and execution; bounded SFTP enumeration/streaming; timeouts/backpressure; common paths/exclusions; source-consistency metadata; transient classification; no local traversal fallback.  
**Verify:** `docker compose -f tests/compose.integration.yaml up -d sshd && python -m pytest tests/integration/test_sftp_backup_restore.py tests/security/test_sftp_host_keys.py -q`  
**Commit:** `feat(v2): add pinned-host SFTP backups`

### M11 — Optional repository encryption and offline recovery

**Depends on:** M6.  
**Deliver:** AEAD stored objects; associated metadata; key IDs and rotation; immutable policy; explicit confidentiality/equality-leakage threat model; separately passphrase-protected recovery bundle; non-disclosing validation; fresh-host metadata/key recovery and encrypted restore. Enable encrypted repository creation only after the host-loss test passes.  
**Verify:** `python -m pytest tests/integration/test_encrypted_repository.py tests/integration/test_recovery_bundle.py tests/security/test_crypto_leakage.py -q`  
**Commit:** `feat(v2): encrypt repositories and recover keys offline`

### M12 — Signed webhooks and auditable email

**Depends on:** M2, M5, and M7-M11.  
**Deliver:** complete operational event catalog covering execution, schedule, retention/GC, reconciliation, SSH, encryption/recovery, restore, verification, capacity, and security outcomes; typed filters/subscriptions; stable event IDs; durable outbox/deliveries/attempts; webhook signature/version and rotation; at-least-once bounded retry; SMTP transient retry; manual test/retry; history; rate limits; write-only credentials; redirect/DNS/IP/private-network/rebinding SSRF policy.  
**Verify:** `python -m pytest tests/contract/test_notification_contract.py tests/integration/test_all_operational_events_deliver.py tests/integration/test_notifications.py tests/fault/test_notification_retries.py tests/security/test_webhook_ssrf.py -q`  
**Commit:** `feat(v2): add webhook and email delivery`

### M13 — OpenAPI-generated client and operator UI

**Depends on:** M7-M12.  
**Deliver:** deterministic committed OpenAPI; generated TypeScript client only; drift CI; Setup/Login, Dashboard, Sources, Repositories, Jobs/Schedule, Executions/SSE, Backups/Verify/Restore/Delete Preview, Security/Recovery, Notifications/History, and Audit. Implement loading, empty, validation, partial failure, reconnect, session expiry, responsive, keyboard, focus, contrast, reduced-motion, and screen-reader states. Hide v2.1 controls completely.  
**Verify:** `python tools/export_openapi.py --check openapi/v2.json && npm --prefix frontend run api:generate && git diff --exit-code -- openapi/v2.json frontend/src/api/generated && npm --prefix frontend test -- --run && npm --prefix frontend run build && npx --prefix frontend playwright test`  
**Commit:** `feat(v2): ship generated-client operator workflows`

### M14 — Production roles, observability, recovery, and packaging

**Depends on:** M13.  
**Deliver:** one pinned non-root OCI image with `web|scheduler|worker|migrate|admin`; same-origin proxy; one scheduler and worker; no reload or embedded roles; role-aware readiness; structured logs; required metrics/alerts; metadata/repository/key/upgrade/disaster runbooks; restart persistence; safe shutdown; SBOM/provenance; v2.0 image contains no DB dump clients.  
**Verify:** `docker compose build --pull && docker compose run --rm migrate upgrade && docker compose up -d && python -m pytest tests/e2e/test_compose_v2.py tests/security/test_container.py -q && docker compose down`  
**Commit:** `build(v2): package isolated roles and recovery operations`

### M15 — Certify and release v2.0

**Depends on:** M0-M14.  
**Deliver:** reference-host definition; workloads for 100 jobs, 1M entries or 10 TiB logical/backup, and 100k cataloged backups; throughput/CPU/memory/DB/lag/pagination/restore/verify evidence; bounded-memory and accepted-regression gates; complete fault/security/leakage/E2E evidence; truthful capabilities/docs; removal of all v2.1 controls.  
**Verify:** `python tools/run_scale_certification.py --jobs 100 --entries 1000000 --backups 100000 && python tools/assert_capabilities.py --release v2.0 --include local,ssh,restore,webhook,email --exclude postgresql,mysql,tar-download`  
**Commit:** `release: certify backup-tool v2.0.0`

### M16-M18 — V2.1 vertical slices

- **M16 PostgreSQL (depends on M15):** secret-safe bounded `pg_dump`, TLS, tool/server matrix, full-only manifest, cancellation/error classification, typed source CRUD/probe, capability metadata, OpenAPI/generated client/UI form, and v2.1 OCI dump-client packaging land atomically → verify: `python -m pytest tests/contract/test_postgres_source_api.py tests/integration/test_postgres_adapter.py tests/security/test_dump_secrets.py tests/security/test_v21_container.py -q && make check-openapi && npm --prefix frontend run build && python tools/assert_capabilities.py --release v2.1 --include postgresql` → `feat(v2.1): add PostgreSQL logical snapshots`.
- **M17 MySQL (depends on M16):** secret-safe bounded `mysqldump`, TLS/locking consistency, compatibility matrix, full-only manifest, typed source CRUD/probe, capability metadata, OpenAPI/generated client/UI form, and OCI client verification land atomically → verify: `python -m pytest tests/contract/test_mysql_source_api.py tests/integration/test_mysql_adapter.py tests/security/test_dump_secrets.py tests/security/test_v21_container.py -q && make check-openapi && npm --prefix frontend run build && python tools/assert_capabilities.py --release v2.1 --include postgresql,mysql` → `feat(v2.1): add MySQL logical snapshots`.
- **M18 TAR/download and v2.1 release (depends on M17):** verify every blob/tag before streaming manifest-derived safe entries; traversal/link/device/disconnect tests; generated API/client/UI capability; then rerun every v2.0 gate, both DB matrices, OpenAPI drift, Compose/E2E, leakage, and reference-host regression → verify: `make check && make test-integration && make test-fault && make test-security && make test-e2e && make check-openapi && make check-traceability && make check-v1-absent && python -m pytest tests/integration/test_postgres_adapter.py tests/integration/test_mysql_adapter.py tests/security/test_dump_secrets.py tests/integration/test_tar_download.py tests/security/test_tar_paths.py -q && python tools/run_scale_certification.py --regression-from docs/release/v2.0-evidence.md && python tools/assert_capabilities.py --release v2.1 --include local,ssh,postgresql,mysql,restore,tar-download,webhook,email && npx --prefix frontend playwright test` → `release: certify backup-tool v2.1.0`.

## 6. Atomic Sets and Stop Rules

The following land as indivisible reviewed sets: publication + reconciliation marker + crash matrix; execution state + uniqueness + leases + cancellation + recovery; auth + secrets + redaction + audit; restore record + containment + verification + atomic placement; retention + tombstone + reference scans + quarantine; scheduler parser + uniqueness + DST/misfire + live sync; encryption + recovery bundle + host-loss restore; notification outbox + signing + SSRF + SMTP + retry/history; OpenAPI + generated client + drift check + corresponding UI.

Stop and request an ADR/design amendment if implementation requires changing manifest canonicalization, blob identity, external IDs, process topology, delivery guarantee, schedule semantics, encryption confidentiality, retention precedence, v2.0/v2.1 capability split, or any D1-D21 decision. Stop a milestone when a failure test is nondeterministic, a destructive test can reach operator data, a zero-test selection can pass, or a secret/path canary appears in any output sink.

## 7. Traceability

| Decisions | Owner and proof | Decisions | Owner and proof |
| --- | --- | --- | --- |
| D1 | M0-M2: UUID/API/no-v1 contract | D2 | M2: concurrent setup test |
| D3 | M1, M5, M14: role/process test | D4 | M1, M14: pragmas/topology test |
| D5 | M2, M5: idempotency/lease tests | D6 | M3-M4: repository-only job/path tests |
| D7 | M0, M3, M6: manifest/independent restore | D8 | M3, M11: immutable policy/AEAD tests |
| D9 | M0, M4, M7, M10: shared matcher corpus | D10 | M8: cron/timezone/DST tests |
| D11 | M4-M5, M8: disabled enqueue tests | D12 | M4, M9: archive/restorability tests |
| D13 | M13: generated-client zero-diff test | D14 | M1: Alembic/no-`create_all` test |
| D15 | M10, M15-M17: capability gates | D16 | M15: certified reference-host report |
| D17 | M11: fresh-host recovery drill | D18 | M0, M15: forbidden-v1 scan/runtime test |
| D19 | M13, M15, M18: capability/route gates | D20 | M12-M13, M15: channel/event E2E |
| D21 | M12: stable-ID retry/history tests | — | — |

| Invariants | Owner and proof | Invariants | Owner and proof |
| --- | --- | --- | --- |
| I1 | M6: publication crash matrix | I2 | M8: replayed occurrence uniqueness |
| I3 | M5: concurrent active-job starts | I4 | M5-M6: transition/restart matrix |
| I5 | M10, M16-M17: adapter-spy tests | I6 | M6-M7: parent-free restore corpus |
| I7 | M9: retain/GC/restore property tests | I8 | M4, M9: archive inventory test |
| I9 | M9: deletion failure/retry test | I10 | M2, M11-M12, M14: canary scan |
| I11 | M10: host-key fail-before-read test | I12 | M7, M10, M16-M17: matcher parity |
| I13 | M5, M8: all-trigger disable tests | I14 | M8: live-sync/misfire/DST tests |
| I15 | M0, M15: protected-v1 no-access test | I16 | M13, M18: OpenAPI/client zero diff |
| I17 | M6: full restore matrix | I18 | M6, M11, M18: corruption matrix |
| I19 | M7: empty/vanished-source test | I20 | M3-M4, M6, M10, M18: containment |
| I21 | M11: recovery-bundle host-loss drill | I22 | M12: webhook/email retry history |

`tools/check_traceability.py` MUST require every D1-D21 and I1-I22 row to name a milestone, test path, runnable command, and immutable evidence artifact.

## 8. Whole-Project Verification

```bash
make test-fast                 # unit + contract; nonempty marker assertion
make test-integration          # temp DB/repository roots only
make test-fault                # deterministic failpoints
make test-security             # canaries, containment, auth, SSRF
make test-e2e                  # production-like Compose
make check-openapi             # export, generate, zero diff, type/build
make check-traceability        # D1-D21 and I1-I22 evidence links
make check-v1-absent           # static + runtime protected-v1 fixture
make check                     # format, lint, type, all non-scale tests
git diff --check
git status --short
```

Fault/destructive tests MUST allocate hermetic temporary source, repository, restore, DB, key, SMTP, and webhook targets. Test teardown MUST reject paths outside its temp root. CI MUST assert each selected marker collected at least one test.

## 9. Release Gates

**V2.0:** M0-M15 green; every D1-D21 item applicable to v2.0 and I1-I22 has immutable evidence; no secret canary; fault matrix green; OpenAPI/client diff clean; local and SSH restore corpus green; encrypted fresh-host recovery green; notifications retry across process restart; Compose roles/non-root/persistence green; scale report approved; capabilities exclude PostgreSQL, MySQL, and TAR download.

**V2.1:** v2.0 gates remain green; PostgreSQL/MySQL compatibility matrices pass; dump credentials never enter argv/logs; TAR corruption/path/disconnect tests pass; generated API/client/UI exposes only installed capabilities; performance regression remains within the accepted threshold.

## 10. Execution Discipline

Use one feature branch/worktree per milestone. Keep commits small within the milestone but merge only when its atomic gate is closed. Conventional commit title matches the milestone; PR body links RED/GREEN evidence, design IDs, invariants, commands with exit codes, changed files, residual risks, and rollback. Run a fresh-context security/reliability review after M2, M6, M9, M11, M12, M14, and before each release.

**Next action:** start M0 by creating the contract fixtures and forbidden-v1 test before scaffolding application modules.

## 11. Plan Verification

```bash
python - <<'PY'
from pathlib import Path
p = Path('docs/superpowers/plans/2026-07-27-backup-tool-v2-implementation.md')
s = p.read_text()
for marker in ['M0', 'M15', 'M18', 'D1-D21', 'I1-I22', 'verify:', 'Release Gates']:
    assert marker in s, marker
assert s.count('**Verify:**') >= 16
print('v2 implementation plan: OK')
PY
```