From f97daa35d09eead6a254d91e1551b9b5f9ec20ac Mon Sep 17 00:00:00 2001 From: Alex Blank Date: Fri, 31 Jul 2026 14:34:34 +0200 Subject: [PATCH] docs: rewrite v2 operator documentation --- README.md | 279 +++++++++++++++---------------------------------- docs/README.md | 42 ++++++++ 2 files changed, 125 insertions(+), 196 deletions(-) create mode 100644 docs/README.md diff --git a/README.md b/README.md index a46b737..b173e42 100644 --- a/README.md +++ b/README.md @@ -1,232 +1,119 @@ -# Backup Tool +# Backup Tool v2 -A modern backup management application with a FastAPI backend and React frontend. +> A self-hosted, integrity-first backup appliance for local and hardened SSH sources. -## Features +Backup Tool creates signed, content-addressed backups, supports encrypted repositories and offline recovery, and runs as isolated web, scheduler, worker, migration, and admin roles. v2.0 supports **local** and **SSH** sources, local repositories, restore, encrypted recovery, email, and signed webhooks. -- **Source Type**: Local filesystem -- **Backup Strategies**: Full and incremental backups -- **Scheduled Backups**: Cron-based scheduling with APScheduler -- **Retention Policies**: Count-based and days-based backup retention -- **Web Dashboard**: React-based UI for managing backups -- **REST API**: Full REST API for programmatic access +## What is supported -## Architecture +- Signed full and incremental backups, restore, retention, verification, and recovery. +- AES-256-GCM repository encryption with key epochs and offline Argon2id recovery bundles. +- Local sources and SSH sources using **private-key-only, pinned-host-key, forced-SFTP chroot** access. +- Durable, at-least-once webhook and STARTTLS SMTP notifications. +- A generated OpenAPI TypeScript client and operator UI. -``` -backup-tool/ -├── backend/ # FastAPI backend -│ ├── app/ # FastAPI application -│ │ ├── routers/ # API endpoints -│ │ ├── models.py # SQLAlchemy models -│ │ ├── schemas.py # Pydantic schemas -│ │ └── main.py # Application entry point -│ ├── backup/ # Backup engine -│ │ ├── adapters/ # Local source adapter -│ │ ├── engine.py # Backup execution engine -│ │ ├── scheduler.py # Job scheduler -│ │ └── retention.py # Retention policies -│ └── requirements.txt # Python dependencies -├── frontend/ # React frontend -│ ├── src/ # Source code -│ └── package.json # Node dependencies -└── docs/ # Documentation +Never use an SSH source with a shell, password, agent forwarding, remote commands, or an unchrooted account. See [SSH source requirements](docs/runbooks/ssh-sources.md). + +## Quick local verification + +Prerequisites: Docker Compose, Python 3.12–3.14, Node/npm, and a working virtual environment. + +```sh +make setup +make check +make test-e2e +make test-ssh-integration ``` -## Quick Start +`make test-e2e` creates temporary test-only keys and source data, starts the complete Compose stack, checks readiness/metrics/restart behavior, and tears it down. `make test-ssh-integration` builds a separate forced-SFTP fixture and verifies key-authenticated SSH probe, backup, verification, and restore. -### Option 1: Docker (Recommended) +## Run locally with Docker Compose -The easiest way to run the backup tool is using Docker Compose: +The production Compose topology binds the proxy to `127.0.0.1:8080`; put a TLS reverse proxy in front of it for remote access. -```bash -# Start the backend +```sh +mkdir -p secrets sources +head -c 32 /dev/urandom > secrets/master.key +chmod 600 secrets/master.key +# The container runs as UID 10001; grant that UID read access to the key. +chown 10001:10001 secrets/master.key + +# Put test source data below ./sources, then start the isolated roles. +docker compose build --pull +docker compose run --rm migrate upgrade docker compose up -d - -# Start with frontend (production) -docker compose --profile prod up -d - -# Start with frontend (development with hot reload) -docker compose --profile dev up -d +curl -fsS http://127.0.0.1:8080/readyz +curl -fsS http://127.0.0.1:8080/metrics ``` -Access the application: +On a fresh non-loopback deployment, set `BACKUP_TOOL_PUBLIC_BASE_URL` to the externally reachable HTTPS URL and set `BACKUP_TOOL_BOOTSTRAP_SECRET` before first setup. Never expose a fresh setup endpoint without bootstrap protection. -- Backend API: -- Frontend: -- API Docs: +Stop the local stack with: -### Option 2: Manual Setup - -#### Prerequisites - -- Python 3.11+ -- Node.js 18+ -- PostgreSQL or MySQL (for database backups) - -#### Backend Setup - -```bash -cd backend -python -m venv venv -source venv/bin/activate # On Windows: venv\Scripts\activate -pip install -e ".[dev]" - -# Run the server -uvicorn app.main:app --reload --port 8000 +```sh +docker compose down +# Add --volumes only when intentionally discarding local metadata and repositories. ``` -#### Frontend Setup +## Operator workflow -```bash -cd frontend -npm install -npm run dev -``` +1. Open `http://127.0.0.1:8080` and create the first administrator. +2. Create a local or encrypted repository. +3. Create a local source, or configure a hardened SSH source as documented in [SSH sources](docs/runbooks/ssh-sources.md). +4. Create a job, run a probe, enqueue a backup, verify it, then perform a test restore. +5. Configure notification subscriptions and test them before relying on delivery. +6. Export and validate a recovery bundle after encrypted repository creation and each key rotation. -#### Production Build - -```bash -cd frontend -npm run build - -cd ../backend -uvicorn app.main:app --host 0.0.0.0 --port 8000 -``` - -## API Documentation - -Once the backend is running, visit: - -- Swagger UI: -- ReDoc: +The browser exposes recovery status and the CLI procedure only; it never transfers recovery bundles or passphrases. ## Configuration -### Environment Variables +Compose supplies the core runtime variables. The important host paths are: -| Variable | Description | Default | -| ---------- | ------------- | --------- | -| `DATABASE_URL` | Database connection string | `sqlite+aiosqlite:///./backup_tool.db` | -| `CORS_ORIGINS` | Comma-separated allowed CORS origins | `http://localhost:3000` | -| `SQL_ECHO` | Enable SQL query logging | `false` | -| `BACKUP_STORAGE_PATH` | Path for storing backups | `/app/backups` | +| Setting | Compose value | Purpose | +| --- | --- | --- | +| `BACKUP_TOOL_MASTER_KEY_FILE` | `/run/backup-tool-secrets/master.key` | Service-owned `0600` master key | +| `BACKUP_TOOL_SOURCES_DIR` | `./sources` | Read-only local-source bind mount | +| `BACKUP_TOOL_PUBLIC_BASE_URL` | `http://localhost:8080` | External URL and bootstrap policy | +| `BACKUP_TOOL_PORT` | `8080` | Loopback proxy port | -### Docker-Specific Configuration +Use absolute, allowlisted paths for repositories, sources, and restores. The service rejects unsafe paths, symlinks where they violate the contract, missing key files, and non-current schemas. -When running with Docker Compose, the following volumes are mounted: +## Operations and security -- `backup-data`: Persisted SQLite database at `/app/data` -- `backup-storage`: Backup files at `/app/backups` - -### Development vs Production - -**Development Mode** (`docker compose --profile dev up -d`): - -- Backend hot reload enabled -- Frontend Vite dev server with HMR -- Source code mounted as volumes - -**Production Mode** (`docker compose --profile prod up -d`): - -- Optimized frontend build served via nginx -- Backend without reload -- Static assets compiled - -## Troubleshooting - -### Docker Issues - -**Port already in use** - -```bash -# Check what's using port 8000 -lsof -i :8000 - -# Or use different ports in docker-compose.yml -``` - -**Container fails to start** - -```bash -# Check logs -docker logs backup-tool-backend - -# Rebuild with no cache -docker compose build --no-cache -``` - -**Permission denied on data directory** - -```bash -# Fix permissions -docker compose exec backend chown -R backup-tool:backup-tool /app/data -``` - -**Tests fail in Docker** -Tests require development dependencies. Install with: - -```bash -docker compose exec backend pip install -e ".[dev]" -``` - -### Manual Setup Issues - -**Python version incompatibility** -Ensure Python 3.11+ is installed: - -```bash -python --version -``` - -**Node modules conflicts** - -```bash -cd frontend -rm -rf node_modules package-lock.json -npm install -``` - -## Deployment - -### Docker Deployment - -1. Clone the repository -2. Run `docker compose --profile prod up -d` -3. Access at - -### Manual Deployment - -1. Install Python 3.11+ and Node.js 18+ -2. Install backend: `cd backend && pip install -e ".[prod]"` -3. Build frontend: `cd frontend && npm run build` -4. Start backend: `cd backend && uvicorn app.main:app --host 0.0.0.0` - -### Production Considerations - -- Use a reverse proxy (nginx, traefik) for SSL termination -- Set strong credentials for database sources -- Configure backup retention policies -- Monitor disk usage for backup storage -- Use `docker compose -f docker-compose.yml up -d` for production without dev tools +- [Observability](docs/runbooks/observability.md): readiness, liveness, metrics, and alert response. +- [Upgrade and rollback](docs/runbooks/upgrade.md) +- [Disaster recovery](docs/runbooks/disaster-recovery.md) +- [Recovery bundles](docs/runbooks/recovery-bundle.md) +- [Repository keys](docs/runbooks/keys.md) +- [Notifications](docs/runbooks/notifications.md) +- [SSH sources](docs/runbooks/ssh-sources.md) +- [Security policies](docs/README.md#security-boundaries) ## Development -### Running Tests - -```bash -cd backend -pytest +```sh +make setup +make test-fast +make test-integration +make test-fault +make test-security +npm --prefix frontend test -- --run +npm --prefix frontend exec playwright test ``` -### Database Migrations +OpenAPI and the generated browser client are committed artifacts: -```bash -cd backend -alembic revision --autogenerate -m "Description" -alembic upgrade head +```sh +.venv/bin/python tools/export_openapi.py --check openapi/v2.json +npm --prefix frontend run api:generate +git diff --exit-code -- openapi/v2.json frontend/src/api/generated ``` +## Release evidence + +Milestone evidence, the SBOM, provenance, and synthetic scale report live in [docs/release](docs/release/). M15 certification is synthetic metadata certification on the reference CI host; it does not claim a physical 10 TiB transfer. + ## License -MIT +MIT — see [LICENSE](LICENSE). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..c1b0612 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,42 @@ +# Backup Tool v2 documentation + +## Reason for existence + +This index points operators and contributors to the authoritative v2.0 procedures. Do not duplicate runbook steps in release evidence or issue comments. + +## Start here + +- [Project quick start](../README.md) +- [Upgrade and rollback](runbooks/upgrade.md) +- [Observability and alert response](runbooks/observability.md) +- [Disaster recovery](runbooks/disaster-recovery.md) + +## Runbooks + +| Need | Authoritative guide | +| --- | --- | +| Repository lifecycle | [repositories](runbooks/repositories.md) | +| Metadata protection | [metadata](runbooks/metadata.md) | +| Master and repository keys | [keys](runbooks/keys.md) | +| Encrypted recovery bundle | [recovery bundle](runbooks/recovery-bundle.md) | +| SSH source hardening | [SSH sources](runbooks/ssh-sources.md) | +| Notification operations | [notifications](runbooks/notifications.md) | +| Upgrade safety | [upgrade](runbooks/upgrade.md) | +| Host loss | [disaster recovery](runbooks/disaster-recovery.md) | + +## Security boundaries + +- [Repository encryption threat model](security/repository-encryption.md) +- [Notification egress and delivery policy](security/notifications.md) +- SSH sources require private-key authentication, exact host-key pinning, and a forced-SFTP chrooted account. No password, shell, agent, tunnel, or remote-command mode exists. + +## Release records + +[Release evidence](release/) records verification rather than replacing runbooks. v2.0 scale certification is documented in `release/m15-evidence.md` and `release/m15-scale-report.json`. + +## Verify + +```sh +grep -c '^## ' README.md +make check +```