diff --git a/docs/superpowers/specs/2026-05-11-backup-tool-design.md b/docs/superpowers/specs/2026-05-11-backup-tool-design.md new file mode 100644 index 0000000..8aa48b6 --- /dev/null +++ b/docs/superpowers/specs/2026-05-11-backup-tool-design.md @@ -0,0 +1,533 @@ +# Backup Tool Design Specification + +**Date:** 2026-05-11 +**Status:** Approved +**Target:** Small team / SMB backup management + +--- + +## 1. Overview + +A web-based backup management tool for small teams and SMBs. The tool pulls data from various sources (local filesystem, remote via SSH, databases) to a central backup server, providing a dashboard for monitoring and management. + +### Goals +- Centralized backup management with web UI +- Support for local, SSH, and database sources +- Full and incremental backup strategies +- Manual and scheduled job execution +- SQLite-based persistence for simplicity +- Extensible architecture for future source types and storage backends + +### Non-Goals +- Enterprise-scale distributed backup (1000+ nodes) +- Real-time continuous backup (near-CDP) +- Built-in cloud storage (S3, Azure Blob) in v1 +- Multi-tenancy or RBAC beyond basic auth + +--- + +## 2. Architecture + +### 2.1 High-Level Design + +``` +┌─────────────────────────────────────────────┐ +│ Frontend Layer (React) │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ Dashboard│ │ Backups │ │ Settings │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +└──────────────────┬──────────────────────────┘ + │ REST API +┌──────────────────▼──────────────────────────┐ +│ Backend Layer (Python) │ +│ ┌────────┐ ┌──────────┐ ┌──────────┐ │ +│ │ API │ │ Backup │ │ Scheduler│ │ +│ │ Server │ │ Engine │ │ (APSched)│ │ +│ └────────┘ └──────────┘ └──────────┘ │ +│ ┌──────────────────────────────────┐ │ +│ │ Source Adapters │ │ +│ │ (Local | SSH | Database | ...) │ │ +│ └──────────────────────────────────┘ │ +└──────────────────┬──────────────────────────┘ + │ +┌──────────────────▼──────────────────────────┐ +│ Data Layer │ +│ ┌────────────────┐ ┌──────────────────┐ │ +│ │ SQLite DB │ │ File Storage │ │ +│ │ (Jobs, History)│ │ (Backup Archives)│ │ +│ └────────────────┘ └──────────────────┘ │ +└─────────────────────────────────────────────┘ +``` + +### 2.2 Key Design Principles + +1. **Source-agnostic**: Adapter pattern enables new source types without core changes +2. **Job-driven**: Everything is a job - manual or scheduled, full or incremental +3. **SQLite simplicity**: Single-file database, zero external dependencies +4. **REST API separation**: Clean frontend/backend boundary +5. **Extensible**: Plugin architecture for sources, storage, notifications + +--- + +## 3. Technology Stack + +### Backend +- **Python 3.11+** +- **FastAPI** - Async web framework with auto-generated OpenAPI docs +- **SQLAlchemy 2.0+** - ORM with async support +- **Alembic** - Database migrations +- **APScheduler** - Job scheduling (cron expressions) +- **Paramiko** - SSH client for remote sources +- **rsync/libsync** - Incremental file synchronization +- **Pydantic** - Data validation and settings management + +### Frontend +- **React 18+** with TypeScript +- **TanStack Query** - Server state management and caching +- **React Router 6+** - Client-side routing +- **Tailwind CSS** - Utility-first styling +- **Recharts** - Dashboard charts and visualizations +- **React Hook Form** - Form management + +### Data Storage +- **SQLite** - Single-file relational database +- **Local filesystem** - Backup archive storage + +--- + +## 4. Data Model + +### 4.1 Entities + +#### sources +Stores backup source configurations. + +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK | Auto-increment primary key | +| name | TEXT | Human-readable source name | +| type | TEXT | Source type: `local`, `ssh`, `database` | +| config | JSON | Type-specific configuration (path, host, credentials, etc.) | +| created_at | DATETIME | Creation timestamp | +| updated_at | DATETIME | Last update timestamp | + +**Config examples by type:** +- `local`: `{"path": "/var/data", "exclude": ["*.tmp", "*.log"]}`` +- `ssh`: `{"host": "server1", "port": 22, "username": "backup", "path": "/data", "key_path": "/keys/id_rsa"}` +- `database`: `{"db_type": "postgresql", "host": "db1", "port": 5432, "database": "app", "username": "backup"}` + +#### jobs +Defines backup jobs with strategy and scheduling. + +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK | Auto-increment primary key | +| name | TEXT | Human-readable job name | +| source_id | INTEGER FK → sources | Source to backup | +| strategy | TEXT | `full` or `incremental` | +| destination_path | TEXT | Local path for backup storage | +| exclude_patterns | JSON | Additional exclude patterns (merged with source config) | +| enabled | BOOLEAN | Whether job is active | +| created_at | DATETIME | Creation timestamp | +| updated_at | DATETIME | Last update timestamp | + +#### schedules +Cron-based scheduling for jobs (1:1 with jobs for simplicity). + +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK | Auto-increment primary key | +| job_id | INTEGER FK → jobs | Associated job | +| cron_expression | TEXT | Cron expression (e.g., "0 2 * * *" for daily 2 AM) | +| enabled | BOOLEAN | Whether schedule is active | +| created_at | DATETIME | Creation timestamp | + +#### job_executions +Tracks each job run with status and metrics. + +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK | Auto-increment primary key | +| job_id | INTEGER FK → jobs | Executed job | +| status | TEXT | `pending`, `running`, `success`, `failed`, `cancelled` | +| started_at | DATETIME | Execution start time | +| completed_at | DATETIME | Execution end time (NULL if running) | +| bytes_processed | INTEGER | Total bytes read from source | +| bytes_backed_up | INTEGER | Total bytes written to destination | +| error_message | TEXT | Error details if failed | +| triggered_by | TEXT | `manual` or `schedule` | + +#### backups +Individual backup archives created by executions. + +| Column | Type | Description | +|--------|------|-------------| +| id | INTEGER PK | Auto-increment primary key | +| execution_id | INTEGER FK → job_executions | Parent execution | +| storage_path | TEXT | Path to backup archive on disk | +| size_bytes | INTEGER | Archive size in bytes | +| checksum | TEXT | SHA-256 checksum for integrity verification | +| type | TEXT | `full` or `incremental` | +| parent_backup_id | INTEGER FK → backups | Previous backup in incremental chain (NULL for full) | +| created_at | DATETIME | Creation timestamp | + +#### settings +Key-value application configuration. + +| Column | Type | Description | +|--------|------|-------------| +| key | TEXT PK | Setting identifier | +| value | TEXT | Setting value (JSON-encoded if complex) | +| updated_at | DATETIME | Last update timestamp | + +### 4.2 Relationships + +- **sources → jobs**: One-to-many (one source can have multiple jobs) +- **jobs → schedules**: One-to-one (each job has one schedule) +- **jobs → job_executions**: One-to-many (job run history) +- **job_executions → backups**: One-to-many (execution produces archives) +- **backups → backups**: Self-referencing (incremental chain via parent_backup_id) + +--- + +## 5. Backend Components + +### 5.1 API Server (FastAPI) + +**Responsibilities:** +- Expose REST endpoints for frontend +- Validate requests with Pydantic schemas +- Handle authentication (basic auth or API keys in v1) +- Serve static frontend files in production + +**Key Endpoints:** +- `GET /api/dashboard` - Dashboard stats and recent activity +- `GET /api/sources` - List all sources +- `POST /api/sources` - Create new source +- `PUT /api/sources/{id}` - Update source +- `DELETE /api/sources/{id}` - Delete source +- `GET /api/jobs` - List all jobs +- `POST /api/jobs` - Create new job +- `PUT /api/jobs/{id}` - Update job +- `DELETE /api/jobs/{id}` - Delete job +- `POST /api/jobs/{id}/run` - Trigger manual execution +- `GET /api/jobs/{id}/executions` - Get execution history +- `GET /api/executions/{id}` - Get execution details +- `GET /api/executions/{id}/logs` - Get execution logs +- `GET /api/backups` - List all backups +- `GET /api/backups/{id}/download` - Download backup archive +- `DELETE /api/backups/{id}` - Delete backup +- `GET /api/settings` - Get all settings +- `PUT /api/settings` - Update settings + +### 5.2 Backup Engine + +**Responsibilities:** +- Execute backup jobs (full and incremental) +- Coordinate source adapters for data retrieval +- Handle compression and encryption +- Calculate and verify checksums +- Update execution status in real-time + +**Flow:** +1. Receive job execution request +2. Load source configuration +3. Instantiate appropriate source adapter +4. Connect to source +5. Determine strategy (full vs incremental based on history) +6. Transfer data using adapter +7. Compress and encrypt (if configured) +8. Calculate checksums +9. Store metadata in database +10. Apply retention policy + +### 5.3 Scheduler (APScheduler) + +**Responsibilities:** +- Parse and evaluate cron expressions +- Trigger job executions at scheduled times +- Handle timezone support +- Provide next-run predictions for UI + +**Configuration:** +- Uses SQLite backend for job persistence (survives restarts) +- AsyncIO executor for non-blocking operation +- Misfire grace period: 15 minutes + +### 5.4 Source Adapters + +**Abstract Base Class Interface:** +```python +class SourceAdapter(ABC): + @abstractmethod + async def connect(self) -> None: ... + + @abstractmethod + async def disconnect(self) -> None: ... + + @abstractmethod + async def list_files(self, path: str) -> List[FileInfo]: ... + + @abstractmethod + async def read_file(self, path: str) -> AsyncIterator[bytes]: ... + + @abstractmethod + async def get_database_dump(self, config: dict) -> AsyncIterator[bytes]: ... +``` + +**Implementations:** +- **LocalAdapter**: Direct filesystem access +- **SSHAdapter**: Paramiko-based SSH/SFTP connection +- **DatabaseAdapter**: Uses native CLI tools (pg_dump, mysqldump) for consistency + +--- + +## 6. Frontend Views + +### 6.1 Dashboard + +**Purpose:** At-a-glance system health and recent activity + +**Components:** +- Stats cards: Active jobs, total backups, storage used, recent failures +- Recent activity feed: Last 10 executions with status +- Storage overview chart: Pie chart showing storage by job +- Quick actions: Run job, view failed executions + +### 6.2 Backups View + +**Purpose:** Job and source management + +**Components:** +- Job listing table: Name, source, strategy, schedule, last run, status +- Source listing: Name, type, connection status +- Create/Edit Job modal: Form with source selection, strategy, destination, schedule +- Create/Edit Source modal: Type-specific configuration forms +- Execution history per job: Expandable rows showing past runs +- Manual run button: Trigger immediate execution with confirmation + +### 6.3 Settings View + +**Purpose:** Application configuration + +**Sections:** +- **General**: Default backup location, retention policy, compression +- **Storage**: Storage path, disk usage warnings +- **Notifications**: Webhook URLs, email settings +- **Security**: Encryption toggle, key management +- **Logs**: Log level, retention, download + +--- + +## 7. Backup Execution Flow + +### 7.1 Normal Flow + +``` +Trigger (Manual/Schedule) + ↓ +Create Execution Record (status: pending) + ↓ +Connect to Source (via adapter) + ↓ +Determine Strategy: + - If no previous full backup → Full + - If strategy = full → Full + - If strategy = incremental → Incremental (link to parent) + ↓ +Execute Backup: + - Stream data from source + - Compress (if enabled) + - Encrypt (if enabled) + - Calculate checksums + ↓ +Verify & Store: + - Verify checksum + - Write metadata to backups table + - Update execution status → success + ↓ +Cleanup & Retention: + - Apply retention policy + - Delete old backups + - Update storage stats +``` + +### 7.2 Incremental Backup Strategy + +For incremental backups, the system uses file-level deduplication: + +1. Compare file metadata (mtime, size) against last backup +2. Only transfer changed files +3. Create hard links or copy-on-write references for unchanged files +4. Store incremental manifest referencing parent backup + +**Storage format:** +``` +backups/ + ├── 2026-05-11_020000_full/ + │ ├── data/ + │ └── manifest.json + └── 2026-05-11_140000_incr/ + ├── data/ (only changed files) + └── manifest.json (references parent) +``` + +--- + +## 8. Error Handling + +### 8.1 Error Scenarios + +| Scenario | Handling | +|----------|----------| +| **Connection failure** | Retry 3x with exponential backoff (1s, 2s, 4s), then mark failed | +| **Partial backup** | Mark as failed if any source file fails; keep partial for inspection | +| **Storage full** | Check before starting (>10% free required); alert if threshold reached | +| **Checksum mismatch** | Delete corrupted backup, retry once, alert admin | +| **Concurrent jobs** | Queue if resource limit reached; configurable max concurrent | +| **Source unavailable** | Mark failed, schedule retry based on policy | +| **Network interruption** | Resume capability for large transfers (SSH/SCP) | + +### 8.2 Logging + +- Structured JSON logging for machine parsing +- Separate logs per execution: `/var/log/backup-tool/executions/{id}.log` +- Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL +- Frontend can stream logs via WebSocket (future) or polling + +--- + +## 9. Extensibility + +### 9.1 Source Adapter Extension + +To add a new source type (e.g., S3): + +1. Create class extending `SourceAdapter` +2. Implement required methods +3. Register in adapter factory +4. Add UI form components for configuration + +### 9.2 Storage Backend Extension + +Storage backends follow similar pattern: +1. Implement `StorageBackend` interface +2. Support `store()`, `retrieve()`, `delete()`, `list()` operations +3. Register in backend factory + +### 9.3 Notification Channels + +Notification system supports pluggable channels: +- Webhook (generic HTTP POST) +- Email (SMTP) +- Slack/Discord (webhook URLs) + +--- + +## 10. Security Considerations + +- **Credential storage**: Source credentials stored encrypted in SQLite (Fernet encryption) +- **API authentication**: JWT tokens or API keys +- **Backup encryption**: Optional AES-256 encryption of archives +- **Transport security**: SSH for remote sources, HTTPS for web UI +- **File permissions**: Backup archives readable only by backup service user + +--- + +## 11. Performance Considerations + +- **Streaming**: Large files streamed rather than loaded into memory +- **Async I/O**: All source adapters use async operations +- **Pagination**: API endpoints paginated (50 items default) +- **Database indexing**: Indexed on frequently queried columns (job_id, status, created_at) +- **Background tasks**: Long-running backups execute in background workers + +--- + +## 12. Deployment + +### 12.1 Development +```bash +# Backend +pip install -r requirements.txt +uvicorn main:app --reload + +# Frontend +cd frontend +npm install +npm run dev +``` + +### 12.2 Production +- Backend: Uvicorn with Gunicorn (workers=4) +- Frontend: Built static files served by FastAPI +- Systemd service for automatic startup +- SQLite: Single file, backup with `.backup` command + +--- + +## 13. Testing Strategy + +### 13.1 Backend Tests +- **Unit tests**: Adapter mocking, engine logic +- **Integration tests**: Database operations, API endpoints +- **End-to-end tests**: Full backup flows with test fixtures + +### 13.2 Frontend Tests +- **Component tests**: React Testing Library +- **Integration tests**: API mocking with MSW +- **E2E tests**: Playwright for critical flows + +--- + +## 14. Future Roadmap + +### v1.1 +- Cloud storage backends (S3, Azure Blob) +- Backup verification (automated restore testing) +- Email notifications + +### v1.2 +- Multi-node backup (agent-based architecture) +- Backup encryption at rest +- WebSocket live log streaming + +### v2.0 +- REST API for external integrations +- Backup reporting and analytics +- Role-based access control + +--- + +## 15. Appendix + +### 15.1 Cron Expression Examples + +| Expression | Schedule | +|------------|----------| +| `0 2 * * *` | Daily at 2:00 AM | +| `0 */6 * * *` | Every 6 hours | +| `0 0 * * 0` | Weekly on Sunday | +| `0 0 1 * *` | Monthly on 1st | + +### 15.2 Database Migration Strategy + +- Alembic for schema migrations +- One migration per release +- Backward compatibility for rolling updates +- Migration tests in CI pipeline + +### 15.3 Backup Archive Format + +``` +{destination_path}/{job_id}/ + ├── 2026-05-11_020000/ + │ ├── manifest.json # Metadata and file list + │ ├── data.tar.gz # Compressed archive (or directory tree) + │ └── checksum.sha256 # Integrity verification + └── latest -> 2026-05-11_020000/ # Symlink to latest backup +``` + +--- + +**End of Specification**