docs: add backup tool design specification
- Architecture: Python FastAPI backend + React frontend - Data model: 6 core tables (sources, jobs, schedules, executions, backups, settings) - Backup flow: 7-step execution with incremental support - Error handling: Retry logic, partial backup handling, retention policies - Extensibility: Adapter pattern for sources and storage backends
This commit is contained in:
@@ -0,0 +1,533 @@
|
||||
# Backup Tool Design Specification
|
||||
|
||||
**Date:** 2026-05-11
|
||||
**Status:** Approved
|
||||
**Target:** Small team / SMB backup management
|
||||
|
||||
---
|
||||
|
||||
## 1. Overview
|
||||
|
||||
A web-based backup management tool for small teams and SMBs. The tool pulls data from various sources (local filesystem, remote via SSH, databases) to a central backup server, providing a dashboard for monitoring and management.
|
||||
|
||||
### Goals
|
||||
- Centralized backup management with web UI
|
||||
- Support for local, SSH, and database sources
|
||||
- Full and incremental backup strategies
|
||||
- Manual and scheduled job execution
|
||||
- SQLite-based persistence for simplicity
|
||||
- Extensible architecture for future source types and storage backends
|
||||
|
||||
### Non-Goals
|
||||
- Enterprise-scale distributed backup (1000+ nodes)
|
||||
- Real-time continuous backup (near-CDP)
|
||||
- Built-in cloud storage (S3, Azure Blob) in v1
|
||||
- Multi-tenancy or RBAC beyond basic auth
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture
|
||||
|
||||
### 2.1 High-Level Design
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Frontend Layer (React) │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Dashboard│ │ Backups │ │ Settings │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ │
|
||||
└──────────────────┬──────────────────────────┘
|
||||
│ REST API
|
||||
┌──────────────────▼──────────────────────────┐
|
||||
│ Backend Layer (Python) │
|
||||
│ ┌────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ API │ │ Backup │ │ Scheduler│ │
|
||||
│ │ Server │ │ Engine │ │ (APSched)│ │
|
||||
│ └────────┘ └──────────┘ └──────────┘ │
|
||||
│ ┌──────────────────────────────────┐ │
|
||||
│ │ Source Adapters │ │
|
||||
│ │ (Local | SSH | Database | ...) │ │
|
||||
│ └──────────────────────────────────┘ │
|
||||
└──────────────────┬──────────────────────────┘
|
||||
│
|
||||
┌──────────────────▼──────────────────────────┐
|
||||
│ Data Layer │
|
||||
│ ┌────────────────┐ ┌──────────────────┐ │
|
||||
│ │ SQLite DB │ │ File Storage │ │
|
||||
│ │ (Jobs, History)│ │ (Backup Archives)│ │
|
||||
│ └────────────────┘ └──────────────────┘ │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 Key Design Principles
|
||||
|
||||
1. **Source-agnostic**: Adapter pattern enables new source types without core changes
|
||||
2. **Job-driven**: Everything is a job - manual or scheduled, full or incremental
|
||||
3. **SQLite simplicity**: Single-file database, zero external dependencies
|
||||
4. **REST API separation**: Clean frontend/backend boundary
|
||||
5. **Extensible**: Plugin architecture for sources, storage, notifications
|
||||
|
||||
---
|
||||
|
||||
## 3. Technology Stack
|
||||
|
||||
### Backend
|
||||
- **Python 3.11+**
|
||||
- **FastAPI** - Async web framework with auto-generated OpenAPI docs
|
||||
- **SQLAlchemy 2.0+** - ORM with async support
|
||||
- **Alembic** - Database migrations
|
||||
- **APScheduler** - Job scheduling (cron expressions)
|
||||
- **Paramiko** - SSH client for remote sources
|
||||
- **rsync/libsync** - Incremental file synchronization
|
||||
- **Pydantic** - Data validation and settings management
|
||||
|
||||
### Frontend
|
||||
- **React 18+** with TypeScript
|
||||
- **TanStack Query** - Server state management and caching
|
||||
- **React Router 6+** - Client-side routing
|
||||
- **Tailwind CSS** - Utility-first styling
|
||||
- **Recharts** - Dashboard charts and visualizations
|
||||
- **React Hook Form** - Form management
|
||||
|
||||
### Data Storage
|
||||
- **SQLite** - Single-file relational database
|
||||
- **Local filesystem** - Backup archive storage
|
||||
|
||||
---
|
||||
|
||||
## 4. Data Model
|
||||
|
||||
### 4.1 Entities
|
||||
|
||||
#### sources
|
||||
Stores backup source configurations.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | INTEGER PK | Auto-increment primary key |
|
||||
| name | TEXT | Human-readable source name |
|
||||
| type | TEXT | Source type: `local`, `ssh`, `database` |
|
||||
| config | JSON | Type-specific configuration (path, host, credentials, etc.) |
|
||||
| created_at | DATETIME | Creation timestamp |
|
||||
| updated_at | DATETIME | Last update timestamp |
|
||||
|
||||
**Config examples by type:**
|
||||
- `local`: `{"path": "/var/data", "exclude": ["*.tmp", "*.log"]}``
|
||||
- `ssh`: `{"host": "server1", "port": 22, "username": "backup", "path": "/data", "key_path": "/keys/id_rsa"}`
|
||||
- `database`: `{"db_type": "postgresql", "host": "db1", "port": 5432, "database": "app", "username": "backup"}`
|
||||
|
||||
#### jobs
|
||||
Defines backup jobs with strategy and scheduling.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | INTEGER PK | Auto-increment primary key |
|
||||
| name | TEXT | Human-readable job name |
|
||||
| source_id | INTEGER FK → sources | Source to backup |
|
||||
| strategy | TEXT | `full` or `incremental` |
|
||||
| destination_path | TEXT | Local path for backup storage |
|
||||
| exclude_patterns | JSON | Additional exclude patterns (merged with source config) |
|
||||
| enabled | BOOLEAN | Whether job is active |
|
||||
| created_at | DATETIME | Creation timestamp |
|
||||
| updated_at | DATETIME | Last update timestamp |
|
||||
|
||||
#### schedules
|
||||
Cron-based scheduling for jobs (1:1 with jobs for simplicity).
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | INTEGER PK | Auto-increment primary key |
|
||||
| job_id | INTEGER FK → jobs | Associated job |
|
||||
| cron_expression | TEXT | Cron expression (e.g., "0 2 * * *" for daily 2 AM) |
|
||||
| enabled | BOOLEAN | Whether schedule is active |
|
||||
| created_at | DATETIME | Creation timestamp |
|
||||
|
||||
#### job_executions
|
||||
Tracks each job run with status and metrics.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | INTEGER PK | Auto-increment primary key |
|
||||
| job_id | INTEGER FK → jobs | Executed job |
|
||||
| status | TEXT | `pending`, `running`, `success`, `failed`, `cancelled` |
|
||||
| started_at | DATETIME | Execution start time |
|
||||
| completed_at | DATETIME | Execution end time (NULL if running) |
|
||||
| bytes_processed | INTEGER | Total bytes read from source |
|
||||
| bytes_backed_up | INTEGER | Total bytes written to destination |
|
||||
| error_message | TEXT | Error details if failed |
|
||||
| triggered_by | TEXT | `manual` or `schedule` |
|
||||
|
||||
#### backups
|
||||
Individual backup archives created by executions.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | INTEGER PK | Auto-increment primary key |
|
||||
| execution_id | INTEGER FK → job_executions | Parent execution |
|
||||
| storage_path | TEXT | Path to backup archive on disk |
|
||||
| size_bytes | INTEGER | Archive size in bytes |
|
||||
| checksum | TEXT | SHA-256 checksum for integrity verification |
|
||||
| type | TEXT | `full` or `incremental` |
|
||||
| parent_backup_id | INTEGER FK → backups | Previous backup in incremental chain (NULL for full) |
|
||||
| created_at | DATETIME | Creation timestamp |
|
||||
|
||||
#### settings
|
||||
Key-value application configuration.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| key | TEXT PK | Setting identifier |
|
||||
| value | TEXT | Setting value (JSON-encoded if complex) |
|
||||
| updated_at | DATETIME | Last update timestamp |
|
||||
|
||||
### 4.2 Relationships
|
||||
|
||||
- **sources → jobs**: One-to-many (one source can have multiple jobs)
|
||||
- **jobs → schedules**: One-to-one (each job has one schedule)
|
||||
- **jobs → job_executions**: One-to-many (job run history)
|
||||
- **job_executions → backups**: One-to-many (execution produces archives)
|
||||
- **backups → backups**: Self-referencing (incremental chain via parent_backup_id)
|
||||
|
||||
---
|
||||
|
||||
## 5. Backend Components
|
||||
|
||||
### 5.1 API Server (FastAPI)
|
||||
|
||||
**Responsibilities:**
|
||||
- Expose REST endpoints for frontend
|
||||
- Validate requests with Pydantic schemas
|
||||
- Handle authentication (basic auth or API keys in v1)
|
||||
- Serve static frontend files in production
|
||||
|
||||
**Key Endpoints:**
|
||||
- `GET /api/dashboard` - Dashboard stats and recent activity
|
||||
- `GET /api/sources` - List all sources
|
||||
- `POST /api/sources` - Create new source
|
||||
- `PUT /api/sources/{id}` - Update source
|
||||
- `DELETE /api/sources/{id}` - Delete source
|
||||
- `GET /api/jobs` - List all jobs
|
||||
- `POST /api/jobs` - Create new job
|
||||
- `PUT /api/jobs/{id}` - Update job
|
||||
- `DELETE /api/jobs/{id}` - Delete job
|
||||
- `POST /api/jobs/{id}/run` - Trigger manual execution
|
||||
- `GET /api/jobs/{id}/executions` - Get execution history
|
||||
- `GET /api/executions/{id}` - Get execution details
|
||||
- `GET /api/executions/{id}/logs` - Get execution logs
|
||||
- `GET /api/backups` - List all backups
|
||||
- `GET /api/backups/{id}/download` - Download backup archive
|
||||
- `DELETE /api/backups/{id}` - Delete backup
|
||||
- `GET /api/settings` - Get all settings
|
||||
- `PUT /api/settings` - Update settings
|
||||
|
||||
### 5.2 Backup Engine
|
||||
|
||||
**Responsibilities:**
|
||||
- Execute backup jobs (full and incremental)
|
||||
- Coordinate source adapters for data retrieval
|
||||
- Handle compression and encryption
|
||||
- Calculate and verify checksums
|
||||
- Update execution status in real-time
|
||||
|
||||
**Flow:**
|
||||
1. Receive job execution request
|
||||
2. Load source configuration
|
||||
3. Instantiate appropriate source adapter
|
||||
4. Connect to source
|
||||
5. Determine strategy (full vs incremental based on history)
|
||||
6. Transfer data using adapter
|
||||
7. Compress and encrypt (if configured)
|
||||
8. Calculate checksums
|
||||
9. Store metadata in database
|
||||
10. Apply retention policy
|
||||
|
||||
### 5.3 Scheduler (APScheduler)
|
||||
|
||||
**Responsibilities:**
|
||||
- Parse and evaluate cron expressions
|
||||
- Trigger job executions at scheduled times
|
||||
- Handle timezone support
|
||||
- Provide next-run predictions for UI
|
||||
|
||||
**Configuration:**
|
||||
- Uses SQLite backend for job persistence (survives restarts)
|
||||
- AsyncIO executor for non-blocking operation
|
||||
- Misfire grace period: 15 minutes
|
||||
|
||||
### 5.4 Source Adapters
|
||||
|
||||
**Abstract Base Class Interface:**
|
||||
```python
|
||||
class SourceAdapter(ABC):
|
||||
@abstractmethod
|
||||
async def connect(self) -> None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def disconnect(self) -> None: ...
|
||||
|
||||
@abstractmethod
|
||||
async def list_files(self, path: str) -> List[FileInfo]: ...
|
||||
|
||||
@abstractmethod
|
||||
async def read_file(self, path: str) -> AsyncIterator[bytes]: ...
|
||||
|
||||
@abstractmethod
|
||||
async def get_database_dump(self, config: dict) -> AsyncIterator[bytes]: ...
|
||||
```
|
||||
|
||||
**Implementations:**
|
||||
- **LocalAdapter**: Direct filesystem access
|
||||
- **SSHAdapter**: Paramiko-based SSH/SFTP connection
|
||||
- **DatabaseAdapter**: Uses native CLI tools (pg_dump, mysqldump) for consistency
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend Views
|
||||
|
||||
### 6.1 Dashboard
|
||||
|
||||
**Purpose:** At-a-glance system health and recent activity
|
||||
|
||||
**Components:**
|
||||
- Stats cards: Active jobs, total backups, storage used, recent failures
|
||||
- Recent activity feed: Last 10 executions with status
|
||||
- Storage overview chart: Pie chart showing storage by job
|
||||
- Quick actions: Run job, view failed executions
|
||||
|
||||
### 6.2 Backups View
|
||||
|
||||
**Purpose:** Job and source management
|
||||
|
||||
**Components:**
|
||||
- Job listing table: Name, source, strategy, schedule, last run, status
|
||||
- Source listing: Name, type, connection status
|
||||
- Create/Edit Job modal: Form with source selection, strategy, destination, schedule
|
||||
- Create/Edit Source modal: Type-specific configuration forms
|
||||
- Execution history per job: Expandable rows showing past runs
|
||||
- Manual run button: Trigger immediate execution with confirmation
|
||||
|
||||
### 6.3 Settings View
|
||||
|
||||
**Purpose:** Application configuration
|
||||
|
||||
**Sections:**
|
||||
- **General**: Default backup location, retention policy, compression
|
||||
- **Storage**: Storage path, disk usage warnings
|
||||
- **Notifications**: Webhook URLs, email settings
|
||||
- **Security**: Encryption toggle, key management
|
||||
- **Logs**: Log level, retention, download
|
||||
|
||||
---
|
||||
|
||||
## 7. Backup Execution Flow
|
||||
|
||||
### 7.1 Normal Flow
|
||||
|
||||
```
|
||||
Trigger (Manual/Schedule)
|
||||
↓
|
||||
Create Execution Record (status: pending)
|
||||
↓
|
||||
Connect to Source (via adapter)
|
||||
↓
|
||||
Determine Strategy:
|
||||
- If no previous full backup → Full
|
||||
- If strategy = full → Full
|
||||
- If strategy = incremental → Incremental (link to parent)
|
||||
↓
|
||||
Execute Backup:
|
||||
- Stream data from source
|
||||
- Compress (if enabled)
|
||||
- Encrypt (if enabled)
|
||||
- Calculate checksums
|
||||
↓
|
||||
Verify & Store:
|
||||
- Verify checksum
|
||||
- Write metadata to backups table
|
||||
- Update execution status → success
|
||||
↓
|
||||
Cleanup & Retention:
|
||||
- Apply retention policy
|
||||
- Delete old backups
|
||||
- Update storage stats
|
||||
```
|
||||
|
||||
### 7.2 Incremental Backup Strategy
|
||||
|
||||
For incremental backups, the system uses file-level deduplication:
|
||||
|
||||
1. Compare file metadata (mtime, size) against last backup
|
||||
2. Only transfer changed files
|
||||
3. Create hard links or copy-on-write references for unchanged files
|
||||
4. Store incremental manifest referencing parent backup
|
||||
|
||||
**Storage format:**
|
||||
```
|
||||
backups/
|
||||
├── 2026-05-11_020000_full/
|
||||
│ ├── data/
|
||||
│ └── manifest.json
|
||||
└── 2026-05-11_140000_incr/
|
||||
├── data/ (only changed files)
|
||||
└── manifest.json (references parent)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Error Handling
|
||||
|
||||
### 8.1 Error Scenarios
|
||||
|
||||
| Scenario | Handling |
|
||||
|----------|----------|
|
||||
| **Connection failure** | Retry 3x with exponential backoff (1s, 2s, 4s), then mark failed |
|
||||
| **Partial backup** | Mark as failed if any source file fails; keep partial for inspection |
|
||||
| **Storage full** | Check before starting (>10% free required); alert if threshold reached |
|
||||
| **Checksum mismatch** | Delete corrupted backup, retry once, alert admin |
|
||||
| **Concurrent jobs** | Queue if resource limit reached; configurable max concurrent |
|
||||
| **Source unavailable** | Mark failed, schedule retry based on policy |
|
||||
| **Network interruption** | Resume capability for large transfers (SSH/SCP) |
|
||||
|
||||
### 8.2 Logging
|
||||
|
||||
- Structured JSON logging for machine parsing
|
||||
- Separate logs per execution: `/var/log/backup-tool/executions/{id}.log`
|
||||
- Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
|
||||
- Frontend can stream logs via WebSocket (future) or polling
|
||||
|
||||
---
|
||||
|
||||
## 9. Extensibility
|
||||
|
||||
### 9.1 Source Adapter Extension
|
||||
|
||||
To add a new source type (e.g., S3):
|
||||
|
||||
1. Create class extending `SourceAdapter`
|
||||
2. Implement required methods
|
||||
3. Register in adapter factory
|
||||
4. Add UI form components for configuration
|
||||
|
||||
### 9.2 Storage Backend Extension
|
||||
|
||||
Storage backends follow similar pattern:
|
||||
1. Implement `StorageBackend` interface
|
||||
2. Support `store()`, `retrieve()`, `delete()`, `list()` operations
|
||||
3. Register in backend factory
|
||||
|
||||
### 9.3 Notification Channels
|
||||
|
||||
Notification system supports pluggable channels:
|
||||
- Webhook (generic HTTP POST)
|
||||
- Email (SMTP)
|
||||
- Slack/Discord (webhook URLs)
|
||||
|
||||
---
|
||||
|
||||
## 10. Security Considerations
|
||||
|
||||
- **Credential storage**: Source credentials stored encrypted in SQLite (Fernet encryption)
|
||||
- **API authentication**: JWT tokens or API keys
|
||||
- **Backup encryption**: Optional AES-256 encryption of archives
|
||||
- **Transport security**: SSH for remote sources, HTTPS for web UI
|
||||
- **File permissions**: Backup archives readable only by backup service user
|
||||
|
||||
---
|
||||
|
||||
## 11. Performance Considerations
|
||||
|
||||
- **Streaming**: Large files streamed rather than loaded into memory
|
||||
- **Async I/O**: All source adapters use async operations
|
||||
- **Pagination**: API endpoints paginated (50 items default)
|
||||
- **Database indexing**: Indexed on frequently queried columns (job_id, status, created_at)
|
||||
- **Background tasks**: Long-running backups execute in background workers
|
||||
|
||||
---
|
||||
|
||||
## 12. Deployment
|
||||
|
||||
### 12.1 Development
|
||||
```bash
|
||||
# Backend
|
||||
pip install -r requirements.txt
|
||||
uvicorn main:app --reload
|
||||
|
||||
# Frontend
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### 12.2 Production
|
||||
- Backend: Uvicorn with Gunicorn (workers=4)
|
||||
- Frontend: Built static files served by FastAPI
|
||||
- Systemd service for automatic startup
|
||||
- SQLite: Single file, backup with `.backup` command
|
||||
|
||||
---
|
||||
|
||||
## 13. Testing Strategy
|
||||
|
||||
### 13.1 Backend Tests
|
||||
- **Unit tests**: Adapter mocking, engine logic
|
||||
- **Integration tests**: Database operations, API endpoints
|
||||
- **End-to-end tests**: Full backup flows with test fixtures
|
||||
|
||||
### 13.2 Frontend Tests
|
||||
- **Component tests**: React Testing Library
|
||||
- **Integration tests**: API mocking with MSW
|
||||
- **E2E tests**: Playwright for critical flows
|
||||
|
||||
---
|
||||
|
||||
## 14. Future Roadmap
|
||||
|
||||
### v1.1
|
||||
- Cloud storage backends (S3, Azure Blob)
|
||||
- Backup verification (automated restore testing)
|
||||
- Email notifications
|
||||
|
||||
### v1.2
|
||||
- Multi-node backup (agent-based architecture)
|
||||
- Backup encryption at rest
|
||||
- WebSocket live log streaming
|
||||
|
||||
### v2.0
|
||||
- REST API for external integrations
|
||||
- Backup reporting and analytics
|
||||
- Role-based access control
|
||||
|
||||
---
|
||||
|
||||
## 15. Appendix
|
||||
|
||||
### 15.1 Cron Expression Examples
|
||||
|
||||
| Expression | Schedule |
|
||||
|------------|----------|
|
||||
| `0 2 * * *` | Daily at 2:00 AM |
|
||||
| `0 */6 * * *` | Every 6 hours |
|
||||
| `0 0 * * 0` | Weekly on Sunday |
|
||||
| `0 0 1 * *` | Monthly on 1st |
|
||||
|
||||
### 15.2 Database Migration Strategy
|
||||
|
||||
- Alembic for schema migrations
|
||||
- One migration per release
|
||||
- Backward compatibility for rolling updates
|
||||
- Migration tests in CI pipeline
|
||||
|
||||
### 15.3 Backup Archive Format
|
||||
|
||||
```
|
||||
{destination_path}/{job_id}/
|
||||
├── 2026-05-11_020000/
|
||||
│ ├── manifest.json # Metadata and file list
|
||||
│ ├── data.tar.gz # Compressed archive (or directory tree)
|
||||
│ └── checksum.sha256 # Integrity verification
|
||||
└── latest -> 2026-05-11_020000/ # Symlink to latest backup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**End of Specification**
|
||||
Reference in New Issue
Block a user