# Backup Tool Design Specification

**Date:** 2026-05-11
**Status:** Approved
**Target:** Small team / SMB backup management

---

## 1. Overview

A web-based backup management tool for small teams and SMBs. The tool pulls data from various sources (local filesystem, remote via SSH, databases) to a central backup server, providing a dashboard for monitoring and management.

### Goals
- Centralized backup management with web UI
- Support for local, SSH, and database sources
- Full and incremental backup strategies
- Manual and scheduled job execution
- SQLite-based persistence for simplicity
- Extensible architecture for future source types and storage backends

### Non-Goals
- Enterprise-scale distributed backup (1000+ nodes)
- Real-time continuous backup (near-CDP)
- Built-in cloud storage (S3, Azure Blob) in v1
- Multi-tenancy or RBAC beyond basic auth

---

## 2. Architecture

### 2.1 High-Level Design

```
┌─────────────────────────────────────────────┐
│           Frontend Layer (React)             │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐    │
│  │ Dashboard│ │ Backups  │ │ Settings │    │
│  └──────────┘ └──────────┘ └──────────┘    │
└──────────────────┬──────────────────────────┘
                   │ REST API
┌──────────────────▼──────────────────────────┐
│          Backend Layer (Python)              │
│  ┌────────┐ ┌──────────┐ ┌──────────┐      │
│  │ API    │ │ Backup   │ │ Scheduler│      │
│  │ Server │ │ Engine   │ │ (APSched)│      │
│  └────────┘ └──────────┘ └──────────┘      │
│  ┌──────────────────────────────────┐       │
│  │     Source Adapters              │       │
│  │  (Local | SSH | Database | ...)  │       │
│  └──────────────────────────────────┘       │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│              Data Layer                      │
│  ┌────────────────┐ ┌──────────────────┐    │
│  │   SQLite DB    │ │   File Storage   │    │
│  │ (Jobs, History)│ │ (Backup Archives)│    │
│  └────────────────┘ └──────────────────┘    │
└─────────────────────────────────────────────┘
```

### 2.2 Key Design Principles

1. **Source-agnostic**: Adapter pattern enables new source types without core changes
2. **Job-driven**: Everything is a job - manual or scheduled, full or incremental
3. **SQLite simplicity**: Single-file database, zero external dependencies
4. **REST API separation**: Clean frontend/backend boundary
5. **Extensible**: Plugin architecture for sources, storage, notifications

---

## 3. Technology Stack

### Backend
- **Python 3.11+**
- **FastAPI** - Async web framework with auto-generated OpenAPI docs
- **SQLAlchemy 2.0+** - ORM with async support
- **Alembic** - Database migrations
- **APScheduler** - Job scheduling (cron expressions)
- **Paramiko** - SSH client for remote sources
- **rsync/libsync** - Incremental file synchronization
- **Pydantic** - Data validation and settings management

### Frontend
- **React 18+** with TypeScript
- **TanStack Query** - Server state management and caching
- **React Router 6+** - Client-side routing
- **Tailwind CSS** - Utility-first styling
- **Recharts** - Dashboard charts and visualizations
- **React Hook Form** - Form management

### Data Storage
- **SQLite** - Single-file relational database
- **Local filesystem** - Backup archive storage

---

## 4. Data Model

### 4.1 Entities

#### sources
Stores backup source configurations.

| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER PK | Auto-increment primary key |
| name | TEXT | Human-readable source name |
| type | TEXT | Source type: `local`, `ssh`, `database` |
| config | JSON | Type-specific configuration (path, host, credentials, etc.) |
| created_at | DATETIME | Creation timestamp |
| updated_at | DATETIME | Last update timestamp |

**Config examples by type:**
- `local`: `{"path": "/var/data", "exclude": ["*.tmp", "*.log"]}``
- `ssh`: `{"host": "server1", "port": 22, "username": "backup", "path": "/data", "key_path": "/keys/id_rsa"}`
- `database`: `{"db_type": "postgresql", "host": "db1", "port": 5432, "database": "app", "username": "backup"}`

#### jobs
Defines backup jobs with strategy and scheduling.

| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER PK | Auto-increment primary key |
| name | TEXT | Human-readable job name |
| source_id | INTEGER FK → sources | Source to backup |
| strategy | TEXT | `full` or `incremental` |
| destination_path | TEXT | Local path for backup storage |
| exclude_patterns | JSON | Additional exclude patterns (merged with source config) |
| enabled | BOOLEAN | Whether job is active |
| created_at | DATETIME | Creation timestamp |
| updated_at | DATETIME | Last update timestamp |

#### schedules
Cron-based scheduling for jobs (1:1 with jobs for simplicity).

| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER PK | Auto-increment primary key |
| job_id | INTEGER FK → jobs | Associated job |
| cron_expression | TEXT | Cron expression (e.g., "0 2 * * *" for daily 2 AM) |
| enabled | BOOLEAN | Whether schedule is active |
| created_at | DATETIME | Creation timestamp |

#### job_executions
Tracks each job run with status and metrics.

| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER PK | Auto-increment primary key |
| job_id | INTEGER FK → jobs | Executed job |
| status | TEXT | `pending`, `running`, `success`, `failed`, `cancelled` |
| started_at | DATETIME | Execution start time |
| completed_at | DATETIME | Execution end time (NULL if running) |
| bytes_processed | INTEGER | Total bytes read from source |
| bytes_backed_up | INTEGER | Total bytes written to destination |
| error_message | TEXT | Error details if failed |
| triggered_by | TEXT | `manual` or `schedule` |

#### backups
Individual backup archives created by executions.

| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER PK | Auto-increment primary key |
| execution_id | INTEGER FK → job_executions | Parent execution |
| storage_path | TEXT | Path to backup archive on disk |
| size_bytes | INTEGER | Archive size in bytes |
| checksum | TEXT | SHA-256 checksum for integrity verification |
| type | TEXT | `full` or `incremental` |
| parent_backup_id | INTEGER FK → backups | Previous backup in incremental chain (NULL for full) |
| created_at | DATETIME | Creation timestamp |

#### settings
Key-value application configuration.

| Column | Type | Description |
|--------|------|-------------|
| key | TEXT PK | Setting identifier |
| value | TEXT | Setting value (JSON-encoded if complex) |
| updated_at | DATETIME | Last update timestamp |

### 4.2 Relationships

- **sources → jobs**: One-to-many (one source can have multiple jobs)
- **jobs → schedules**: One-to-one (each job has one schedule)
- **jobs → job_executions**: One-to-many (job run history)
- **job_executions → backups**: One-to-many (execution produces archives)
- **backups → backups**: Self-referencing (incremental chain via parent_backup_id)

---

## 5. Backend Components

### 5.1 API Server (FastAPI)

**Responsibilities:**
- Expose REST endpoints for frontend
- Validate requests with Pydantic schemas
- Handle authentication (basic auth or API keys in v1)
- Serve static frontend files in production

**Key Endpoints:**
- `GET /api/dashboard` - Dashboard stats and recent activity
- `GET /api/sources` - List all sources
- `POST /api/sources` - Create new source
- `PUT /api/sources/{id}` - Update source
- `DELETE /api/sources/{id}` - Delete source
- `GET /api/jobs` - List all jobs
- `POST /api/jobs` - Create new job
- `PUT /api/jobs/{id}` - Update job
- `DELETE /api/jobs/{id}` - Delete job
- `POST /api/jobs/{id}/run` - Trigger manual execution
- `GET /api/jobs/{id}/executions` - Get execution history
- `GET /api/executions/{id}` - Get execution details
- `GET /api/executions/{id}/logs` - Get execution logs
- `GET /api/backups` - List all backups
- `GET /api/backups/{id}/download` - Download backup archive
- `DELETE /api/backups/{id}` - Delete backup
- `GET /api/settings` - Get all settings
- `PUT /api/settings` - Update settings

### 5.2 Backup Engine

**Responsibilities:**
- Execute backup jobs (full and incremental)
- Coordinate source adapters for data retrieval
- Handle compression and encryption
- Calculate and verify checksums
- Update execution status in real-time

**Flow:**
1. Receive job execution request
2. Load source configuration
3. Instantiate appropriate source adapter
4. Connect to source
5. Determine strategy (full vs incremental based on history)
6. Transfer data using adapter
7. Compress and encrypt (if configured)
8. Calculate checksums
9. Store metadata in database
10. Apply retention policy

### 5.3 Scheduler (APScheduler)

**Responsibilities:**
- Parse and evaluate cron expressions
- Trigger job executions at scheduled times
- Handle timezone support
- Provide next-run predictions for UI

**Configuration:**
- Uses SQLite backend for job persistence (survives restarts)
- AsyncIO executor for non-blocking operation
- Misfire grace period: 15 minutes

### 5.4 Source Adapters

**Abstract Base Class Interface:**
```python
class SourceAdapter(ABC):
    @abstractmethod
    async def connect(self) -> None: ...
    
    @abstractmethod
    async def disconnect(self) -> None: ...
    
    @abstractmethod
    async def list_files(self, path: str) -> List[FileInfo]: ...
    
    @abstractmethod
    async def read_file(self, path: str) -> AsyncIterator[bytes]: ...
    
    @abstractmethod
    async def get_database_dump(self, config: dict) -> AsyncIterator[bytes]: ...
```

**Implementations:**
- **LocalAdapter**: Direct filesystem access
- **SSHAdapter**: Paramiko-based SSH/SFTP connection
- **DatabaseAdapter**: Uses native CLI tools (pg_dump, mysqldump) for consistency

---

## 6. Frontend Views

### 6.1 Dashboard

**Purpose:** At-a-glance system health and recent activity

**Components:**
- Stats cards: Active jobs, total backups, storage used, recent failures
- Recent activity feed: Last 10 executions with status
- Storage overview chart: Pie chart showing storage by job
- Quick actions: Run job, view failed executions

### 6.2 Backups View

**Purpose:** Job and source management

**Components:**
- Job listing table: Name, source, strategy, schedule, last run, status
- Source listing: Name, type, connection status
- Create/Edit Job modal: Form with source selection, strategy, destination, schedule
- Create/Edit Source modal: Type-specific configuration forms
- Execution history per job: Expandable rows showing past runs
- Manual run button: Trigger immediate execution with confirmation

### 6.3 Settings View

**Purpose:** Application configuration

**Sections:**
- **General**: Default backup location, retention policy, compression
- **Storage**: Storage path, disk usage warnings
- **Notifications**: Webhook URLs, email settings
- **Security**: Encryption toggle, key management
- **Logs**: Log level, retention, download

---

## 7. Backup Execution Flow

### 7.1 Normal Flow

```
Trigger (Manual/Schedule)
    ↓
Create Execution Record (status: pending)
    ↓
Connect to Source (via adapter)
    ↓
Determine Strategy:
    - If no previous full backup → Full
    - If strategy = full → Full
    - If strategy = incremental → Incremental (link to parent)
    ↓
Execute Backup:
    - Stream data from source
    - Compress (if enabled)
    - Encrypt (if enabled)
    - Calculate checksums
    ↓
Verify & Store:
    - Verify checksum
    - Write metadata to backups table
    - Update execution status → success
    ↓
Cleanup & Retention:
    - Apply retention policy
    - Delete old backups
    - Update storage stats
```

### 7.2 Incremental Backup Strategy

For incremental backups, the system uses file-level deduplication:

1. Compare file metadata (mtime, size) against last backup
2. Only transfer changed files
3. Create hard links or copy-on-write references for unchanged files
4. Store incremental manifest referencing parent backup

**Storage format:**
```
backups/
  ├── 2026-05-11_020000_full/
  │   ├── data/
  │   └── manifest.json
  └── 2026-05-11_140000_incr/
      ├── data/ (only changed files)
      └── manifest.json (references parent)
```

---

## 8. Error Handling

### 8.1 Error Scenarios

| Scenario | Handling |
|----------|----------|
| **Connection failure** | Retry 3x with exponential backoff (1s, 2s, 4s), then mark failed |
| **Partial backup** | Mark as failed if any source file fails; keep partial for inspection |
| **Storage full** | Check before starting (>10% free required); alert if threshold reached |
| **Checksum mismatch** | Delete corrupted backup, retry once, alert admin |
| **Concurrent jobs** | Queue if resource limit reached; configurable max concurrent |
| **Source unavailable** | Mark failed, schedule retry based on policy |
| **Network interruption** | Resume capability for large transfers (SSH/SCP) |

### 8.2 Logging

- Structured JSON logging for machine parsing
- Separate logs per execution: `/var/log/backup-tool/executions/{id}.log`
- Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
- Frontend can stream logs via WebSocket (future) or polling

---

## 9. Extensibility

### 9.1 Source Adapter Extension

To add a new source type (e.g., S3):

1. Create class extending `SourceAdapter`
2. Implement required methods
3. Register in adapter factory
4. Add UI form components for configuration

### 9.2 Storage Backend Extension

Storage backends follow similar pattern:
1. Implement `StorageBackend` interface
2. Support `store()`, `retrieve()`, `delete()`, `list()` operations
3. Register in backend factory

### 9.3 Notification Channels

Notification system supports pluggable channels:
- Webhook (generic HTTP POST)
- Email (SMTP)
- Slack/Discord (webhook URLs)

---

## 10. Security Considerations

- **Credential storage**: Source credentials stored encrypted in SQLite (Fernet encryption)
- **API authentication**: JWT tokens or API keys
- **Backup encryption**: Optional AES-256 encryption of archives
- **Transport security**: SSH for remote sources, HTTPS for web UI
- **File permissions**: Backup archives readable only by backup service user

---

## 11. Performance Considerations

- **Streaming**: Large files streamed rather than loaded into memory
- **Async I/O**: All source adapters use async operations
- **Pagination**: API endpoints paginated (50 items default)
- **Database indexing**: Indexed on frequently queried columns (job_id, status, created_at)
- **Background tasks**: Long-running backups execute in background workers

---

## 12. Deployment

### 12.1 Development
```bash
# Backend
pip install -r requirements.txt
uvicorn main:app --reload

# Frontend
cd frontend
npm install
npm run dev
```

### 12.2 Production
- Backend: Uvicorn with Gunicorn (workers=4)
- Frontend: Built static files served by FastAPI
- Systemd service for automatic startup
- SQLite: Single file, backup with `.backup` command

---

## 13. Testing Strategy

### 13.1 Backend Tests
- **Unit tests**: Adapter mocking, engine logic
- **Integration tests**: Database operations, API endpoints
- **End-to-end tests**: Full backup flows with test fixtures

### 13.2 Frontend Tests
- **Component tests**: React Testing Library
- **Integration tests**: API mocking with MSW
- **E2E tests**: Playwright for critical flows

---

## 14. Future Roadmap

### v1.1
- Cloud storage backends (S3, Azure Blob)
- Backup verification (automated restore testing)
- Email notifications

### v1.2
- Multi-node backup (agent-based architecture)
- Backup encryption at rest
- WebSocket live log streaming

### v2.0
- REST API for external integrations
- Backup reporting and analytics
- Role-based access control

---

## 15. Appendix

### 15.1 Cron Expression Examples

| Expression | Schedule |
|------------|----------|
| `0 2 * * *` | Daily at 2:00 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 0 * * 0` | Weekly on Sunday |
| `0 0 1 * *` | Monthly on 1st |

### 15.2 Database Migration Strategy

- Alembic for schema migrations
- One migration per release
- Backward compatibility for rolling updates
- Migration tests in CI pipeline

### 15.3 Backup Archive Format

```
{destination_path}/{job_id}/
  ├── 2026-05-11_020000/
  │   ├── manifest.json      # Metadata and file list
  │   ├── data.tar.gz        # Compressed archive (or directory tree)
  │   └── checksum.sha256    # Integrity verification
  └── latest -> 2026-05-11_020000/  # Symlink to latest backup
```

---

**End of Specification**
