backup-tool/docs/superpowers/specs/2026-05-11-backup-tool-design.md

Backup Tool Design Specification

Date: 2026-05-11 Status: Approved Target: Small team / SMB backup management

---

1. Overview

A web-based backup management tool for small teams and SMBs. The tool pulls data from various sources (local filesystem, remote via SSH, databases) to a central backup server, providing a dashboard for monitoring and management.

Goals

• Centralized backup management with web UI
• Support for local, SSH, and database sources
• Full and incremental backup strategies
• Manual and scheduled job execution
• SQLite-based persistence for simplicity
• Extensible architecture for future source types and storage backends

Non-Goals

• Enterprise-scale distributed backup (1000+ nodes)
• Real-time continuous backup (near-CDP)
• Built-in cloud storage (S3, Azure Blob) in v1
• Multi-tenancy or RBAC beyond basic auth

---

2. Architecture

2.1 High-Level Design

┌─────────────────────────────────────────────┐
│           Frontend Layer (React)             │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐    │
│  │ Dashboard│ │ Backups  │ │ Settings │    │
│  └──────────┘ └──────────┘ └──────────┘    │
└──────────────────┬──────────────────────────┘
                   │ REST API
┌──────────────────▼──────────────────────────┐
│          Backend Layer (Python)              │
│  ┌────────┐ ┌──────────┐ ┌──────────┐      │
│  │ API    │ │ Backup   │ │ Scheduler│      │
│  │ Server │ │ Engine   │ │ (APSched)│      │
│  └────────┘ └──────────┘ └──────────┘      │
│  ┌──────────────────────────────────┐       │
│  │     Source Adapters              │       │
│  │  (Local | SSH | Database | ...)  │       │
│  └──────────────────────────────────┘       │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│              Data Layer                      │
│  ┌────────────────┐ ┌──────────────────┐    │
│  │   SQLite DB    │ │   File Storage   │    │
│  │ (Jobs, History)│ │ (Backup Archives)│    │
│  └────────────────┘ └──────────────────┘    │
└─────────────────────────────────────────────┘

2.2 Key Design Principles

1. Source-agnostic: Adapter pattern enables new source types without core changes
2. Job-driven: Everything is a job - manual or scheduled, full or incremental
3. SQLite simplicity: Single-file database, zero external dependencies
4. REST API separation: Clean frontend/backend boundary
5. Extensible: Plugin architecture for sources, storage, notifications

---

3. Technology Stack

Backend

• Python 3.11+
• FastAPI - Async web framework with auto-generated OpenAPI docs
• SQLAlchemy 2.0+ - ORM with async support
• Alembic - Database migrations
• APScheduler - Job scheduling (cron expressions)
• Paramiko - SSH client for remote sources
• rsync/libsync - Incremental file synchronization
• Pydantic - Data validation and settings management

Frontend

• React 18+ with TypeScript
• TanStack Query - Server state management and caching
• React Router 6+ - Client-side routing
• Tailwind CSS - Utility-first styling
• Recharts - Dashboard charts and visualizations
• React Hook Form - Form management

Data Storage

• SQLite - Single-file relational database
• Local filesystem - Backup archive storage

---

4. Data Model

4.1 Entities

sources

Stores backup source configurations.

Column	Type	Description
id	INTEGER PK	Auto-increment primary key
name	TEXT	Human-readable source name
type	TEXT	Source type: local, ssh, database
config	JSON	Type-specific configuration (path, host, credentials, etc.)
created_at	DATETIME	Creation timestamp
updated_at	DATETIME	Last update timestamp

Config examples by type:

• local: `{"path": "/var/data", "exclude": [".tmp", ".log"]}``
• ssh: {"host": "server1", "port": 22, "username": "backup", "path": "/data", "key_path": "/keys/id_rsa"}
• database: {"db_type": "postgresql", "host": "db1", "port": 5432, "database": "app", "username": "backup"}

jobs

Defines backup jobs with strategy and scheduling.

Column	Type	Description
id	INTEGER PK	Auto-increment primary key
name	TEXT	Human-readable job name
source_id	INTEGER FK → sources	Source to backup
strategy	TEXT	full or incremental
destination_path	TEXT	Local path for backup storage
exclude_patterns	JSON	Additional exclude patterns (merged with source config)
enabled	BOOLEAN	Whether job is active
created_at	DATETIME	Creation timestamp
updated_at	DATETIME	Last update timestamp

schedules

Cron-based scheduling for jobs (1:1 with jobs for simplicity).

Column	Type	Description
id	INTEGER PK	Auto-increment primary key
job_id	INTEGER FK → jobs	Associated job
cron_expression	TEXT	Cron expression (e.g., "0 2 * * *" for daily 2 AM)
enabled	BOOLEAN	Whether schedule is active
created_at	DATETIME	Creation timestamp

job_executions

Tracks each job run with status and metrics.

Column	Type	Description
id	INTEGER PK	Auto-increment primary key
job_id	INTEGER FK → jobs	Executed job
status	TEXT	pending, running, success, failed, cancelled
started_at	DATETIME	Execution start time
completed_at	DATETIME	Execution end time (NULL if running)
bytes_processed	INTEGER	Total bytes read from source
bytes_backed_up	INTEGER	Total bytes written to destination
error_message	TEXT	Error details if failed
triggered_by	TEXT	manual or schedule

backups

Individual backup archives created by executions.

Column	Type	Description
id	INTEGER PK	Auto-increment primary key
execution_id	INTEGER FK → job_executions	Parent execution
storage_path	TEXT	Path to backup archive on disk
size_bytes	INTEGER	Archive size in bytes
checksum	TEXT	SHA-256 checksum for integrity verification
type	TEXT	full or incremental
parent_backup_id	INTEGER FK → backups	Previous backup in incremental chain (NULL for full)
created_at	DATETIME	Creation timestamp

settings

Key-value application configuration.

Column	Type	Description
key	TEXT PK	Setting identifier
value	TEXT	Setting value (JSON-encoded if complex)
updated_at	DATETIME	Last update timestamp

4.2 Relationships

• sources → jobs: One-to-many (one source can have multiple jobs)
• jobs → schedules: One-to-one (each job has one schedule)
• jobs → job_executions: One-to-many (job run history)
• job_executions → backups: One-to-many (execution produces archives)
• backups → backups: Self-referencing (incremental chain via parent_backup_id)

---

5. Backend Components

5.1 API Server (FastAPI)

Responsibilities:

• Expose REST endpoints for frontend
• Validate requests with Pydantic schemas
• Handle authentication (basic auth or API keys in v1)
• Serve static frontend files in production

Key Endpoints:

• GET /api/dashboard - Dashboard stats and recent activity
• GET /api/sources - List all sources
• POST /api/sources - Create new source
• PUT /api/sources/{id} - Update source
• DELETE /api/sources/{id} - Delete source
• GET /api/jobs - List all jobs
• POST /api/jobs - Create new job
• PUT /api/jobs/{id} - Update job
• DELETE /api/jobs/{id} - Delete job
• POST /api/jobs/{id}/run - Trigger manual execution
• GET /api/jobs/{id}/executions - Get execution history
• GET /api/executions/{id} - Get execution details
• GET /api/executions/{id}/logs - Get execution logs
• GET /api/backups - List all backups
• GET /api/backups/{id}/download - Download backup archive
• DELETE /api/backups/{id} - Delete backup
• GET /api/settings - Get all settings
• PUT /api/settings - Update settings

5.2 Backup Engine

Responsibilities:

• Execute backup jobs (full and incremental)
• Coordinate source adapters for data retrieval
• Handle compression and encryption
• Calculate and verify checksums
• Update execution status in real-time

Flow:

1. Receive job execution request
2. Load source configuration
3. Instantiate appropriate source adapter
4. Connect to source
5. Determine strategy (full vs incremental based on history)
6. Transfer data using adapter
7. Compress and encrypt (if configured)
8. Calculate checksums
9. Store metadata in database
10. Apply retention policy

5.3 Scheduler (APScheduler)

Responsibilities:

• Parse and evaluate cron expressions
• Trigger job executions at scheduled times
• Handle timezone support
• Provide next-run predictions for UI

Configuration:

• Uses SQLite backend for job persistence (survives restarts)
• AsyncIO executor for non-blocking operation
• Misfire grace period: 15 minutes

5.4 Source Adapters

Abstract Base Class Interface:

class SourceAdapter(ABC):
    @abstractmethod
    async def connect(self) -> None: ...
    
    @abstractmethod
    async def disconnect(self) -> None: ...
    
    @abstractmethod
    async def list_files(self, path: str) -> List[FileInfo]: ...
    
    @abstractmethod
    async def read_file(self, path: str) -> AsyncIterator[bytes]: ...
    
    @abstractmethod
    async def get_database_dump(self, config: dict) -> AsyncIterator[bytes]: ...

Implementations:

• LocalAdapter: Direct filesystem access
• SSHAdapter: Paramiko-based SSH/SFTP connection
• DatabaseAdapter: Uses native CLI tools (pg_dump, mysqldump) for consistency

---

6. Frontend Views

6.1 Dashboard

Purpose: At-a-glance system health and recent activity

Components:

• Stats cards: Active jobs, total backups, storage used, recent failures
• Recent activity feed: Last 10 executions with status
• Storage overview chart: Pie chart showing storage by job
• Quick actions: Run job, view failed executions

6.2 Backups View

Purpose: Job and source management

Components:

• Job listing table: Name, source, strategy, schedule, last run, status
• Source listing: Name, type, connection status
• Create/Edit Job modal: Form with source selection, strategy, destination, schedule
• Create/Edit Source modal: Type-specific configuration forms
• Execution history per job: Expandable rows showing past runs
• Manual run button: Trigger immediate execution with confirmation

6.3 Settings View

Purpose: Application configuration

Sections:

• General: Default backup location, retention policy, compression
• Storage: Storage path, disk usage warnings
• Notifications: Webhook URLs, email settings
• Security: Encryption toggle, key management
• Logs: Log level, retention, download

---

7. Backup Execution Flow

7.1 Normal Flow

Trigger (Manual/Schedule)
    ↓
Create Execution Record (status: pending)
    ↓
Connect to Source (via adapter)
    ↓
Determine Strategy:
    - If no previous full backup → Full
    - If strategy = full → Full
    - If strategy = incremental → Incremental (link to parent)
    ↓
Execute Backup:
    - Stream data from source
    - Compress (if enabled)
    - Encrypt (if enabled)
    - Calculate checksums
    ↓
Verify & Store:
    - Verify checksum
    - Write metadata to backups table
    - Update execution status → success
    ↓
Cleanup & Retention:
    - Apply retention policy
    - Delete old backups
    - Update storage stats

7.2 Incremental Backup Strategy

For incremental backups, the system uses file-level deduplication:

1. Compare file metadata (mtime, size) against last backup
2. Only transfer changed files
3. Create hard links or copy-on-write references for unchanged files
4. Store incremental manifest referencing parent backup

Storage format:

backups/
  ├── 2026-05-11_020000_full/
  │   ├── data/
  │   └── manifest.json
  └── 2026-05-11_140000_incr/
      ├── data/ (only changed files)
      └── manifest.json (references parent)

---

8. Error Handling

8.1 Error Scenarios

Scenario	Handling
Connection failure	Retry 3x with exponential backoff (1s, 2s, 4s), then mark failed
Partial backup	Mark as failed if any source file fails; keep partial for inspection
Storage full	Check before starting (>10% free required); alert if threshold reached
Checksum mismatch	Delete corrupted backup, retry once, alert admin
Concurrent jobs	Queue if resource limit reached; configurable max concurrent
Source unavailable	Mark failed, schedule retry based on policy
Network interruption	Resume capability for large transfers (SSH/SCP)

8.2 Logging

• Structured JSON logging for machine parsing
• Separate logs per execution: /var/log/backup-tool/executions/{id}.log
• Log levels: DEBUG, INFO, WARNING, ERROR, CRITICAL
• Frontend can stream logs via WebSocket (future) or polling

---

9. Extensibility

9.1 Source Adapter Extension

To add a new source type (e.g., S3):

1. Create class extending SourceAdapter
2. Implement required methods
3. Register in adapter factory
4. Add UI form components for configuration

9.2 Storage Backend Extension

Storage backends follow similar pattern:

1. Implement StorageBackend interface
2. Support store(), retrieve(), delete(), list() operations
3. Register in backend factory

9.3 Notification Channels

Notification system supports pluggable channels:

• Webhook (generic HTTP POST)
• Email (SMTP)
• Slack/Discord (webhook URLs)

---

10. Security Considerations

• Credential storage: Source credentials stored encrypted in SQLite (Fernet encryption)
• API authentication: JWT tokens or API keys
• Backup encryption: Optional AES-256 encryption of archives
• Transport security: SSH for remote sources, HTTPS for web UI
• File permissions: Backup archives readable only by backup service user

---

11. Performance Considerations

• Streaming: Large files streamed rather than loaded into memory
• Async I/O: All source adapters use async operations
• Pagination: API endpoints paginated (50 items default)
• Database indexing: Indexed on frequently queried columns (job_id, status, created_at)
• Background tasks: Long-running backups execute in background workers

---

12. Deployment

12.1 Development

# Backend
pip install -r requirements.txt
uvicorn main:app --reload

# Frontend
cd frontend
npm install
npm run dev

12.2 Production

• Backend: Uvicorn with Gunicorn (workers=4)
• Frontend: Built static files served by FastAPI
• Systemd service for automatic startup
• SQLite: Single file, backup with .backup command

---

13. Testing Strategy

13.1 Backend Tests

• Unit tests: Adapter mocking, engine logic
• Integration tests: Database operations, API endpoints
• End-to-end tests: Full backup flows with test fixtures

13.2 Frontend Tests

• Component tests: React Testing Library
• Integration tests: API mocking with MSW
• E2E tests: Playwright for critical flows

---

14. Future Roadmap

v1.1

• Cloud storage backends (S3, Azure Blob)
• Backup verification (automated restore testing)
• Email notifications

v1.2

• Multi-node backup (agent-based architecture)
• Backup encryption at rest
• WebSocket live log streaming

v2.0

• REST API for external integrations
• Backup reporting and analytics
• Role-based access control

---

15. Appendix

15.1 Cron Expression Examples

Expression	Schedule
0 2 * * *	Daily at 2:00 AM
0 */6 * * *	Every 6 hours
0 0 * * 0	Weekly on Sunday
0 0 1 * *	Monthly on 1st

15.2 Database Migration Strategy

• Alembic for schema migrations
• One migration per release
• Backward compatibility for rolling updates
• Migration tests in CI pipeline

15.3 Backup Archive Format

{destination_path}/{job_id}/
  ├── 2026-05-11_020000/
  │   ├── manifest.json      # Metadata and file list
  │   ├── data.tar.gz        # Compressed archive (or directory tree)
  │   └── checksum.sha256    # Integrity verification
  └── latest -> 2026-05-11_020000/  # Symlink to latest backup

---

End of Specification