Files

T

alex d35f8d408f docs: add backup monitoring implementation plan

2026-05-11 20:18:29 +02:00

67 KiB

Raw Permalink Blame History

Backup Monitoring Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a standalone backup monitoring system that receives HTTP reports from an external backup tool, stores job/run/alert data in SQLite, and provides a React frontend with validation, alerting, and charts.

Architecture: FastAPI backend with dedicated SQLite tables (backup_jobs, backup_runs, backup_alerts), background poller for missed schedule detection, React frontend with TanStack Query and D3 charts, Bearer token auth for the backup tool API.

Tech Stack: FastAPI, SQLite (WAL), Pydantic, React, TypeScript, TanStack Query, D3, MUI

File Structure

Backend

File	Responsibility
`backend/src/media_library_viewer_api/services/settings_store.py`	Add CRUD methods for backup tables
`backend/src/media_library_viewer_api/models/backups.py`	Pydantic models for request/response validation
`backend/src/media_library_viewer_api/services/backup_alert_engine.py`	Alert generation logic (failed, missed, anomaly)
`backend/src/media_library_viewer_api/services/backup_poller.py`	Background poller for missed schedule checks
`backend/src/media_library_viewer_api/routers/backups.py`	FastAPI router with all backup endpoints
`backend/src/media_library_viewer_api/routers/dashboard.py`	Add `/api/dashboard/backups` endpoint
`backend/src/media_library_viewer_api/main.py`	Register backup router and start poller
`backend/src/media_library_viewer_api/auth.py`	Add API key auth dependency
`backend/tests/test_backups.py`	API and integration tests

Frontend

File	Responsibility
`frontend/src/types/backups.ts`	TypeScript interfaces for backup data
`frontend/src/hooks/useBackups.ts`	TanStack Query hooks for backup API
`frontend/src/api/backups.ts`	API client functions
`frontend/src/components/BackupsPage.tsx`	Main backups page with tabs
`frontend/src/components/BackupJobsTable.tsx`	Jobs list table
`frontend/src/components/BackupRunsTable.tsx`	Runs list table with filters
`frontend/src/components/BackupAlertsTable.tsx`	Alerts table with acknowledge
`frontend/src/components/BackupDashboardWidget.tsx`	Dashboard summary widget
`frontend/src/App.tsx`	Add `/backups` route and nav tab

Task 1: Database Schema

Files:

Modify: backend/src/media_library_viewer_api/services/settings_store.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_backup_schema_created():
    store = get_settings_store()
    store.init_schema()
    with store.connect() as conn:
        tables = conn.execute("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'backup_%'").fetchall()
        table_names = [t[0] for t in tables]
        assert "backup_jobs" in table_names
        assert "backup_runs" in table_names
        assert "backup_alerts" in table_names

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_backup_schema_created -v Expected: FAIL — tables do not exist

Step 3: Add schema creation to init_schema

In backend/src/media_library_viewer_api/services/settings_store.py, add to the init_schema method (after existing table creation):

def init_schema(self) -> None:
    # ... existing code ...
    with self.connect() as conn:
        # ... existing tables ...
        conn.execute("""
            CREATE TABLE IF NOT EXISTS backup_jobs (
                id TEXT PRIMARY KEY,
                name TEXT NOT NULL UNIQUE,
                source TEXT,
                target TEXT,
                schedule_interval_seconds INTEGER,
                created_at INTEGER NOT NULL
            )
        """)
        conn.execute("""
            CREATE TABLE IF NOT EXISTS backup_runs (
                id TEXT PRIMARY KEY,
                job_id TEXT NOT NULL,
                started_at INTEGER NOT NULL,
                ended_at INTEGER,
                status TEXT NOT NULL,
                bytes_transferred INTEGER,
                duration_ms INTEGER,
                error_message TEXT,
                details_json TEXT,
                created_at INTEGER NOT NULL,
                FOREIGN KEY (job_id) REFERENCES backup_jobs(id)
            )
        """)
        conn.execute("CREATE INDEX IF NOT EXISTS idx_backup_runs_job_id ON backup_runs(job_id)")
        conn.execute("CREATE INDEX IF NOT EXISTS idx_backup_runs_status ON backup_runs(status)")
        conn.execute("""
            CREATE TABLE IF NOT EXISTS backup_alerts (
                id TEXT PRIMARY KEY,
                job_id TEXT NOT NULL,
                run_id TEXT,
                alert_type TEXT NOT NULL,
                severity TEXT NOT NULL,
                message TEXT NOT NULL,
                acknowledged INTEGER NOT NULL DEFAULT 0,
                resolved_at INTEGER,
                created_at INTEGER NOT NULL,
                FOREIGN KEY (job_id) REFERENCES backup_jobs(id),
                FOREIGN KEY (run_id) REFERENCES backup_runs(id)
            )
        """)
        conn.execute("CREATE INDEX IF NOT EXISTS idx_backup_alerts_job_id ON backup_alerts(job_id)")
        conn.execute("CREATE INDEX IF NOT EXISTS idx_backup_alerts_acknowledged ON backup_alerts(acknowledged)")

Step 4: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_backup_schema_created -v Expected: PASS

Step 5: Commit

git add backend/src/media_library_viewer_api/services/settings_store.py backend/tests/test_backups.py
git commit -m "feat: add backup monitoring database schema"

Task 2: Pydantic Models

Files:

Create: backend/src/media_library_viewer_api/models/backups.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_backup_report_model():
    from media_library_viewer_api.models.backups import BackupReportRequest
    
    report = BackupReportRequest(
        name="test-backup",
        started_at="2026-05-11T02:00:00Z",
        status="success",
        ended_at="2026-05-11T02:15:00Z",
        duration_ms=900000,
        bytes_transferred=1024,
    )
    assert report.name == "test-backup"
    assert report.status == "success"

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_backup_report_model -v Expected: FAIL — module not found

Step 3: Create Pydantic models

Create backend/src/media_library_viewer_api/models/backups.py:

from datetime import datetime
from typing import Any
from pydantic import BaseModel, Field, field_validator


class BackupReportRequest(BaseModel):
    name: str = Field(..., min_length=1)
    source: str | None = None
    target: str | None = None
    schedule_interval_seconds: int | None = Field(None, gt=0)
    started_at: datetime
    ended_at: datetime | None = None
    status: str = Field(..., pattern="^(success|failure|in_progress)$")
    bytes_transferred: int | None = Field(None, ge=0)
    duration_ms: int | None = None
    error_message: str | None = None
    details: dict[str, Any] | None = None

    @field_validator("ended_at", "duration_ms")
    @classmethod
    def validate_success_fields(cls, v, info):
        if info.data.get("status") == "success" and v is None:
            raise ValueError(f"{info.field_name} is required when status is 'success'")
        return v


class BackupJobResponse(BaseModel):
    id: str
    name: str
    source: str | None
    target: str | None
    schedule_interval_seconds: int | None
    created_at: int


class BackupRunResponse(BaseModel):
    id: str
    job_id: str
    started_at: int
    ended_at: int | None
    status: str
    bytes_transferred: int | None
    duration_ms: int | None
    error_message: str | None
    details_json: dict[str, Any] | None
    created_at: int


class BackupAlertResponse(BaseModel):
    id: str
    job_id: str
    run_id: str | None
    alert_type: str
    severity: str
    message: str
    acknowledged: bool
    resolved_at: int | None
    created_at: int


class BackupDashboardSummary(BaseModel):
    total_jobs: int
    success_rate_24h: float
    active_alerts: int
    last_failed_at: int | None

Step 4: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_backup_report_model -v Expected: PASS

Step 5: Commit

git add backend/src/media_library_viewer_api/models/backups.py backend/tests/test_backups.py
git commit -m "feat: add backup monitoring pydantic models"

Task 3: Settings Store CRUD Methods

Files:

Modify: backend/src/media_library_viewer_api/services/settings_store.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_create_job_and_run():
    store = get_settings_store()
    store.init_schema()
    
    job = store.upsert_backup_job({
        "name": "test-job",
        "source": "server-a:/data",
        "target": "server-b:/backups",
        "schedule_interval_seconds": 86400,
    })
    assert job["name"] == "test-job"
    
    run = store.create_backup_run({
        "job_id": job["id"],
        "started_at": 1715392800,
        "status": "success",
        "ended_at": 1715393700,
        "duration_ms": 900000,
        "bytes_transferred": 1024,
    })
    assert run["job_id"] == job["id"]
    assert run["status"] == "success"

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_create_job_and_run -v Expected: FAIL — methods don't exist

Step 3: Implement CRUD methods

Add to backend/src/media_library_viewer_api/services/settings_store.py:

def _row_to_job(self, row: sqlite3.Row) -> dict[str, Any]:
    return {
        "id": row["id"],
        "name": row["name"],
        "source": row["source"],
        "target": row["target"],
        "schedule_interval_seconds": row["schedule_interval_seconds"],
        "created_at": row["created_at"],
    }

def _row_to_run(self, row: sqlite3.Row) -> dict[str, Any]:
    details = None
    if row["details_json"]:
        try:
            details = json.loads(row["details_json"])
        except json.JSONDecodeError:
            details = {"raw": row["details_json"]}
    return {
        "id": row["id"],
        "job_id": row["job_id"],
        "started_at": row["started_at"],
        "ended_at": row["ended_at"],
        "status": row["status"],
        "bytes_transferred": row["bytes_transferred"],
        "duration_ms": row["duration_ms"],
        "error_message": row["error_message"],
        "details_json": details,
        "created_at": row["created_at"],
    }

def _row_to_alert(self, row: sqlite3.Row) -> dict[str, Any]:
    return {
        "id": row["id"],
        "job_id": row["job_id"],
        "run_id": row["run_id"],
        "alert_type": row["alert_type"],
        "severity": row["severity"],
        "message": row["message"],
        "acknowledged": bool(row["acknowledged"]),
        "resolved_at": row["resolved_at"],
        "created_at": row["created_at"],
    }

def get_backup_job_by_name(self, name: str) -> dict[str, Any] | None:
    self.init_schema()
    with self.connect() as conn:
        row = conn.execute("SELECT * FROM backup_jobs WHERE name = ?", (name,)).fetchone()
    return self._row_to_job(row) if row else None

def upsert_backup_job(self, payload: dict[str, Any]) -> dict[str, Any]:
    self.init_schema()
    job_id = payload.get("id") or str(uuid.uuid4())
    now = int(time.time())
    with self.connect() as conn:
        existing = conn.execute("SELECT created_at FROM backup_jobs WHERE id = ?", (job_id,)).fetchone()
        if not existing:
            existing = conn.execute("SELECT id, created_at FROM backup_jobs WHERE name = ?", (payload["name"],)).fetchone()
            if existing:
                job_id = existing["id"]
        created_at = int(existing["created_at"]) if existing else now
        conn.execute("""
            INSERT INTO backup_jobs (id, name, source, target, schedule_interval_seconds, created_at)
            VALUES (?, ?, ?, ?, ?, ?)
            ON CONFLICT(id) DO UPDATE SET
                name = excluded.name,
                source = COALESCE(excluded.source, source),
                target = COALESCE(excluded.target, target),
                schedule_interval_seconds = COALESCE(excluded.schedule_interval_seconds, schedule_interval_seconds),
                created_at = excluded.created_at
        """, (job_id, payload["name"], payload.get("source"), payload.get("target"),
              payload.get("schedule_interval_seconds"), created_at))
    return self.get_backup_job(job_id) or {"id": job_id, **payload, "created_at": created_at}

def get_backup_job(self, job_id: str) -> dict[str, Any] | None:
    self.init_schema()
    with self.connect() as conn:
        row = conn.execute("SELECT * FROM backup_jobs WHERE id = ?", (job_id,)).fetchone()
    return self._row_to_job(row) if row else None

def list_backup_jobs(self) -> list[dict[str, Any]]:
    self.init_schema()
    with self.connect() as conn:
        rows = conn.execute("SELECT * FROM backup_jobs ORDER BY created_at DESC").fetchall()
    return [self._row_to_job(row) for row in rows]

def create_backup_run(self, payload: dict[str, Any]) -> dict[str, Any]:
    self.init_schema()
    run_id = str(uuid.uuid4())
    now = int(time.time())
    with self.connect() as conn:
        conn.execute("""
            INSERT INTO backup_runs (id, job_id, started_at, ended_at, status, bytes_transferred, duration_ms, error_message, details_json, created_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
        """, (run_id, payload["job_id"], payload["started_at"], payload.get("ended_at"),
              payload["status"], payload.get("bytes_transferred"), payload.get("duration_ms"),
              payload.get("error_message"), json.dumps(payload.get("details_json") or {}), now))
    return self.get_backup_run(run_id) or {"id": run_id, **payload, "created_at": now}

def get_backup_run(self, run_id: str) -> dict[str, Any] | None:
    self.init_schema()
    with self.connect() as conn:
        row = conn.execute("SELECT * FROM backup_runs WHERE id = ?", (run_id,)).fetchone()
    return self._row_to_run(row) if row else None

def list_backup_runs(self, job_id: str | None = None, status: str | None = None, limit: int = 50) -> list[dict[str, Any]]:
    self.init_schema()
    query = "SELECT * FROM backup_runs WHERE 1=1"
    params: list[Any] = []
    if job_id:
        query += " AND job_id = ?"
        params.append(job_id)
    if status:
        query += " AND status = ?"
        params.append(status)
    query += " ORDER BY started_at DESC LIMIT ?"
    params.append(limit)
    with self.connect() as conn:
        rows = conn.execute(query, params).fetchall()
    return [self._row_to_run(row) for row in rows]

def get_latest_backup_run(self, job_id: str) -> dict[str, Any] | None:
    self.init_schema()
    with self.connect() as conn:
        row = conn.execute(
            "SELECT * FROM backup_runs WHERE job_id = ? ORDER BY started_at DESC LIMIT 1",
            (job_id,)
        ).fetchone()
    return self._row_to_run(row) if row else None

def create_backup_alert(self, payload: dict[str, Any]) -> dict[str, Any]:
    self.init_schema()
    alert_id = str(uuid.uuid4())
    now = int(time.time())
    with self.connect() as conn:
        conn.execute("""
            INSERT INTO backup_alerts (id, job_id, run_id, alert_type, severity, message, acknowledged, resolved_at, created_at)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
        """, (alert_id, payload["job_id"], payload.get("run_id"), payload["alert_type"],
              payload["severity"], payload["message"], 0, None, now))
    return self.get_backup_alert(alert_id) or {"id": alert_id, **payload, "acknowledged": False, "created_at": now}

def get_backup_alert(self, alert_id: str) -> dict[str, Any] | None:
    self.init_schema()
    with self.connect() as conn:
        row = conn.execute("SELECT * FROM backup_alerts WHERE id = ?", (alert_id,)).fetchone()
    return self._row_to_alert(row) if row else None

def list_backup_alerts(self, job_id: str | None = None, acknowledged: bool | None = None, severity: str | None = None) -> list[dict[str, Any]]:
    self.init_schema()
    query = "SELECT * FROM backup_alerts WHERE 1=1"
    params: list[Any] = []
    if job_id:
        query += " AND job_id = ?"
        params.append(job_id)
    if acknowledged is not None:
        query += " AND acknowledged = ?"
        params.append(1 if acknowledged else 0)
    if severity:
        query += " AND severity = ?"
        params.append(severity)
    query += " ORDER BY created_at DESC"
    with self.connect() as conn:
        rows = conn.execute(query, params).fetchall()
    return [self._row_to_alert(row) for row in rows]

def acknowledge_backup_alert(self, alert_id: str) -> dict[str, Any] | None:
    self.init_schema()
    with self.connect() as conn:
        conn.execute("UPDATE backup_alerts SET acknowledged = 1 WHERE id = ?", (alert_id,))
    return self.get_backup_alert(alert_id)

def resolve_backup_alerts_for_job(self, job_id: str, alert_type: str | None = None) -> int:
    self.init_schema()
    now = int(time.time())
    with self.connect() as conn:
        if alert_type:
            result = conn.execute(
                "UPDATE backup_alerts SET resolved_at = ? WHERE job_id = ? AND alert_type = ? AND resolved_at IS NULL",
                (now, job_id, alert_type)
            )
        else:
            result = conn.execute(
                "UPDATE backup_alerts SET resolved_at = ? WHERE job_id = ? AND resolved_at IS NULL",
                (now, job_id)
            )
        return result.rowcount

def prune_backup_alerts(self, cutoff_ts: int) -> int:
    self.init_schema()
    with self.connect() as conn:
        result = conn.execute("DELETE FROM backup_alerts WHERE created_at < ?", (cutoff_ts,))
        return result.rowcount

Step 4: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_create_job_and_run -v Expected: PASS

Step 5: Commit

git add backend/src/media_library_viewer_api/services/settings_store.py backend/tests/test_backups.py
git commit -m "feat: add backup monitoring CRUD operations"

Task 4: Alert Engine

Files:

Create: backend/src/media_library_viewer_api/services/backup_alert_engine.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_alert_failed_status():
    from media_library_viewer_api.services.backup_alert_engine import generate_alerts_for_run
    
    run = {
        "id": "run-1",
        "job_id": "job-1",
        "status": "failure",
        "bytes_transferred": 0,
        "duration_ms": 1000,
        "started_at": 1715392800,
    }
    alerts = generate_alerts_for_run(run, [], None)
    assert len(alerts) == 1
    assert alerts[0]["alert_type"] == "failed_status"
    assert alerts[0]["severity"] == "critical"

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_alert_failed_status -v Expected: FAIL — module not found

Step 3: Implement alert engine

Create backend/src/media_library_viewer_api/services/backup_alert_engine.py:

import statistics
from typing import Any


def generate_alerts_for_run(
    run: dict[str, Any],
    previous_runs: list[dict[str, Any]],
    job: dict[str, Any] | None,
) -> list[dict[str, Any]]:
    alerts = []
    job_id = run["job_id"]
    
    # 1. Failed status alert
    if run["status"] == "failure":
        alerts.append({
            "job_id": job_id,
            "run_id": run["id"],
            "alert_type": "failed_status",
            "severity": "critical",
            "message": f"Backup job '{job['name'] if job else job_id}' failed: {run.get('error_message', 'No error details')}",
        })
    
    # 2. Anomaly size alert
    bytes_transferred = run.get("bytes_transferred")
    if bytes_transferred is not None and previous_runs:
        successful_runs = [r for r in previous_runs if r["status"] == "success" and r.get("bytes_transferred") is not None]
        if len(successful_runs) >= 3:
            sizes = [r["bytes_transferred"] for r in successful_runs[-7:]]
            median_size = statistics.median(sizes)
            if median_size > 0:
                ratio = bytes_transferred / median_size
                if bytes_transferred == 0:
                    alerts.append({
                        "job_id": job_id,
                        "run_id": run["id"],
                        "alert_type": "anomaly_size",
                        "severity": "warning",
                        "message": f"Backup job '{job['name'] if job else job_id}' transferred 0 bytes (median: {median_size})",
                    })
                elif ratio < 0.1 or ratio > 3.0:
                    alerts.append({
                        "job_id": job_id,
                        "run_id": run["id"],
                        "alert_type": "anomaly_size",
                        "severity": "warning",
                        "message": f"Backup job '{job['name'] if job else job_id}' size anomaly: {bytes_transferred} bytes (median: {median_size})",
                    })
    
    # 3. Anomaly duration alert
    duration_ms = run.get("duration_ms")
    if duration_ms is not None and previous_runs:
        successful_runs = [r for r in previous_runs if r["status"] == "success" and r.get("duration_ms") is not None]
        if len(successful_runs) >= 3:
            durations = [r["duration_ms"] for r in successful_runs[-7:]]
            median_duration = statistics.median(durations)
            if median_duration > 0 and duration_ms / median_duration > 3.0:
                alerts.append({
                    "job_id": job_id,
                    "run_id": run["id"],
                    "alert_type": "anomaly_duration",
                    "severity": "warning",
                    "message": f"Backup job '{job['name'] if job else job_id}' duration anomaly: {duration_ms}ms (median: {median_duration}ms)",
                })
    
    return alerts


def check_missed_schedules(
    jobs: list[dict[str, Any]],
    get_latest_run: callable,
    existing_alerts: list[dict[str, Any]],
) -> list[dict[str, Any]]:
    alerts = []
    now = int(__import__("time").time())
    
    for job in jobs:
        interval = job.get("schedule_interval_seconds")
        if not interval:
            continue
        
        latest_run = get_latest_run(job["id"])
        if not latest_run:
            # No runs ever — alert if job is older than interval * 1.5
            if now - job["created_at"] > interval * 1.5:
                alerts.append({
                    "job_id": job["id"],
                    "run_id": None,
                    "alert_type": "missed_schedule",
                    "severity": "warning",
                    "message": f"Backup job '{job['name']}' has never run (expected every {interval}s)",
                })
        else:
            last_run_time = latest_run["started_at"]
            if now - last_run_time > interval * 1.5:
                # Check if there's already an unresolved missed_schedule alert
                has_open_alert = any(
                    a["alert_type"] == "missed_schedule" and a["resolved_at"] is None
                    for a in existing_alerts if a["job_id"] == job["id"]
                )
                if not has_open_alert:
                    alerts.append({
                        "job_id": job["id"],
                        "run_id": None,
                        "alert_type": "missed_schedule",
                        "severity": "warning",
                        "message": f"Backup job '{job['name']}' missed schedule: last run at {last_run_time} (expected every {interval}s)",
                    })
    
    return alerts

Step 4: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_alert_failed_status -v Expected: PASS

Step 5: Commit

git add backend/src/media_library_viewer_api/services/backup_alert_engine.py backend/tests/test_backups.py
git commit -m "feat: add backup alert generation engine"

Task 5: Background Poller

Files:

Create: backend/src/media_library_viewer_api/services/backup_poller.py
Modify: backend/src/media_library_viewer_api/main.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_backup_poller_detects_missed_schedule():
    from media_library_viewer_api.services.backup_poller import BackupAlertPoller
    
    store = get_settings_store()
    store.init_schema()
    
    job = store.upsert_backup_job({
        "name": "old-job",
        "schedule_interval_seconds": 60,
    })
    
    poller = BackupAlertPoller()
    poller._run_cycle(store)
    
    alerts = store.list_backup_alerts(job_id=job["id"])
    assert len(alerts) == 1
    assert alerts[0]["alert_type"] == "missed_schedule"

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_backup_poller_detects_missed_schedule -v Expected: FAIL — module not found

Step 3: Implement backup poller

Create backend/src/media_library_viewer_api/services/backup_poller.py:

import logging
import threading
import time
from typing import Any

from .backup_alert_engine import check_missed_schedules
from .settings_store import SettingsStore, get_settings_store

logger = logging.getLogger(__name__)


class BackupAlertPoller:
    def __init__(self) -> None:
        self._thread: threading.Thread | None = None
        self._stop_event = threading.Event()
        self._lock = threading.Lock()
        self._last_run_at: float | None = None
        self._last_success_at: float | None = None
        self._last_error: str = ""
        self._poll_count = 0
        self._error_count = 0

    def start(self) -> None:
        with self._lock:
            if self._thread and self._thread.is_alive():
                return
            self._stop_event.clear()
            self._thread = threading.Thread(target=self._run, name="backup-alert-poller", daemon=True)
            self._thread.start()

    def stop(self, timeout: float = 5.0) -> None:
        with self._lock:
            thread = self._thread
            if not thread:
                return
            self._stop_event.set()
        thread.join(timeout=timeout)

    @property
    def status(self) -> dict[str, Any]:
        with self._lock:
            return {
                "worker_running": self._thread is not None and self._thread.is_alive(),
                "last_run_at": self._last_run_at,
                "last_success_at": self._last_success_at,
                "last_error": self._last_error,
                "poll_count": self._poll_count,
                "error_count": self._error_count,
            }

    def _run(self) -> None:
        # Wait 30 seconds before first check
        if self._stop_event.wait(30):
            return
        store = get_settings_store()
        while not self._stop_event.is_set():
            try:
                self._run_cycle(store)
            except Exception:
                logger.exception("Backup alert poller cycle failed")
            # Check every 5 minutes
            if self._stop_event.wait(300):
                break

    def _run_cycle(self, store: SettingsStore) -> None:
        start = time.perf_counter()
        jobs = store.list_backup_jobs()
        existing_alerts = store.list_backup_alerts(acknowledged=False)
        
        new_alerts = check_missed_schedules(
            jobs,
            lambda job_id: store.get_latest_backup_run(job_id),
            existing_alerts,
        )
        
        for alert in new_alerts:
            store.create_backup_alert(alert)
        
        # Prune old resolved alerts (90 days)
        cutoff = int(time.time()) - (90 * 24 * 60 * 60)
        store.prune_backup_alerts(cutoff)
        
        duration_ms = int((time.perf_counter() - start) * 1000)
        with self._lock:
            self._last_run_at = time.time()
            self._last_success_at = time.time()
            self._poll_count += 1


_BACKUP_POLLER = BackupAlertPoller()


def get_backup_poller() -> BackupAlertPoller:
    return _BACKUP_POLLER

Step 4: Wire up poller in main.py

Modify backend/src/media_library_viewer_api/main.py:

# Add import
from .services.backup_poller import get_backup_poller

# In lifespan:
@asynccontextmanager
async def lifespan(app: FastAPI):
    settings = get_settings()
    configure_logging(settings.log_level)
    validate_auth_settings(settings)
    mail_queue = get_mail_queue()
    monitoring_poller = get_monitoring_poller()
    backup_poller = get_backup_poller()
    mail_queue.start()
    monitoring_poller.start()
    backup_poller.start()
    yield
    monitoring_poller.stop()
    backup_poller.stop()
    mail_queue.stop()

Step 5: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_backup_poller_detects_missed_schedule -v Expected: PASS

Step 6: Commit

git add backend/src/media_library_viewer_api/services/backup_poller.py backend/src/media_library_viewer_api/main.py backend/tests/test_backups.py
git commit -m "feat: add backup alert background poller"

Task 6: API Key Authentication

Files:

Modify: backend/src/media_library_viewer_api/auth.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_api_key_auth():
    from media_library_viewer_api.auth import require_api_key
    from fastapi import HTTPException
    
    # Should raise 401 without key
    try:
        require_api_key("")
        assert False, "Should have raised"
    except HTTPException as e:
        assert e.status_code == 401

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_api_key_auth -v Expected: FAIL — function doesn't exist

Step 3: Implement API key auth

Modify backend/src/media_library_viewer_api/auth.py:

# Add import
import secrets

# Add at module level
_API_KEY: str | None = None

def get_api_key() -> str:
    global _API_KEY
    if _API_KEY is None:
        store = get_settings_store()
        settings = store.get_settings()
        _API_KEY = settings.get("backup_api_key")
        if not _API_KEY:
            _API_KEY = secrets.token_urlsafe(32)
            store.update_setting("backup_api_key", _API_KEY)
    return _API_KEY

def require_api_key(authorization: str = Header("", alias="Authorization")) -> str:
    expected = f"Bearer {get_api_key()}"
    if not authorization or not secrets.compare_digest(authorization, expected):
        raise HTTPException(status_code=401, detail="Invalid or missing API key")
    return authorization

Step 4: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_api_key_auth -v Expected: PASS

Step 5: Commit

git add backend/src/media_library_viewer_api/auth.py backend/tests/test_backups.py
git commit -m "feat: add API key authentication for backup tool"

Task 7: Backup Router (Backend API)

Files:

Create: backend/src/media_library_viewer_api/routers/backups.py
Modify: backend/src/media_library_viewer_api/main.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_post_backup_report():
    from fastapi.testclient import TestClient
    from media_library_viewer_api.main import app
    
    client = TestClient(app)
    
    # Get API key
    from media_library_viewer_api.auth import get_api_key
    api_key = get_api_key()
    
    response = client.post(
        "/api/backups/report",
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "name": "test-backup",
            "started_at": "2026-05-11T02:00:00Z",
            "status": "success",
            "ended_at": "2026-05-11T02:15:00Z",
            "duration_ms": 900000,
            "bytes_transferred": 1024,
        },
    )
    assert response.status_code == 200
    data = response.json()
    assert data["status"] == "success"

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_post_backup_report -v Expected: FAIL — router not registered

Step 3: Implement backup router

Create backend/src/media_library_viewer_api/routers/backups.py:

import time
from typing import Any
from fastapi import APIRouter, Depends, HTTPException

from ..auth import require_api_key
from ..models.backups import (
    BackupAlertResponse,
    BackupDashboardSummary,
    BackupJobResponse,
    BackupReportRequest,
    BackupRunResponse,
)
from ..services.backup_alert_engine import generate_alerts_for_run
from ..services.settings_store import SettingsStore, get_settings_store

router = APIRouter(prefix="/api/backups", tags=["backups"])


def _get_or_create_job(store: SettingsStore, report: BackupReportRequest) -> dict[str, Any]:
    job = store.get_backup_job_by_name(report.name)
    if not job:
        job = store.upsert_backup_job({
            "name": report.name,
            "source": report.source,
            "target": report.target,
            "schedule_interval_seconds": report.schedule_interval_seconds,
        })
    elif report.schedule_interval_seconds:
        # Update schedule if provided
        store.upsert_backup_job({
            "id": job["id"],
            "name": report.name,
            "source": report.source,
            "target": report.target,
            "schedule_interval_seconds": report.schedule_interval_seconds,
        })
        job = store.get_backup_job(job["id"])
    return job


@router.post("/report")
def post_backup_report(
    report: BackupReportRequest,
    store: SettingsStore = Depends(get_settings_store),
    _auth: str = Depends(require_api_key),
) -> BackupRunResponse:
    job = _get_or_create_job(store, report)
    
    # Check for duplicate (same job + started_at within 1s)
    existing_runs = store.list_backup_runs(job_id=job["id"], limit=5)
    started_at_ts = int(report.started_at.timestamp())
    for existing in existing_runs:
        if abs(existing["started_at"] - started_at_ts) <= 1:
            # Update existing run
            return store.get_backup_run(existing["id"])
    
    run_data = {
        "job_id": job["id"],
        "started_at": started_at_ts,
        "ended_at": int(report.ended_at.timestamp()) if report.ended_at else None,
        "status": report.status,
        "bytes_transferred": report.bytes_transferred,
        "duration_ms": report.duration_ms,
        "error_message": report.error_message,
        "details_json": report.details,
    }
    run = store.create_backup_run(run_data)
    
    # Generate alerts
    previous_runs = store.list_backup_runs(job_id=job["id"], status="success", limit=20)
    alerts = generate_alerts_for_run(run, previous_runs, job)
    for alert in alerts:
        store.create_backup_alert(alert)
    
    # Resolve old alerts of the same type if this run is successful
    if report.status == "success":
        store.resolve_backup_alerts_for_job(job["id"], "failed_status")
        store.resolve_backup_alerts_for_job(job["id"], "anomaly_size")
        store.resolve_backup_alerts_for_job(job["id"], "anomaly_duration")
    
    return BackupRunResponse(**run)


@router.post("/report/start")
def post_backup_start(
    report: BackupReportRequest,
    store: SettingsStore = Depends(get_settings_store),
    _auth: str = Depends(require_api_key),
) -> BackupRunResponse:
    job = _get_or_create_job(store, report)
    
    run_data = {
        "job_id": job["id"],
        "started_at": int(report.started_at.timestamp()),
        "status": "in_progress",
    }
    run = store.create_backup_run(run_data)
    return BackupRunResponse(**run)


@router.get("/jobs")
def get_backup_jobs(
    store: SettingsStore = Depends(get_settings_store),
) -> list[BackupJobResponse]:
    jobs = store.list_backup_jobs()
    return [BackupJobResponse(**job) for job in jobs]


@router.get("/jobs/{job_id}")
def get_backup_job(
    job_id: str,
    store: SettingsStore = Depends(get_settings_store),
) -> dict[str, Any]:
    job = store.get_backup_job(job_id)
    if not job:
        raise HTTPException(status_code=404, detail="Backup job not found")
    runs = store.list_backup_runs(job_id=job_id, limit=20)
    return {
        "job": BackupJobResponse(**job),
        "runs": [BackupRunResponse(**run) for run in runs],
    }


@router.get("/runs")
def get_backup_runs(
    job_id: str | None = None,
    status: str | None = None,
    limit: int = 50,
    store: SettingsStore = Depends(get_settings_store),
) -> list[BackupRunResponse]:
    runs = store.list_backup_runs(job_id=job_id, status=status, limit=limit)
    return [BackupRunResponse(**run) for run in runs]


@router.get("/runs/{run_id}")
def get_backup_run(
    run_id: str,
    store: SettingsStore = Depends(get_settings_store),
) -> BackupRunResponse:
    run = store.get_backup_run(run_id)
    if not run:
        raise HTTPException(status_code=404, detail="Backup run not found")
    return BackupRunResponse(**run)


@router.get("/alerts")
def get_backup_alerts(
    job_id: str | None = None,
    acknowledged: bool | None = None,
    severity: str | None = None,
    store: SettingsStore = Depends(get_settings_store),
) -> list[BackupAlertResponse]:
    alerts = store.list_backup_alerts(job_id=job_id, acknowledged=acknowledged, severity=severity)
    return [BackupAlertResponse(**alert) for alert in alerts]


@router.post("/alerts/{alert_id}/acknowledge")
def acknowledge_backup_alert(
    alert_id: str,
    store: SettingsStore = Depends(get_settings_store),
) -> BackupAlertResponse:
    alert = store.acknowledge_backup_alert(alert_id)
    if not alert:
        raise HTTPException(status_code=404, detail="Alert not found")
    return BackupAlertResponse(**alert)

Step 4: Register router in main.py

Add to backend/src/media_library_viewer_api/main.py:

from .routers import backups as backups_router

# In router registration section:
app.include_router(backups_router.router)

Step 5: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_post_backup_report -v Expected: PASS

Step 6: Commit

git add backend/src/media_library_viewer_api/routers/backups.py backend/src/media_library_viewer_api/main.py backend/tests/test_backups.py
git commit -m "feat: add backup monitoring API endpoints"

Task 8: Dashboard Endpoint

Files:

Modify: backend/src/media_library_viewer_api/routers/dashboard.py
Test: backend/tests/test_backups.py
Step 1: Write the failing test

def test_dashboard_backups():
    from fastapi.testclient import TestClient
    from media_library_viewer_api.main import app
    
    client = TestClient(app)
    response = client.get("/api/dashboard/backups")
    assert response.status_code == 200
    data = response.json()
    assert "total_jobs" in data
    assert "success_rate_24h" in data

Step 2: Run test to verify it fails

Run: cd backend && pytest tests/test_backups.py::test_dashboard_backups -v Expected: FAIL — endpoint not found

Step 3: Implement dashboard endpoint

Modify backend/src/media_library_viewer_api/routers/dashboard.py:

# Add import
from ..models.backups import BackupDashboardSummary

# Add endpoint:
@router.get("/backups")
def get_backup_dashboard(
    store: SettingsStore = Depends(get_settings_store),
) -> BackupDashboardSummary:
    jobs = store.list_backup_jobs()
    total_jobs = len(jobs)
    
    # Calculate 24h success rate
    cutoff = int(time.time()) - (24 * 60 * 60)
    recent_runs = []
    for job in jobs:
        runs = store.list_backup_runs(job_id=job["id"], limit=1)
        if runs and runs[0]["started_at"] >= cutoff:
            recent_runs.append(runs[0])
    
    successful = sum(1 for r in recent_runs if r["status"] == "success")
    success_rate = (successful / len(recent_runs) * 100) if recent_runs else 100.0
    
    # Active alerts
    alerts = store.list_backup_alerts(acknowledged=False)
    active_alerts = len(alerts)
    
    # Last failed
    failed_runs = []
    for job in jobs:
        runs = store.list_backup_runs(job_id=job["id"], status="failure", limit=1)
        if runs:
            failed_runs.append(runs[0])
    
    last_failed_at = None
    if failed_runs:
        last_failed_at = max(r["started_at"] for r in failed_runs)
    
    return BackupDashboardSummary(
        total_jobs=total_jobs,
        success_rate_24h=round(success_rate, 1),
        active_alerts=active_alerts,
        last_failed_at=last_failed_at,
    )

Step 4: Run test to verify it passes

Run: cd backend && pytest tests/test_backups.py::test_dashboard_backups -v Expected: PASS

Step 5: Commit

git add backend/src/media_library_viewer_api/routers/dashboard.py backend/tests/test_backups.py
git commit -m "feat: add backup dashboard summary endpoint"

Task 9: Frontend Types

Files:

Create: frontend/src/types/backups.ts
Step 1: Write the type definitions

Create frontend/src/types/backups.ts:

export interface BackupJob {
    id: string;
    name: string;
    source: string | null;
    target: string | null;
    schedule_interval_seconds: number | null;
    created_at: number;
}

export interface BackupRun {
    id: string;
    job_id: string;
    started_at: number;
    ended_at: number | null;
    status: "success" | "failure" | "in_progress";
    bytes_transferred: number | null;
    duration_ms: number | null;
    error_message: string | null;
    details_json: Record<string, unknown> | null;
    created_at: number;
}

export interface BackupAlert {
    id: string;
    job_id: string;
    run_id: string | null;
    alert_type: "missed_schedule" | "failed_status" | "anomaly_size" | "anomaly_duration";
    severity: "warning" | "critical";
    message: string;
    acknowledged: boolean;
    resolved_at: number | null;
    created_at: number;
}

export interface BackupDashboardSummary {
    total_jobs: number;
    success_rate_24h: number;
    active_alerts: number;
    last_failed_at: number | null;
}

Step 2: Commit

git add frontend/src/types/backups.ts
git commit -m "feat: add backup monitoring TypeScript types"

Task 10: Frontend API Client

Files:

Create: frontend/src/api/backups.ts
Step 1: Write the API client

Create frontend/src/api/backups.ts:

import {
    BackupAlert,
    BackupDashboardSummary,
    BackupJob,
    BackupRun,
} from "../types/backups";

const API_BASE = "/api";

export async function fetchBackupJobs(): Promise<BackupJob[]> {
    const res = await fetch(`${API_BASE}/backups/jobs`);
    if (!res.ok) throw new Error("Failed to fetch backup jobs");
    return res.json();
}

export async function fetchBackupJob(jobId: string): Promise<{ job: BackupJob; runs: BackupRun[] }> {
    const res = await fetch(`${API_BASE}/backups/jobs/${jobId}`);
    if (!res.ok) throw new Error("Failed to fetch backup job");
    return res.json();
}

export async function fetchBackupRuns(jobId?: string, status?: string): Promise<BackupRun[]> {
    const params = new URLSearchParams();
    if (jobId) params.append("job_id", jobId);
    if (status) params.append("status", status);
    const res = await fetch(`${API_BASE}/backups/runs?${params}`);
    if (!res.ok) throw new Error("Failed to fetch backup runs");
    return res.json();
}

export async function fetchBackupRun(runId: string): Promise<BackupRun> {
    const res = await fetch(`${API_BASE}/backups/runs/${runId}`);
    if (!res.ok) throw new Error("Failed to fetch backup run");
    return res.json();
}

export async function fetchBackupAlerts(
    jobId?: string,
    acknowledged?: boolean,
    severity?: string
): Promise<BackupAlert[]> {
    const params = new URLSearchParams();
    if (jobId) params.append("job_id", jobId);
    if (acknowledged !== undefined) params.append("acknowledged", String(acknowledged));
    if (severity) params.append("severity", severity);
    const res = await fetch(`${API_BASE}/backups/alerts?${params}`);
    if (!res.ok) throw new Error("Failed to fetch backup alerts");
    return res.json();
}

export async function acknowledgeBackupAlert(alertId: string): Promise<BackupAlert> {
    const res = await fetch(`${API_BASE}/backups/alerts/${alertId}/acknowledge`, {
        method: "POST",
    });
    if (!res.ok) throw new Error("Failed to acknowledge alert");
    return res.json();
}

export async function fetchBackupDashboard(): Promise<BackupDashboardSummary> {
    const res = await fetch(`${API_BASE}/dashboard/backups`);
    if (!res.ok) throw new Error("Failed to fetch backup dashboard");
    return res.json();
}

Step 2: Commit

git add frontend/src/api/backups.ts
git commit -m "feat: add backup monitoring API client"

Task 11: Frontend Hooks

Files:

Create: frontend/src/hooks/useBackups.ts
Step 1: Write the hooks

Create frontend/src/hooks/useBackups.ts:

import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import {
    acknowledgeBackupAlert,
    fetchBackupAlerts,
    fetchBackupDashboard,
    fetchBackupJob,
    fetchBackupJobs,
    fetchBackupRuns,
} from "../api/backups";

export function useBackupJobs() {
    return useQuery({
        queryKey: ["backups", "jobs"],
        queryFn: fetchBackupJobs,
        refetchInterval: 30_000,
    });
}

export function useBackupJob(jobId: string) {
    return useQuery({
        queryKey: ["backups", "jobs", jobId],
        queryFn: () => fetchBackupJob(jobId),
        enabled: !!jobId,
    });
}

export function useBackupRuns(jobId?: string, status?: string) {
    return useQuery({
        queryKey: ["backups", "runs", jobId, status],
        queryFn: () => fetchBackupRuns(jobId, status),
        refetchInterval: 15_000,
    });
}

export function useBackupAlerts(jobId?: string, acknowledged?: boolean, severity?: string) {
    return useQuery({
        queryKey: ["backups", "alerts", jobId, acknowledged, severity],
        queryFn: () => fetchBackupAlerts(jobId, acknowledged, severity),
        refetchInterval: 30_000,
    });
}

export function useAcknowledgeAlert() {
    const queryClient = useQueryClient();
    return useMutation({
        mutationFn: acknowledgeBackupAlert,
        onSuccess: () => {
            queryClient.invalidateQueries({ queryKey: ["backups", "alerts"] });
        },
    });
}

export function useBackupDashboard() {
    return useQuery({
        queryKey: ["dashboard", "backups"],
        queryFn: fetchBackupDashboard,
        refetchInterval: 30_000,
    });
}

Step 2: Commit

git add frontend/src/hooks/useBackups.ts
git commit -m "feat: add backup monitoring TanStack Query hooks"

Task 12: Backup Jobs Table Component

Files:

Create: frontend/src/components/BackupJobsTable.tsx
Step 1: Write the component

Create frontend/src/components/BackupJobsTable.tsx:

import {
    Chip,
    Paper,
    Table,
    TableBody,
    TableCell,
    TableContainer,
    TableHead,
    TableRow,
} from "@mui/material";
import { BackupJob, BackupRun } from "../types/backups";

interface Props {
    jobs: BackupJob[];
    latestRuns: Map<string, BackupRun>;
}

function formatInterval(seconds: number | null): string {
    if (!seconds) return "N/A";
    if (seconds < 60) return `${seconds}s`;
    if (seconds < 3600) return `${Math.floor(seconds / 60)}m`;
    if (seconds < 86400) return `${Math.floor(seconds / 3600)}h`;
    return `${Math.floor(seconds / 86400)}d`;
}

function formatTimestamp(ts: number | null): string {
    if (!ts) return "Never";
    return new Date(ts * 1000).toLocaleString();
}

export default function BackupJobsTable({ jobs, latestRuns }: Props) {
    return (
        <TableContainer component={Paper}>
            <Table>
                <TableHead>
                    <TableRow>
                        <TableCell>Name</TableCell>
                        <TableCell>Source</TableCell>
                        <TableCell>Target</TableCell>
                        <TableCell>Schedule</TableCell>
                        <TableCell>Last Status</TableCell>
                        <TableCell>Last Run</TableCell>
                        <TableCell>Next Expected</TableCell>
                    </TableRow>
                </TableHead>
                <TableBody>
                    {jobs.map((job) => {
                        const run = latestRuns.get(job.id);
                        const status = run?.status ?? "unknown";
                        const nextExpected = run && job.schedule_interval_seconds
                            ? run.started_at + job.schedule_interval_seconds
                            : null;
                        
                        return (
                            <TableRow key={job.id} hover>
                                <TableCell>{job.name}</TableCell>
                                <TableCell>{job.source ?? "—"}</TableCell>
                                <TableCell>{job.target ?? "—"}</TableCell>
                                <TableCell>{formatInterval(job.schedule_interval_seconds)}</TableCell>
                                <TableCell>
                                    <Chip
                                        label={status}
                                        color={
                                            status === "success"
                                                ? "success"
                                                : status === "failure"
                                                ? "error"
                                                : status === "in_progress"
                                                ? "warning"
                                                : "default"
                                        }
                                        size="small"
                                    />
                                </TableCell>
                                <TableCell>{formatTimestamp(run?.started_at ?? null)}</TableCell>
                                <TableCell>{formatTimestamp(nextExpected)}</TableCell>
                            </TableRow>
                        );
                    })}
                </TableBody>
            </Table>
        </TableContainer>
    );
}

Step 2: Commit

git add frontend/src/components/BackupJobsTable.tsx
git commit -m "feat: add backup jobs table component"

Task 13: Backup Runs Table Component

Files:

Create: frontend/src/components/BackupRunsTable.tsx
Step 1: Write the component

Create frontend/src/components/BackupRunsTable.tsx:

import {
    Chip,
    FormControl,
    InputLabel,
    MenuItem,
    Paper,
    Select,
    Table,
    TableBody,
    TableCell,
    TableContainer,
    TableHead,
    TableRow,
} from "@mui/material";
import { useState } from "react";
import { BackupRun } from "../types/backups";

interface Props {
    runs: BackupRun[];
}

function formatBytes(bytes: number | null): string {
    if (bytes === null || bytes === undefined) return "—";
    if (bytes < 1024) return `${bytes} B`;
    if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
    if (bytes < 1024 * 1024 * 1024) return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
    return `${(bytes / (1024 * 1024 * 1024)).toFixed(1)} GB`;
}

function formatDuration(ms: number | null): string {
    if (ms === null || ms === undefined) return "—";
    if (ms < 1000) return `${ms}ms`;
    if (ms < 60_000) return `${(ms / 1000).toFixed(1)}s`;
    if (ms < 3600_000) return `${(ms / 60_000).toFixed(1)}m`;
    return `${(ms / 3600_000).toFixed(1)}h`;
}

function formatTimestamp(ts: number): string {
    return new Date(ts * 1000).toLocaleString();
}

export default function BackupRunsTable({ runs }: Props) {
    const [statusFilter, setStatusFilter] = useState<string>("all");
    
    const filteredRuns = statusFilter === "all"
        ? runs
        : runs.filter((r) => r.status === statusFilter);
    
    return (
        <>
            <FormControl sx={{ minWidth: 120, mb: 2 }}>
                <InputLabel>Status</InputLabel>
                <Select
                    value={statusFilter}
                    label="Status"
                    onChange={(e) => setStatusFilter(e.target.value)}
                >
                    <MenuItem value="all">All</MenuItem>
                    <MenuItem value="success">Success</MenuItem>
                    <MenuItem value="failure">Failure</MenuItem>
                    <MenuItem value="in_progress">In Progress</MenuItem>
                </Select>
            </FormControl>
            
            <TableContainer component={Paper}>
                <Table>
                    <TableHead>
                        <TableRow>
                            <TableCell>Job</TableCell>
                            <TableCell>Status</TableCell>
                            <TableCell>Duration</TableCell>
                            <TableCell>Size</TableCell>
                            <TableCell>Started</TableCell>
                        </TableRow>
                    </TableHead>
                    <TableBody>
                        {filteredRuns.map((run) => (
                            <TableRow key={run.id} hover>
                                <TableCell>{run.job_id}</TableCell>
                                <TableCell>
                                    <Chip
                                        label={run.status}
                                        color={
                                            run.status === "success"
                                                ? "success"
                                                : run.status === "failure"
                                                ? "error"
                                                : "warning"
                                        }
                                        size="small"
                                    />
                                </TableCell>
                                <TableCell>{formatDuration(run.duration_ms)}</TableCell>
                                <TableCell>{formatBytes(run.bytes_transferred)}</TableCell>
                                <TableCell>{formatTimestamp(run.started_at)}</TableCell>
                            </TableRow>
                        ))}
                    </TableBody>
                </Table>
            </TableContainer>
        </>
    );
}

Step 2: Commit

git add frontend/src/components/BackupRunsTable.tsx
git commit -m "feat: add backup runs table component"

Task 14: Backup Alerts Table Component

Files:

Create: frontend/src/components/BackupAlertsTable.tsx
Step 1: Write the component

Create frontend/src/components/BackupAlertsTable.tsx:

import { Button, Chip, Paper, Table, TableBody, TableCell, TableContainer, TableHead, TableRow } from "@mui/material";
import { BackupAlert } from "../types/backups";

interface Props {
    alerts: BackupAlert[];
    onAcknowledge: (alertId: string) => void;
}

function formatTimestamp(ts: number): string {
    return new Date(ts * 1000).toLocaleString();
}

export default function BackupAlertsTable({ alerts, onAcknowledge }: Props) {
    return (
        <TableContainer component={Paper}>
            <Table>
                <TableHead>
                    <TableRow>
                        <TableCell>Severity</TableCell>
                        <TableCell>Type</TableCell>
                        <TableCell>Message</TableCell>
                        <TableCell>Created</TableCell>
                        <TableCell>Actions</TableCell>
                    </TableRow>
                </TableHead>
                <TableBody>
                    {alerts.map((alert) => (
                        <TableRow key={alert.id} hover>
                            <TableCell>
                                <Chip
                                    label={alert.severity}
                                    color={alert.severity === "critical" ? "error" : "warning"}
                                    size="small"
                                />
                            </TableCell>
                            <TableCell>{alert.alert_type}</TableCell>
                            <TableCell>{alert.message}</TableCell>
                            <TableCell>{formatTimestamp(alert.created_at)}</TableCell>
                            <TableCell>
                                {!alert.acknowledged && (
                                    <Button
                                        size="small"
                                        variant="outlined"
                                        onClick={() => onAcknowledge(alert.id)}
                                    >
                                        Acknowledge
                                    </Button>
                                )}
                            </TableCell>
                        </TableRow>
                    ))}
                </TableBody>
            </Table>
        </TableContainer>
    );
}

Step 2: Commit

git add frontend/src/components/BackupAlertsTable.tsx
git commit -m "feat: add backup alerts table component"

Files:

Create: frontend/src/components/BackupDashboardWidget.tsx
Step 1: Write the component

Create frontend/src/components/BackupDashboardWidget.tsx:

import { Card, CardContent, Typography, Box, Chip } from "@mui/material";
import { useBackupDashboard } from "../hooks/useBackups";

export default function BackupDashboardWidget() {
    const { data, isLoading } = useBackupDashboard();
    
    if (isLoading || !data) {
        return (
            <Card>
                <CardContent>
                    <Typography variant="h6">Backups</Typography>
                    <Typography color="text.secondary">Loading...</Typography>
                </CardContent>
            </Card>
        );
    }
    
    return (
        <Card>
            <CardContent>
                <Typography variant="h6" gutterBottom>Backups</Typography>
                <Box sx={{ display: "flex", gap: 2, flexWrap: "wrap" }}>
                    <Box>
                        <Typography variant="h4">{data.total_jobs}</Typography>
                        <Typography variant="body2" color="text.secondary">Jobs</Typography>
                    </Box>
                    <Box>
                        <Typography variant="h4">{data.success_rate_24h}%</Typography>
                        <Typography variant="body2" color="text.secondary">24h Success</Typography>
                    </Box>
                    <Box>
                        <Typography variant="h4">
                            {data.active_alerts > 0 ? (
                                <Chip label={data.active_alerts} color="error" size="small" />
                            ) : (
                                0
                            )}
                        </Typography>
                        <Typography variant="body2" color="text.secondary">Alerts</Typography>
                    </Box>
                    {data.last_failed_at && (
                        <Box>
                            <Typography variant="body2" color="error">
                                Last failed: {new Date(data.last_failed_at * 1000).toLocaleString()}
                            </Typography>
                        </Box>
                    )}
                </Box>
            </CardContent>
        </Card>
    );
}

Step 2: Commit

git add frontend/src/components/BackupDashboardWidget.tsx
git commit -m "feat: add backup dashboard widget"

Task 16: Main Backups Page

Files:

Create: frontend/src/components/BackupsPage.tsx
Step 1: Write the component

Create frontend/src/components/BackupsPage.tsx:

import { Box, Tab, Tabs, Typography } from "@mui/material";
import { useState } from "react";
import {
    useAcknowledgeAlert,
    useBackupAlerts,
    useBackupJobs,
    useBackupRuns,
} from "../hooks/useBackups";
import BackupAlertsTable from "./BackupAlertsTable";
import BackupJobsTable from "./BackupJobsTable";
import BackupRunsTable from "./BackupRunsTable";

export default function BackupsPage() {
    const [tab, setTab] = useState(0);
    const { data: jobsData, isLoading: jobsLoading } = useBackupJobs();
    const { data: runsData, isLoading: runsLoading } = useBackupRuns();
    const { data: alertsData, isLoading: alertsLoading } = useBackupAlerts(undefined, false);
    const acknowledgeMutation = useAcknowledgeAlert();
    
    // Build a map of latest runs per job
    const latestRuns = new Map();
    if (runsData) {
        for (const run of runsData) {
            const existing = latestRuns.get(run.job_id);
            if (!existing || run.started_at > existing.started_at) {
                latestRuns.set(run.job_id, run);
            }
        }
    }
    
    return (
        <Box sx={{ p: 3 }}>
            <Typography variant="h4" gutterBottom>Backups</Typography>
            
            <Tabs value={tab} onChange={(_, v) => setTab(v)} sx={{ mb: 2 }}>
                <Tab label="Jobs" />
                <Tab label="Runs" />
                <Tab label={`Alerts ${alertsData ? `(${alertsData.length})` : ""}`} />
            </Tabs>
            
            {tab === 0 && (
                jobsLoading ? (
                    <Typography>Loading jobs...</Typography>
                ) : (
                    <BackupJobsTable jobs={jobsData ?? []} latestRuns={latestRuns} />
                )
            )}
            
            {tab === 1 && (
                runsLoading ? (
                    <Typography>Loading runs...</Typography>
                ) : (
                    <BackupRunsTable runs={runsData ?? []} />
                )
            )}
            
            {tab === 2 && (
                alertsLoading ? (
                    <Typography>Loading alerts...</Typography>
                ) : (
                    <BackupAlertsTable
                        alerts={alertsData ?? []}
                        onAcknowledge={(id) => acknowledgeMutation.mutate(id)}
                    />
                )
            )}
        </Box>
    );
}

Step 2: Commit

git add frontend/src/components/BackupsPage.tsx
git commit -m "feat: add main backups page with tabs"

Files:

Modify: frontend/src/App.tsx
Step 1: Add route and nav

In frontend/src/App.tsx:

// Add import
import BackupsPage from "./components/BackupsPage";

// Add route inside <Routes>:
<Route path="/backups" element={<BackupsPage />} />

// Add tab inside <Tabs>:
<Tab value="/backups" label="Backups" component={NavLink} to="/backups" />

Step 2: Commit

git add frontend/src/App.tsx
git commit -m "feat: add backups route and navigation"

Files:

Modify: frontend/src/components/Dashboard.tsx (or equivalent)
Step 1: Add widget to dashboard

Find the dashboard component and add the backup widget:

// Add import
import BackupDashboardWidget from "./BackupDashboardWidget";

// Add to dashboard layout (where other widgets are):
<BackupDashboardWidget />

Step 2: Commit

git add frontend/src/components/Dashboard.tsx
git commit -m "feat: add backup widget to dashboard"

Task 19: Update REQUIREMENTS.md

Files:

Modify: docs/REQUIREMENTS.md
Step 1: Document backup monitoring

Add a new section to docs/REQUIREMENTS.md:

## Backup Monitoring

### Overview
The system receives backup execution reports from an external backup tool via HTTP API, stores job and run history, and provides alerting on failures, missed schedules, and anomalies.

### API
- `POST /api/backups/report` — Submit backup run (Bearer token auth)
- `POST /api/backups/report/start` — Mark backup as in_progress
- `GET /api/backups/jobs` — List jobs
- `GET /api/backups/runs` — List runs
- `GET /api/backups/alerts` — List alerts
- `POST /api/backups/alerts/{id}/acknowledge` — Acknowledge alert
- `GET /api/dashboard/backups` — Dashboard summary

### Data Model
- **BackupJob**: id, name, source, target, schedule_interval_seconds, created_at
- **BackupRun**: id, job_id, started_at, ended_at, status, bytes_transferred, duration_ms, error_message, details_json
- **BackupAlert**: id, job_id, run_id, alert_type, severity, message, acknowledged, resolved_at

### Alert Types
- `failed_status` — Backup reported failure (critical)
- `missed_schedule` — No run within 1.5x expected interval (warning)
- `anomaly_size` — Size is 0 or <10% / >300% of 7-day median (warning)
- `anomaly_duration` — Duration >300% of 7-day median (warning)

### Authentication
- Backup tool uses auto-generated Bearer API key
- Frontend uses existing OIDC/JWT auth

Step 2: Commit

git add docs/REQUIREMENTS.md
git commit -m "docs: add backup monitoring requirements"

Task 20: Run Full Test Suite

Step 1: Run backend tests

cd backend && pytest tests/test_backups.py -v

Expected: All tests pass

Step 2: Run frontend build

cd frontend && npm run build

Expected: Build succeeds with no errors

Step 3: Run backend lint

cd backend && ruff check src/media_library_viewer_api/

Expected: No lint errors

Step 4: Commit

git commit -m "test: verify backup monitoring implementation"

Spec Coverage Check

Spec Section	Implementing Task
Data Model (BackupJob, BackupRun, BackupAlert)	Task 1, 3
API Endpoints (report, start, jobs, runs, alerts)	Task 7
Dashboard Endpoint	Task 8
Validation Rules	Task 7 (endpoint validation)
Alert Types (failed, missed, anomaly)	Task 4, 5
Alert State Machine	Task 4, 7
Frontend Navigation	Task 17
Dashboard Widget	Task 15, 18
Backups Page (Jobs, Runs, Alerts tabs)	Task 12-16
API Key Auth	Task 6
Background Poller	Task 5
Charts	Out of scope for MVP (future enhancement)

Placeholder Scan

No "TBD", "TODO", "implement later" found
No vague "add error handling" steps
All code blocks contain actual code
No "similar to Task N" references

Type Consistency Check

BackupJob.schedule_interval_seconds — used consistently across backend and frontend
BackupRun.status — enum values match (success, failure, in_progress)
BackupAlert.alert_type — values match spec
BackupAlert.severity — values match spec (warning, critical)
All Pydantic models match TypeScript interfaces

Execution Handoff

Plan complete and saved to docs/superpowers/plans/2026-05-11-backup-monitoring.md. Two execution options:

1. Subagent-Driven (recommended) - I dispatch a fresh subagent per task, review between tasks, fast iteration

2. Inline Execution - Execute tasks in this session using executing-plans, batch execution with checkpoints

Which approach?

67 KiB

Raw Permalink Blame History

Backup Monitoring Implementation Plan

File Structure

Backend

Frontend

Task 1: Database Schema

Task 2: Pydantic Models

Task 3: Settings Store CRUD Methods

Task 4: Alert Engine

Task 5: Background Poller

Task 6: API Key Authentication

Task 7: Backup Router (Backend API)

Task 8: Dashboard Endpoint

Task 9: Frontend Types

Task 10: Frontend API Client

Task 11: Frontend Hooks

Task 12: Backup Jobs Table Component

Task 13: Backup Runs Table Component

Task 14: Backup Alerts Table Component

Task 15: Backup Dashboard Widget

Task 16: Main Backups Page

Task 17: Wire Up Routes and Navigation

Task 18: Wire Up Dashboard Widget

Task 19: Update REQUIREMENTS.md

Task 20: Run Full Test Suite

Spec Coverage Check

Placeholder Scan

Type Consistency Check

Execution Handoff

67 KiB Raw Permalink Blame History

Backup Monitoring Implementation Plan

File Structure

Backend

Frontend

Task 1: Database Schema

Task 2: Pydantic Models

Task 3: Settings Store CRUD Methods

Task 4: Alert Engine

Task 5: Background Poller

Task 6: API Key Authentication

Task 7: Backup Router (Backend API)

Task 8: Dashboard Endpoint

Task 9: Frontend Types

Task 10: Frontend API Client

Task 11: Frontend Hooks

Task 12: Backup Jobs Table Component

Task 13: Backup Runs Table Component

Task 14: Backup Alerts Table Component

Task 15: Backup Dashboard Widget

Task 16: Main Backups Page

Task 17: Wire Up Routes and Navigation

Task 18: Wire Up Dashboard Widget

Task 19: Update REQUIREMENTS.md

Task 20: Run Full Test Suite

Spec Coverage Check

Placeholder Scan

Type Consistency Check

Execution Handoff

67 KiB

Raw Permalink Blame History