alex/manage

Fork 0

Files

T

alex 83886ea7a4 docs: fix schedule field and add resolved_at to backup monitoring spec

2026-05-11 20:04:24 +02:00

12 KiB

Raw Permalink Blame History

Backup Monitoring Design

Date: 2026-05-11 Status: Draft

Overview

This document specifies a standalone backup monitoring module for the media library viewer. The system receives backup run reports from an external backup tool via HTTP, stores them in a structured yet extensible format, and provides both validation and automated alerting capabilities.

Goals

Receive backup statistics from an external tool via HTTP API
Store backup job definitions and execution history
Validate incoming data and generate alerts on anomalies
Provide a dedicated frontend UI for backup monitoring
Remain completely independent of the existing machine monitoring system

Non-Goals

Integrate with the existing monitoring machine infrastructure
Run backup jobs or manage backup schedules
Replace external backup tools

Architecture

The backup monitoring system is a standalone module within the FastAPI backend with a dedicated React frontend page.

┌─────────────────┐     HTTP POST      ┌──────────────────┐
│  Backup Tool    │ ─────────────────> │  POST /api/      │
│  (external)     │  Bearer token      │  backups/report  │
└─────────────────┘                    └──────────────────┘
                                               │
                                               ▼
┌─────────────────┐     Query          ┌──────────────────┐
│  React Frontend │ <───────────────── │  GET /api/       │
│  /backups       │  OIDC/JWT          │  backups/*       │
└─────────────────┘                    └──────────────────┘
                                               │
                                               ▼
                                       ┌──────────────────┐
                                       │  SQLite Database │
                                       │  backup_jobs     │
                                       │  backup_runs     │
                                       │  backup_alerts   │
                                       └──────────────────┘

Data Model

`BackupJob`

Represents a backup job definition. Auto-created when the first report arrives.

Field	Type	Description
`id`	UUID	Primary key
`name`	string	Unique job name (e.g., "photos-daily")
`source`	string	Source description (free text, e.g., "server-a:/data/photos")
`target`	string	Target description (free text, e.g., "server-b:/backups/photos")
`schedule_interval_seconds`	integer	Expected interval between runs in seconds (e.g., 86400 for daily)
`created_at`	datetime	When the job was first seen

`BackupRun`

Represents a single backup execution.

Field	Type	Description
`id`	UUID	Primary key
`job_id`	UUID	Foreign key to BackupJob
`started_at`	datetime	When the backup started
`ended_at`	datetime	When the backup ended (nullable)
`status`	enum	`success`, `failure`, `in_progress`
`bytes_transferred`	integer	Bytes transferred (nullable, >= 0)
`duration_ms`	integer	Duration in milliseconds (nullable)
`error_message`	string	Error details if status is failure (nullable)
`details_json`	JSON	Extensibility buffer for arbitrary fields
`created_at`	datetime	When the report was received

`BackupAlert`

Represents an alert condition detected for a job.

Field	Type	Description
`id`	UUID	Primary key
`job_id`	UUID	Foreign key to BackupJob
`run_id`	UUID	Related run, if applicable (nullable)
`alert_type`	enum	`missed_schedule`, `failed_status`, `anomaly_size`, `anomaly_duration`
`severity`	enum	`warning`, `critical`
`message`	string	Human-readable description
`acknowledged`	boolean	Whether a user has acknowledged this alert
`resolved_at`	datetime	When the alert was auto-resolved (nullable)
`created_at`	datetime	When the alert was generated

API Specification

Backup Tool Endpoints (Bearer Token Auth)

`POST /api/backups/report`

Submit a backup run report. Creates the job if it doesn't exist.

Request Body:

{
  "name": "photos-daily",
  "source": "server-a:/data/photos",
  "target": "server-b:/backups/photos",
  "schedule_interval_seconds": 86400,
  "started_at": "2026-05-11T02:00:00Z",
  "ended_at": "2026-05-11T02:15:30Z",
  "status": "success",
  "bytes_transferred": 10737418240,
  "duration_ms": 930000,
  "error_message": null,
  "details": {
    "compression_ratio": 0.75,
    "files_count": 15420
  }
}

Validation Rules:

name: required, non-empty string
started_at: required, ISO 8601 timestamp, must be in the past
status: required, one of success, failure, in_progress
schedule_interval_seconds: optional, if provided must be > 0. Updates the job's schedule if the job exists or is being created.
If status == "success": ended_at and duration_ms are required
bytes_transferred: if provided, must be >= 0
Unknown fields in details are stored in details_json

Response: 200 OK with the created run object.

`POST /api/backups/report/start`

Mark a backup job as in_progress. Useful for long-running backups.

Request Body:

{
  "name": "photos-daily",
  "source": "server-a:/data/photos",
  "schedule_interval_seconds": 86400,
  "started_at": "2026-05-11T02:00:00Z"
}

Response: 200 OK with the created run object (status: in_progress).

Frontend Endpoints (OIDC/JWT Auth)

Endpoint	Method	Description
`GET /api/backups/jobs`	GET	List all backup jobs with last run info
`GET /api/backups/jobs/{id}`	GET	Get job details with recent runs
`GET /api/backups/runs`	GET	List recent runs (filterable by job_id, status, date range)
`GET /api/backups/runs/{id}`	GET	Get run details
`GET /api/backups/alerts`	GET	List alerts (filterable by severity, acknowledged)
`POST /api/backups/alerts/{id}/acknowledge`	POST	Acknowledge an alert
`GET /api/dashboard/backups`	GET	Dashboard summary (job count, 24h success rate, active alerts)

Validation & Alerting

Alert Generation

Alerts are generated in two ways:

On report ingestion: When a new run report is received
By background poller: Periodically checks for missed schedules

Alert Types

Alert Type	Trigger	Severity	Auto-Resolve
`failed_status`	`status == "failure"`	`critical`	On next successful run
`missed_schedule`	No run within `schedule_interval_seconds * 1.5`	`warning`	On next run arrival
`anomaly_size`	`bytes_transferred == 0` or < 10% or > 300% of 7-day median	`warning`	On next normal run
`anomaly_duration`	`duration_ms` > 300% of 7-day median	`warning`	On next normal run

Alert State Machine

Alerts are created once per condition (no duplicates for the same condition)
failed_status alerts reference the specific failed run
missed_schedule alerts reference the job (no run_id)
When a new run resolves the condition, the alert is auto-resolved (add resolved_at timestamp)
Users can manually acknowledge alerts to prevent repeated notifications
Acknowledged alerts remain visible but are filtered out of active alert lists

Frontend Design

Add a "Backups" item to the main navigation bar, linking to /backups.

Dashboard Integration

A small "Backups" widget on the dashboard (/) showing:

Total number of backup jobs
24-hour success rate
Count of active (unacknowledged) alerts
Time of last failed backup (if any)

Backups Page (`/backups`)

Tabbed layout with three tabs:

Tab 1: Jobs

Sortable table with columns:

Job name
Source
Target
Schedule interval (human-readable, e.g., "Every 24 hours")
Last run status (color-coded chip)
Last run time
Next expected run time (calculated from last_run_started_at + schedule_interval_seconds)

Clicking a job row navigates to a job detail view showing historical runs.

Tab 2: Runs

Filterable table with columns:

Job name
Status (success/failure/in_progress)
Duration
Bytes transferred (human-readable, e.g., "10.7 GB")
Timestamp
Actions: view details

Filters: job name, status, date range.

Tab 3: Alerts

Table of active alerts with:

Severity badge (warning/critical)
Job name
Alert type
Message
Created time
Actions: acknowledge, view related run

Charts

Reusing existing D3 chart infrastructure:

Success/failure rate over time (line chart, stacked area)
Bytes transferred per job over time (grouped bar or multi-line chart)
Duration trends per job (line chart)
Time range selection via brush (same pattern as monitoring charts)

Authentication

Backup Tool Authentication

A single API key stored in application settings
Backup tool sends Authorization: Bearer <api_key> header
API key is auto-generated on first startup if not configured
Key can be rotated via settings

Frontend Authentication

Uses existing OIDC/JWT authentication
No additional authorization logic needed (all authenticated users can view backups)

Error Handling

Invalid report data returns 422 Unprocessable Entity with detailed validation errors
Duplicate reports: if a run with the same job_id and started_at (within 1-second precision) already exists, the report updates the existing run instead of creating a new one
Malformed JSON in details field is accepted but stored as string in details_json
Database errors are logged and return 500 Internal Server Error

Implementation Notes

Database

Use the existing SQLite database with WAL mode
Create three new tables: backup_jobs, backup_runs, backup_alerts
Add indexes on frequently queried columns: backup_runs.job_id, backup_runs.status, backup_alerts.job_id, backup_alerts.acknowledged

Background Poller

Extend the existing MonitoringPoller or create a new BackupAlertPoller
Runs every 5 minutes (configurable)
Checks for missed schedules and generates alerts
Prunes old alerts (configurable retention, default 90 days)

Extensibility

The details_json field allows the backup tool to include arbitrary metadata without schema changes. Future enhancements might include:

Custom alert thresholds per job
Alert notification channels (email, webhook)
Backup verification (test restores)
Cross-job dependency tracking

Testing Strategy

Unit tests: Validation logic, alert generation rules, median calculation
API tests: Report submission, job creation, alert acknowledgment
Integration tests: End-to-end flow from report submission to alert display
Frontend tests: Component rendering, filtering, chart interactions

Open Questions

Should we support multiple API keys (one per backup tool instance)?
Should alerts be sent via email or other notification channels?
Should we support backup job configuration via UI (pre-registering jobs)?
What retention policy should apply to backup runs (vs. alerts)?

REQUIREMENTS.md - Overall project requirements
Existing monitoring system documentation (for patterns to follow)

12 KiB Raw Permalink Blame History