feat(v2): complete v2 reimplementation
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
# Disaster recovery runbook
|
||||
|
||||
1. Isolate the failed host and preserve the metadata volume, repository roots, logs,
|
||||
image digest, and master-key backup. Do not restart writers repeatedly.
|
||||
2. Provision a clean host with the same pinned image and non-root volume permissions.
|
||||
3. Restore the master key securely, restore repository roots read-only first, and
|
||||
restore metadata from a verified backup or the passphrase-protected recovery bundle.
|
||||
4. Run `migrate current`, start only `web`, validate `/readyz` and repository
|
||||
inspection, then start scheduler and worker one at a time.
|
||||
5. Perform and document a test restore before enabling scheduled work. Rotate secrets
|
||||
if host compromise is possible.
|
||||
@@ -0,0 +1,13 @@
|
||||
# Master and repository key runbook
|
||||
|
||||
- Store the Compose master key outside the checkout. It must be a regular file,
|
||||
at least 32 bytes, mode `0600`, owned by the service UID (`10001` for Compose).
|
||||
Loss of this key destroys access to encrypted metadata secrets.
|
||||
- Back up the master key independently from metadata and repositories; do not put it
|
||||
in an image, Compose environment variable, log, ticket, or recovery bundle.
|
||||
- Use the `admin recovery export` command with a passphrase file descriptor to create
|
||||
a separately protected recovery bundle. Validate it on an isolated host.
|
||||
- Rotate repository data keys only with `admin repository-key rotate`; retain prior
|
||||
recovery material until a restore drill succeeds.
|
||||
- If compromise is suspected, stop the worker, preserve evidence, rotate credentials,
|
||||
export a fresh recovery bundle, and run a restore drill before resuming writes.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Metadata backup and restore runbook
|
||||
|
||||
## Backup
|
||||
|
||||
1. Confirm `docker compose ps` shows exactly one scheduler and one worker.
|
||||
2. For an online SQLite backup, run a host-side SQLite `.backup` against the mounted
|
||||
`metadata.db`; do **not** copy only the main file while WAL writers run.
|
||||
3. For a filesystem copy, stop `web`, `scheduler`, and `worker` first, then retain
|
||||
`metadata.db`, `metadata.db-wal`, and `metadata.db-shm` together.
|
||||
4. Encrypt and test the backup outside the appliance. Never place a database dump in
|
||||
the repository or OCI image.
|
||||
|
||||
## Restore
|
||||
|
||||
1. Stop all runtime roles and preserve the failed metadata volume unchanged.
|
||||
2. Restore the complete SQLite backup into the metadata volume with the service user
|
||||
ownership (UID 10001 in the supplied Compose deployment).
|
||||
3. Run `docker compose run --rm migrate current`; only start the stack when it reports
|
||||
the expected revision. Validate `/readyz` and a read-only API request after startup.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Notifications runbook
|
||||
|
||||
## Configure
|
||||
|
||||
Create a filtered email or webhook subscription through `/api/v2/notifications/subscriptions`. Webhooks require a write-only signing secret. Configure SMTP separately at `/api/v2/notifications/email-settings`; only authenticated STARTTLS SMTP is accepted. Verify a channel with `POST /subscriptions/{id}/test` and inspect delivery/attempt history before relying on it.
|
||||
|
||||
Filters are a nonempty set of exact catalog IDs or family wildcards such as `execution.*`; they may be narrowed by job IDs, repository IDs, or severity. The public catalog is live-events-only: every listed type is emitted by a currently available operation. Deferred channels and source capabilities have no catalog entries.
|
||||
|
||||
## Rotate and recover
|
||||
|
||||
Rotate webhook keys using the signing-key rotate endpoint with an idempotency key and an explicit bounded overlap. Receivers must accept both signatures during overlap, then remove the old key after expiry. A recovery bundle deliberately excludes subscriptions, SMTP settings, signing secrets, event history, and delivery attempts. Reconfigure notifications after a fresh-host recovery.
|
||||
|
||||
## Failure handling
|
||||
|
||||
The worker claims due deliveries with a lease. Transient errors enter bounded exponential retry; interrupted leases recover as retryable work and may send an event again. Inspect response class and redacted diagnostics in history. A terminal failed delivery can be retried manually once the destination is corrected. To stop outbound traffic, disable/archive subscriptions or stop the worker; do not delete outbox history. Rollback consists of disabling subscriptions and worker dispatch while retaining audit/outbox records for investigation.
|
||||
@@ -0,0 +1,17 @@
|
||||
# Observability and alert response
|
||||
|
||||
The proxy exposes `/livez`, `/readyz`, and Prometheus text at `/metrics`. Metrics use
|
||||
no source paths, IDs, credentials, tokens, or secret values. Runtime logs are JSON
|
||||
records with an event, timestamp, role, and request correlation ID where applicable.
|
||||
|
||||
Alert when any of the following remains non-zero or grows:
|
||||
|
||||
- `backup_tool_stale_execution_leases`
|
||||
- `backup_tool_failed_executions`
|
||||
- `backup_tool_corrupt_backups`
|
||||
- `backup_tool_unavailable_repositories`
|
||||
- `backup_tool_schedule_lag_seconds`
|
||||
|
||||
Also alert on low `backup_tool_filesystem_free_bytes`. For any alert, preserve logs,
|
||||
validate `/readyz`, stop the worker before destructive repository investigation, and
|
||||
use the matching metadata, repository, key, upgrade, or disaster-recovery runbook.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Recovery bundle export and validation
|
||||
|
||||
M11 recovery exports an offline, passphrase-encrypted catalog and key bundle.
|
||||
Import is a local CLI operation that reconstructs only the metadata required to
|
||||
restore existing encrypted backups; it does not reactivate backup scheduling.
|
||||
|
||||
## Export
|
||||
|
||||
Choose an absolute path in a trusted, non-symlinked directory. The destination
|
||||
must not already exist; export creates it with mode `0600` and never overwrites
|
||||
it.
|
||||
|
||||
```sh
|
||||
read -r -s recovery_passphrase
|
||||
printf '\n'
|
||||
printf '%s\n' "$recovery_passphrase" | \
|
||||
backup-tool admin recovery export \
|
||||
--output /secure/offline/backup-tool-recovery.btrec \
|
||||
--passphrase-fd 0
|
||||
unset recovery_passphrase
|
||||
```
|
||||
|
||||
The passphrase is read from the inherited file descriptor. It is never a CLI
|
||||
argument. Store the resulting `BTREC` file away from the host and away from the
|
||||
live repository-key directories.
|
||||
|
||||
## Validate
|
||||
|
||||
Validation authenticates and decrypts the bundle, checks the versioned Argon2id
|
||||
and AES-GCM format, and verifies the included catalog/key relationships. It
|
||||
prints only a status and repository count.
|
||||
|
||||
```sh
|
||||
read -r -s recovery_passphrase
|
||||
printf '\n'
|
||||
printf '%s\n' "$recovery_passphrase" | \
|
||||
backup-tool admin recovery validate \
|
||||
--input /secure/offline/backup-tool-recovery.btrec \
|
||||
--passphrase-fd 0
|
||||
unset recovery_passphrase
|
||||
```
|
||||
|
||||
Wrong passphrases, tampering, malformed headers, unsupported KDF parameters,
|
||||
and invalid encrypted payloads intentionally produce the same validation error.
|
||||
Do not use a failed validation result to diagnose which of those conditions
|
||||
occurred.
|
||||
|
||||
## Fresh-host import
|
||||
|
||||
Before importing, run migrations on the replacement host and configure its
|
||||
repository allowlist to include the surviving repository directory. The
|
||||
repository must pass normal metadata/path inspection. The replacement metadata
|
||||
database must be current and otherwise empty; import rejects a non-empty
|
||||
destination and any existing/conflicting key files.
|
||||
|
||||
```sh
|
||||
backup-tool migrate upgrade
|
||||
read -r -s recovery_passphrase
|
||||
printf '\n'
|
||||
printf '%s\n' "$recovery_passphrase" | \
|
||||
backup-tool admin recovery import \
|
||||
--input /secure/offline/backup-tool-recovery.btrec \
|
||||
--passphrase-fd 0
|
||||
unset recovery_passphrase
|
||||
```
|
||||
|
||||
Import installs signing and data keys with restrictive modes, restores the
|
||||
repository/source/job/execution/backup catalog needed for restore, and marks
|
||||
sources unavailable plus jobs archived and disabled. Reconfigure sources and
|
||||
explicitly create or enable new jobs before taking another backup.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Repository recovery runbook
|
||||
|
||||
1. Stop `worker` before inspecting or repairing a repository; never edit a live
|
||||
repository behind an active lease.
|
||||
2. Preserve the repository directory and its metadata volume before remediation.
|
||||
3. Verify repository state through the operator API and verify individual backups
|
||||
before any restore. Treat a corrupt verification result as an incident, not a
|
||||
deletion request.
|
||||
4. Mount replacement repository roots at the same allowlisted path, restore metadata,
|
||||
then start `migrate`, `web`, `scheduler`, and finally `worker`.
|
||||
5. Keep archived repositories mounted until retention and restore obligations expire.
|
||||
Do not remove manifests or blobs manually.
|
||||
@@ -0,0 +1,19 @@
|
||||
# SSH sources
|
||||
|
||||
SSH sources require a dedicated account confined by an OpenSSH `ChrootDirectory`
|
||||
and `ForceCommand internal-sftp`. The chroot directory is root-owned; writable
|
||||
content is below it. Disable passwords, keyboard-interactive authentication,
|
||||
shells, PTYs, TCP/X11/agent forwarding, and tunnelling. Configure the source
|
||||
root as `/` only.
|
||||
|
||||
Generate a dedicated unencrypted Ed25519, ECDSA, or RSA-3072+ client key and
|
||||
store it through the write-only `ssh_private_key` secret endpoint. Do not put a
|
||||
key, passphrase, password, command, agent path, or key file path in source
|
||||
configuration. Pin the server's exact OpenSSH public host key (`algorithm
|
||||
base64`) before probing. On host-key rotation, obtain the replacement through
|
||||
an out-of-band administrative channel, update the source pin, then probe.
|
||||
|
||||
The server administrator controls mutable content inside the chroot. The client
|
||||
rejects traversal names, symlinks, special files, changed files, and configured
|
||||
resource-limit overflows, but cannot claim atomic no-follow behavior against a
|
||||
maliciously changing filesystem inside that server-controlled boundary.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Upgrade and rollback runbook
|
||||
|
||||
1. Record the running image digest and take a tested metadata backup plus repository
|
||||
recovery evidence before changing the image.
|
||||
2. Pull/build the pinned image, then run `docker compose run --rm migrate upgrade`.
|
||||
Do not start web, scheduler, or worker against an unverified schema.
|
||||
3. Start the stack, wait for `/readyz`, and inspect `/metrics` for stale leases,
|
||||
schedule lag, unavailable repositories, and corrupt backups.
|
||||
4. If migration fails, stop and restore the prior metadata backup and matching image;
|
||||
do not attempt to downgrade an unknown partially migrated database in place.
|
||||
5. Preserve migration logs and verify a representative backup restore before closing
|
||||
the change.
|
||||
Reference in New Issue
Block a user