headquarter/openspec/changes/archive/2026-05-18-database-setup-recovery/design.md

Context

Currently, the application crashes with cryptic SQLAlchemy errors when:

1. Database tables don't exist (fresh deployment)
2. Migrations haven't been applied
3. The database is temporarily unavailable during startup

This requires manual intervention to run alembic upgrade head and restart containers.

Goals / Non-Goals

Goals:

• Automatically run migrations on application startup
• Handle missing tables gracefully with informative error messages
• Add database connection retries for transient failures
• Ensure seed data runs after migrations complete
• Support both development and production Docker deployments

Non-Goals:

• Database backup/recovery (out of scope)
• Complex migration rollback handling
• Multi-master database support

Decisions

1. Run migrations in startup event

   • Rationale: Ensures database is always up-to-date before handling requests
   • Alternative: Separate init container. Rejected to keep deployment simple.

2. Use alembic programmatic API

   • Rationale: Avoids subprocess calls and captures errors properly
   • Run via alembic.command.upgrade() in async context

3. Add connection retry with backoff

   • Rationale: Database may not be ready when app starts
   • 5 retries with 2-second exponential backoff

4. Graceful error handling

   • Rationale: Clear error messages for operators
   • Distinguish between: connection refused, auth failed, missing migrations

Risks / Trade-offs

• [Startup delay] -> Migrations run on every startup, but Alembic is idempotent
• [Concurrent startup] -> Multiple instances could race; use advisory locks if needed later
• [Migration failures] -> App won't start; this is correct behavior

Migration Plan

1. Add init_database() function that runs migrations and seed data
2. Modify startup event to call init_database() with retries
3. Update Docker CMD to ensure database is ready
4. Test with fresh database volume

Open Questions

• Should we add a separate db init CLI command for manual runs?
• Do we need database connection pooling configuration?