docs: add OpenSpec change for backend-frontend refactoring

Recovers and adapts the structural refactoring from overwritten main merge (b6f89f9) to current dev reality. Scope: - Schema extraction into apps/api/src/schemas/ - Docker service split into services/docker/ package - Instance lifecycle extraction from api/tool_instances.py - Config profile service extraction from api/config_profiles.py - Auth dependency refactor (get_current_user) - Frontend reorganization into features/ dirs + kebab-case naming Exclusions (already in dev): seeding, defaults, unique constraint, SSH key mounting, terminal backend, tunnel regex, session auto-numbering.
2026-06-04 09:42:03 +02:00
parent ab55da280c
commit 2757ef3b4f
5 changed files with 419 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-06-03
@@ -0,0 +1,172 @@
+## Context
+
+Current `dev` has all behavioral features from the overwritten `main` merge, but the code structure is pre-refactor:
+- Monolithic `api/tool_instances.py` (~3000 lines)
+- Monolithic `api/config_profiles.py` (~1000 lines)
+- Monolithic `services/docker.py`
+- No `schemas/` directory
+- Flat frontend component structure with inconsistent naming
+
+The `b6f89f9` merge from `main` had a clean refactoring that we need to redo, but adapted to our current reality.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Extract Pydantic schemas from API routers into `src/schemas/`
+- Split `services/docker.py` into `services/docker/` package
+- Extract instance lifecycle logic from `api/tool_instances.py` into `services/instance_lifecycle.py`
+- Extract config profile business logic from `api/config_profiles.py` into `services/config_profiles.py`
+- Add `get_current_user` auth dependency and migrate routers that need the full user object
+- Reorganize frontend components into `features/` directories
+- Standardize frontend API file naming to kebab-case
+- Standardize frontend page naming to `*Page.tsx`
+
+**Non-Goals:**
+- Changing any API request/response shapes
+- Changing any database schemas
+- Adding new features
+- Modifying frontend component behavior or styling
+- Converting `ConfigProfile` mounts from JSON to relation tables (out of scope — would require migration)
+
+## Decisions
+
+### 1. Schema Extraction: One File Per Domain
+**Decision:** Each domain gets its own schema file: `schemas/tool_type.py`, `schemas/tool_instance.py`, etc.
+**Rationale:** Keeps schemas close to their domain. Avoids a giant `schemas.py`.
+
+### 2. Docker Service Split: Functional Boundaries
+**Decision:** Split by responsibility:
+- `compose.py` — compose file generation, modification, port injection, network injection
+- `container.py` — container status, IP lookup, network connect, logs
+- `config_staging.py` — staging config files into instance directories
+- `tunnel.py` — extracting tunnel URLs from cloudflared output
+**Rationale:** Each module has a single reason to change. `docker.py` mixed compose logic with container runtime queries.
+
+### 3. Instance Lifecycle: Service Receives Raw Params, Not Request Objects
+**Decision:** Service functions receive model instances and primitive parameters, not FastAPI request objects.
+**Example:** `create_instance(session, user, project, repo, tool_type, data: CreateInstanceRequest)` → service extracts fields.
+**Rationale:** Keeps service layer independent of HTTP framework. Easier to test.
+
+### 4. Auth Pattern: Gradual Migration, Not Big Bang
+**Decision:** Add `get_current_user` alongside existing `get_current_user_id`. Migrate routers incrementally.
+**Rationale:** Reduces risk. Endpoints that only need the ID can keep the old pattern.
+
+### 5. Frontend Naming: Align with `b6f89f9` Conventions
+**Decision:** Use kebab-case for API files, PascalCase for page files with `Page` suffix, `features/` for component directories.
+**Rationale:** Matches the `b6f89f9` structure that was already reviewed and accepted.
+
+## Module Map
+
+### Backend — Before
+```
+api/
+  tool_instances.py         (~3000 lines) — HTTP + Docker + Git + Lifecycle
+  config_profiles.py        (~1000 lines) — HTTP + Validation + Defaults
+  tool_types.py             (~500 lines)  — HTTP + Schemas
+  health.py                 (~150 lines)  — HTTP + Schemas
+  users.py                  (~100 lines)  — HTTP + Schemas
+  ...
+services/
+  docker.py                 (~600 lines)  — Compose + Container + Tunnel
+```
+
+### Backend — After
+```
+schemas/
+  tool_type.py              (~200 lines)
+  tool_instance.py          (~40 lines)
+  config_profile.py         (~130 lines)
+  health.py                 (~50 lines)
+  user.py                   (~20 lines)
+  ...
+api/
+  tool_instances.py         (~300 lines)  — HTTP routing only
+  config_profiles.py        (~200 lines)  — HTTP routing only
+  tool_types.py             (~250 lines)  — HTTP + validation endpoints
+  health.py                 (~80 lines)   — HTTP only
+  users.py                  (~60 lines)   — HTTP only
+  ...
+services/
+  instance_lifecycle.py     (~420 lines) — Create/Start/Stop/Restart/Delete
+  config_profiles.py        (~300 lines) — CRUD + Defaults + Validation
+  docker/
+    __init__.py             (~40 lines)  — Re-exports
+    compose.py              (~240 lines) — Compose generation
+    container.py            (~120 lines) — Container queries
+    config_staging.py       (~80 lines)  — File staging
+    tunnel.py               (~150 lines) — Tunnel URL extraction
+```
+
+### Frontend — Before
+```
+src/
+  api/
+    tool_types.ts
+    ssh_keys.ts
+    git_repositories.ts
+    sessions.ts
+  components/
+    git-toolbar.tsx
+    file-editor.tsx
+    commit-dialog.tsx
+    ...
+  pages/
+    dashboard.tsx
+    projects.tsx
+    sessions.tsx
+    ...
+```
+
+### Frontend — After
+```
+src/
+  api/
+    tool-types.ts
+    ssh-keys.ts
+    git-repositories.ts
+    sessions.ts
+  components/
+    features/
+      git/
+        GitToolbar.tsx
+        FileBrowser.tsx
+        FileEditor.tsx
+        CommitDialog.tsx
+        MergeDialog.tsx
+        WorkspaceSidebar.tsx
+      dashboard/
+        DashboardSummary.tsx
+        ActiveSessionsList.tsx
+        ProjectsSection.tsx
+        QuickCreateForm.tsx
+        RecentSessionsSection.tsx
+      project/
+        RepositoriesSettingsTab.tsx
+      tool-workshop/
+        ToolTypesTab.tsx
+    ProtectedRoute.tsx
+    AppShell.tsx
+  pages/
+    DashboardPage.tsx
+    ProjectsPage.tsx
+    SessionsPage.tsx
+    ...
+```
+
+## Risks / Trade-offs
+
+**[Risk] Import cycles during extraction** → **Mitigation:** Extract schemas first (no service dependencies), then services, then thin routers last. Use TYPE_CHECKING guards.
+
+**[Risk] Merge conflicts with in-flight features** → **Mitigation:** Coordinate timing. This refactor should be the only large change on `dev` while it's in progress. Freeze other backend work.
+
+**[Risk] Frontend renaming breaks imports** → **Mitigation:** Use `git mv` for renames so git tracks history. Update all imports in a single commit.
+
+**[Risk] Missing re-export in docker/__init__.py breaks consumers** → **Mitigation:** After splitting, run a full import test across all backend files. Add any missing re-exports.
+
+## Migration Plan
+
+1. **Phase 1: Schemas** — Extract all Pydantic models into `schemas/`. Update imports in API routers. No logic changes.
+2. **Phase 2: Services** — Split `docker.py`, extract `instance_lifecycle.py`, extract `config_profiles.py`. Update imports.
+3. **Phase 3: Auth** — Add `get_current_user`, migrate routers that need the full user object.
+4. **Phase 4: Frontend** — Rename files, move components, update imports.
+5. **Phase 5: Verification** — Run full test suite, typecheck, build.
@@ -0,0 +1,52 @@
+## Why
+
+The `main` branch was previously merged into `dev` (commit `b6f89f9`) bringing a large structural refactoring: schema extraction, service splitting, auth dependency pattern changes, and frontend reorganization. This merge was later overwritten when `dev` was reset to a pre-merge clean state (`51a399c`).
+
+We have since forward-ported all behavioral features (built-in tool type seeding, config profile defaults, unique constraints, SSH key mounting, terminal backend, etc.) onto the clean `dev` base. **The codebase now works functionally but lacks the structural cleanliness of the refactoring.**
+
+Monolithic files make the backend harder to navigate, test, and maintain:
+- `api/tool_instances.py` is ~3000 lines (mixing HTTP handling with Docker orchestration)
+- `api/config_profiles.py` is ~1000 lines (mixing HTTP handling with business logic)
+- `services/docker.py` is a monolith of ~600 lines covering compose, container, and tunnel logic
+- Frontend API files use inconsistent naming (`tool_types.ts` vs `tool-types.ts`)
+- Frontend components are flat in `components/` instead of organized by feature domain
+
+## What Changes
+
+Redo the structural refactoring from `b6f89f9`, **adapted to current dev reality**:
+
+1. **Backend schema extraction** — Extract Pydantic request/response schemas from API routers into `apps/api/src/schemas/`
+2. **Backend service splits** — Split `services/docker.py` into `services/docker/` package; extract `services/instance_lifecycle.py` and `services/config_profiles.py`
+3. **Auth dependency refactor** — Change from `get_current_user_id` + manual `_get_user` calls to `get_current_user` dependency returning `User` directly
+4. **Frontend reorganization** — Move components into `features/` directories; rename API files to kebab-case
+
+**No behavioral changes.** This is a pure structural refactor. All existing endpoints, models, and UI flows remain identical.
+
+## Capabilities
+
+### New Capabilities
+- None (pure refactor)
+
+### Modified Capabilities
+- `backend-structure`: Cleaner module boundaries, smaller files, separated concerns
+- `frontend-structure`: Feature-organized components, consistent file naming
+
+## Impact
+
+- **Backend**: `apps/api/src/schemas/*` (new), `apps/api/src/services/docker/*` (new package), `apps/api/src/services/instance_lifecycle.py` (new), `apps/api/src/services/config_profiles.py` (new)
+- **Backend**: `apps/api/src/api/*.py` (reduced in size, imports change)
+- **Backend**: `apps/api/src/auth/dependencies.py` (new `get_current_user`)
+- **Frontend**: `apps/web/src/components/features/*` (new directories), `apps/web/src/api/*` (renamed to kebab-case)
+- **Frontend**: `apps/web/src/pages/*` (renamed to `*Page.tsx`)
+
+## Exclusions (Already Done)
+
+The following behavioral features from `b6f89f9` are **already present** in current `dev` and out of scope for this refactor:
+- Built-in tool type seeding (`src/seeds/builtin_tool_types.py`)
+- Config profile default management (endpoints + `UserConfig` properties)
+- Config profile unique constraint (`uq_config_profiles_user_name`)
+- SSH key mounting in instance lifecycle
+- Terminal backend (WebSocket, session management)
+- Tunnel URL regex fix
+- Session auto-numbering
+- Config profile resolver (`services/config_profile_resolver.py`)
@@ -0,0 +1,75 @@
+## Scope
+
+This change is a **pure structural refactoring** of the backend and frontend codebase. No API contracts, database schemas, or user-facing behaviors change.
+
+### In Scope
+
+1. **Schema extraction** (`apps/api/src/schemas/`)
+   - Extract Pydantic models from API routers into dedicated schema modules
+   - Schemas to extract: `tool_type`, `tool_instance`, `config_profile`, `user`, `user_config`, `project`, `ssh_key`, `git_repository`, `health`
+   - Each API router imports schemas from `src.schemas.*` instead of defining inline
+
+2. **Docker service package** (`apps/api/src/services/docker/`)
+   - Split `services/docker.py` monolith into focused modules:
+     - `docker/compose.py` — compose file generation, modification, validation
+     - `docker/container.py` — container lifecycle (status, IP, network, logs)
+     - `docker/config_staging.py` — config file staging for instances
+     - `docker/tunnel.py` — tunnel URL extraction (moved from `services/tunnel.py`)
+     - `docker/__init__.py` — re-exports for backward compatibility
+   - Update all imports across the backend
+
+3. **Instance lifecycle extraction** (`apps/api/src/services/instance_lifecycle.py`)
+   - Extract instance creation, start, stop, restart, and deletion logic from `api/tool_instances.py`
+   - The API router becomes thin: validates auth, calls service, returns response
+   - Service functions are async and receive `AsyncSession`, models, and raw parameters
+
+4. **Config profile service extraction** (`apps/api/src/services/config_profiles.py`)
+   - Extract business logic from `api/config_profiles.py`: CRUD helpers, validation, default profile management
+   - API router delegates to service functions
+
+5. **Auth dependency refactor** (`apps/api/src/auth/dependencies.py`)
+   - Add `get_current_user` dependency that returns a `User` model directly
+   - Update API routers to use `user: User = Depends(get_current_user)` where the full user object is needed
+   - Keep `get_current_user_id` for endpoints that only need the ID
+
+6. **Frontend reorganization**
+   - Move components into `features/` directories by domain:
+     - `features/git/` — CommitDialog, FileBrowser, FileEditor, GitToolbar, MergeDialog, WorkspaceSidebar
+     - `features/dashboard/` — ActiveSessionsList, DashboardSummary, ProjectsSection, QuickCreateForm, RecentSessionsSection
+     - `features/project/` — RepositoriesSettingsTab
+     - `features/tool-workshop/` — ToolTypesTab (already exists)
+   - Rename API files from snake_case to kebab-case:
+     - `tool_types.ts` → `tool-types.ts`
+     - `ssh_keys.ts` → `ssh-keys.ts`
+     - `git_repositories.ts` → `git-repositories.ts`
+   - Rename page files to `*Page.tsx`:
+     - `dashboard.tsx` → `DashboardPage.tsx`
+     - `projects.tsx` → `ProjectsPage.tsx`
+     - etc.
+
+### Out of Scope
+
+- Any new features or behavioral changes
+- Database schema changes (no migrations)
+- API contract changes (same endpoints, same request/response shapes)
+- Frontend UI behavior changes (same components, same interactions)
+- Removing or modifying the `ConfigProfileInclude` model (already exists as a relation)
+- Extracting `ConfigMount` into a separate model (current dev uses JSON arrays; this is a schema decision, not a refactor)
+- Changes to `tool_definition_manifest` or `workspace` models
+
+## Acceptance Criteria
+
+1. All existing tests pass without modification (behavior unchanged)
+2. All existing API endpoints return identical responses for identical requests
+3. All frontend pages render identically
+4. `docker compose up` starts successfully
+5. Backend `ruff check` passes
+6. Frontend `npm run typecheck` passes
+7. Frontend `npm run build` passes
+8. File sizes are reduced: no API router > 500 lines, no service > 400 lines
+
+## Preconditions
+
+- Current `dev` branch is stable and all behavioral forward-ports are complete
+- All legacy `ToolConfig`/`ConfigFolder` code has been removed
+- Database migrations are at a single head
@@ -0,0 +1,118 @@
+## 1. Backend Schema Extraction
+
+- [ ] 1.1 Create `apps/api/src/schemas/__init__.py` with re-exports
+- [ ] 1.2 Extract `schemas/tool_type.py` from `api/tool_types.py` (ToolTypeCreate, ToolTypeUpdate, validation schemas)
+- [ ] 1.3 Extract `schemas/tool_instance.py` from `api/tool_instances.py` (CreateInstanceRequest, StartInstanceRequest, SessionItemResponse, SessionListResponse)
+- [ ] 1.4 Extract `schemas/config_profile.py` from `api/config_profiles.py` (ConfigProfileCreate, ConfigProfileUpdate, ConfigProfileResponse, DefaultProfilesUpdate, ValidateGitUrlRequest, ValidateGitUrlResponse, GitMountItem, MountItem)
+- [ ] 1.5 Extract `schemas/health.py` from `api/health.py` (DatabaseHealth, DiskHealth, HealthChecks, HealthResponse, DatabaseHealthResponse)
+- [ ] 1.6 Extract `schemas/user.py` from `api/users.py` (UserProfileResponse, UserProfileUpdate)
+- [ ] 1.7 Extract `schemas/user_config.py` from `api/user_config.py` (any request/response schemas)
+- [ ] 1.8 Extract `schemas/project.py` from `api/projects.py` (any request/response schemas)
+- [ ] 1.9 Extract `schemas/ssh_key.py` from `api/ssh_keys.py` (any request/response schemas)
+- [ ] 1.10 Extract `schemas/git_repository.py` from `api/git_repositories.py` (any request/response schemas)
+- [ ] 1.11 Update all API routers to import schemas from `src.schemas.*` instead of defining inline
+- [ ] 1.12 Verify `py_compile` and `ruff` pass on all schema files
+
+## 2. Docker Service Package Split
+
+- [ ] 2.1 Create `apps/api/src/services/docker/__init__.py` with re-exports for backward compatibility
+- [ ] 2.2 Create `apps/api/src/services/docker/compose.py` from `services/docker.py`:
+  - `render_compose_template`, `write_compose_file`, `_modify_compose_file`, `_ensure_container_name_in_compose`, `_ensure_web_bind_address`, `_ensure_backend_network_in_compose`, `_sanitize_compose_file`, `sort_volumes_by_specificity`
+- [ ] 2.3 Create `apps/api/src/services/docker/container.py` from `services/docker.py`:
+  - `get_container_status`, `get_container_logs`, `get_container_id`, `wait_for_container_running`, `get_container_ip_on_network`, `is_container_on_network`, `connect_container_to_network`, `get_backend_network_name`
+- [ ] 2.4 Create `apps/api/src/services/docker/config_staging.py` from `services/docker.py`:
+  - `write_env_file`, `write_config_files`, `ensure_instance_directory`
+- [ ] 2.5 Create `apps/api/src/services/docker/tunnel.py` from `services/tunnel.py`:
+  - `extract_tunnel_url` (move tunnel URL regex extraction here), `start_tunnel`, `stop_tunnel`, `check_tunnel_health`, `recreate_tunnel`
+- [ ] 2.6 Remove `apps/api/src/services/docker.py` after verifying all imports updated
+- [ ] 2.7 Update `services/tunnel.py` to delegate URL extraction to `docker/tunnel.py` or remove if fully subsumed
+- [ ] 2.8 Update all consumers (`api/tool_instances.py`, `services/instance_lifecycle.py`, etc.) to import from `services.docker` package
+- [ ] 2.9 Verify `py_compile` and `ruff` pass
+
+## 3. Instance Lifecycle Extraction
+
+- [ ] 3.1 Create `apps/api/src/services/instance_lifecycle.py`:
+  - Extract `create_new_instance`, `start_existing_instance`, `stop_existing_instance`, `restart_existing_instance`, `delete_existing_instance` from `api/tool_instances.py`
+  - Extract helper functions: `_prepare_manifest_instance`, `_modify_compose_file`, `_ensure_container_name_in_compose`, `_ensure_web_bind_address`, `_ensure_backend_network_in_compose`, `_sanitize_compose_file`, `_resolve_git_mounts`, `_clone_git_repo`, etc.
+- [ ] 3.2 Thin `api/tool_instances.py` to ~300 lines:
+  - HTTP routing, auth validation, request parsing
+  - Delegate to `instance_lifecycle.py` service functions
+- [ ] 3.3 Move `sessions_router` from `api/tool_instances.py` to `api/users.py` or keep as separate `api/sessions.py` (align with `b6f89f9` pattern)
+- [ ] 3.4 Verify all instance endpoints (create, start, stop, restart, delete, list, get) still work
+- [ ] 3.5 Verify `py_compile` and `ruff` pass
+
+## 4. Config Profile Service Extraction
+
+- [ ] 4.1 Create `apps/api/src/services/config_profiles.py`:
+  - Extract `get_owned_profile`, `check_duplicate_name`, `profile_to_dict`, `_validate_default_profiles`, `get_default_profiles`, `set_default_profiles`, `get_default_profile_for_tool_type`, `get_or_create_user_config`, `list_includes_for_profile`, `list_mounts_for_profile` from `api/config_profiles.py`
+  - Add cycle detection helpers (`_detect_cycle`, `validate_includes_no_cycle`)
+- [ ] 4.2 Thin `api/config_profiles.py` to ~200 lines:
+  - HTTP routing, request parsing
+  - Delegate to `services/config_profiles.py`
+- [ ] 4.3 Verify all config profile endpoints (CRUD, includes, defaults, preview, validate-git-url) still work
+- [ ] 4.4 Verify `py_compile` and `ruff` pass
+
+## 5. Auth Dependency Refactor
+
+- [ ] 5.1 Add `get_current_user` to `apps/api/src/auth/dependencies.py`:
+  - Decode session cookie, look up user in DB, return `User` model
+  - Raise 401 if missing/invalid session or user not found
+- [ ] 5.2 Migrate `api/users.py` to use `get_current_user` instead of `get_current_user_id` + `_get_user`
+- [ ] 5.3 Migrate `api/tool_instances.py` sessions_router to use `get_current_user` where appropriate
+- [ ] 5.4 Migrate other routers incrementally (dashboard, projects, etc.) where the full user object is needed
+- [ ] 5.5 Keep `get_current_user_id` for endpoints that only need the ID
+- [ ] 5.6 Verify `py_compile` and `ruff` pass
+
+## 6. Frontend Reorganization
+
+- [ ] 6.1 Rename API files to kebab-case:
+  - `tool_types.ts` → `tool-types.ts`
+  - `ssh_keys.ts` → `ssh-keys.ts`
+  - `git_repositories.ts` → `git-repositories.ts`
+  - Update all imports in pages and components
+- [ ] 6.2 Rename page files to `*Page.tsx`:
+  - `dashboard.tsx` → `DashboardPage.tsx`
+  - `projects.tsx` → `ProjectsPage.tsx`
+  - `sessions.tsx` → `SessionsPage.tsx`
+  - `settings.tsx` → `SettingsPage.tsx`
+  - `ssh-keys.tsx` → `SshKeysPage.tsx`
+  - `terminal.tsx` → `TerminalPage.tsx`
+  - `tool-workshop.tsx` → `ToolWorkshopPage.tsx`
+  - `config-profiles.tsx` → `ConfigProfilesPage.tsx`
+  - `git-repositories.tsx` → `GitRepositoriesPage.tsx`
+  - `repo-workspace.tsx` → `RepoWorkspacePage.tsx`
+  - `workspaces.tsx` → `WorkspacesPage.tsx`
+  - `workspace-detail.tsx` → `WorkspaceDetailPage.tsx`
+  - `profile.tsx` → `ProfilePage.tsx`
+  - `project-settings.tsx` → `ProjectSettingsPage.tsx`
+  - `git-history.tsx` → `GitHistoryPage.tsx`
+  - Update `router.tsx` imports
+- [ ] 6.3 Move components into `features/` directories:
+  - Create `components/features/git/` and move git-related components
+  - Create `components/features/dashboard/` and move dashboard components
+  - Create `components/features/project/` and move project components
+  - Update all imports
+- [ ] 6.4 Verify `npm run typecheck` passes
+- [ ] 6.5 Verify `npm run build` passes
+
+## 7. Integration and Verification
+
+- [ ] 7.1 Run backend tests: `docker exec hq-api pytest`
+- [ ] 7.2 Run backend typecheck: `pyright` or equivalent
+- [ ] 7.3 Run backend lint: `ruff check`
+- [ ] 7.4 Run frontend typecheck: `npm run typecheck`
+- [ ] 7.5 Run frontend build: `npm run build`
+- [ ] 7.6 Run `docker compose up --build` and verify API starts
+- [ ] 7.7 Verify key user flows manually:
+  - Create a tool instance
+  - Start/stop an instance
+  - Create a config profile
+  - Set a default config profile
+  - View sessions list
+  - Open terminal
+- [ ] 7.8 Verify no 404s or import errors in browser console
+
+## 8. Documentation
+
+- [ ] 8.1 Update `AGENTS.md` or backend README with new module structure
+- [ ] 8.2 Document the `get_current_user` vs `get_current_user_id` pattern for future contributors