diff --git a/openspec/changes/backend-frontend-refactoring/.openspec.yaml b/openspec/changes/backend-frontend-refactoring/.openspec.yaml new file mode 100644 index 0000000..0ba725f --- /dev/null +++ b/openspec/changes/backend-frontend-refactoring/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-03 diff --git a/openspec/changes/backend-frontend-refactoring/design.md b/openspec/changes/backend-frontend-refactoring/design.md new file mode 100644 index 0000000..97d3c6e --- /dev/null +++ b/openspec/changes/backend-frontend-refactoring/design.md @@ -0,0 +1,172 @@ +## Context + +Current `dev` has all behavioral features from the overwritten `main` merge, but the code structure is pre-refactor: +- Monolithic `api/tool_instances.py` (~3000 lines) +- Monolithic `api/config_profiles.py` (~1000 lines) +- Monolithic `services/docker.py` +- No `schemas/` directory +- Flat frontend component structure with inconsistent naming + +The `b6f89f9` merge from `main` had a clean refactoring that we need to redo, but adapted to our current reality. + +## Goals / Non-Goals + +**Goals:** +- Extract Pydantic schemas from API routers into `src/schemas/` +- Split `services/docker.py` into `services/docker/` package +- Extract instance lifecycle logic from `api/tool_instances.py` into `services/instance_lifecycle.py` +- Extract config profile business logic from `api/config_profiles.py` into `services/config_profiles.py` +- Add `get_current_user` auth dependency and migrate routers that need the full user object +- Reorganize frontend components into `features/` directories +- Standardize frontend API file naming to kebab-case +- Standardize frontend page naming to `*Page.tsx` + +**Non-Goals:** +- Changing any API request/response shapes +- Changing any database schemas +- Adding new features +- Modifying frontend component behavior or styling +- Converting `ConfigProfile` mounts from JSON to relation tables (out of scope — would require migration) + +## Decisions + +### 1. Schema Extraction: One File Per Domain +**Decision:** Each domain gets its own schema file: `schemas/tool_type.py`, `schemas/tool_instance.py`, etc. +**Rationale:** Keeps schemas close to their domain. Avoids a giant `schemas.py`. + +### 2. Docker Service Split: Functional Boundaries +**Decision:** Split by responsibility: +- `compose.py` — compose file generation, modification, port injection, network injection +- `container.py` — container status, IP lookup, network connect, logs +- `config_staging.py` — staging config files into instance directories +- `tunnel.py` — extracting tunnel URLs from cloudflared output +**Rationale:** Each module has a single reason to change. `docker.py` mixed compose logic with container runtime queries. + +### 3. Instance Lifecycle: Service Receives Raw Params, Not Request Objects +**Decision:** Service functions receive model instances and primitive parameters, not FastAPI request objects. +**Example:** `create_instance(session, user, project, repo, tool_type, data: CreateInstanceRequest)` → service extracts fields. +**Rationale:** Keeps service layer independent of HTTP framework. Easier to test. + +### 4. Auth Pattern: Gradual Migration, Not Big Bang +**Decision:** Add `get_current_user` alongside existing `get_current_user_id`. Migrate routers incrementally. +**Rationale:** Reduces risk. Endpoints that only need the ID can keep the old pattern. + +### 5. Frontend Naming: Align with `b6f89f9` Conventions +**Decision:** Use kebab-case for API files, PascalCase for page files with `Page` suffix, `features/` for component directories. +**Rationale:** Matches the `b6f89f9` structure that was already reviewed and accepted. + +## Module Map + +### Backend — Before +``` +api/ + tool_instances.py (~3000 lines) — HTTP + Docker + Git + Lifecycle + config_profiles.py (~1000 lines) — HTTP + Validation + Defaults + tool_types.py (~500 lines) — HTTP + Schemas + health.py (~150 lines) — HTTP + Schemas + users.py (~100 lines) — HTTP + Schemas + ... +services/ + docker.py (~600 lines) — Compose + Container + Tunnel +``` + +### Backend — After +``` +schemas/ + tool_type.py (~200 lines) + tool_instance.py (~40 lines) + config_profile.py (~130 lines) + health.py (~50 lines) + user.py (~20 lines) + ... +api/ + tool_instances.py (~300 lines) — HTTP routing only + config_profiles.py (~200 lines) — HTTP routing only + tool_types.py (~250 lines) — HTTP + validation endpoints + health.py (~80 lines) — HTTP only + users.py (~60 lines) — HTTP only + ... +services/ + instance_lifecycle.py (~420 lines) — Create/Start/Stop/Restart/Delete + config_profiles.py (~300 lines) — CRUD + Defaults + Validation + docker/ + __init__.py (~40 lines) — Re-exports + compose.py (~240 lines) — Compose generation + container.py (~120 lines) — Container queries + config_staging.py (~80 lines) — File staging + tunnel.py (~150 lines) — Tunnel URL extraction +``` + +### Frontend — Before +``` +src/ + api/ + tool_types.ts + ssh_keys.ts + git_repositories.ts + sessions.ts + components/ + git-toolbar.tsx + file-editor.tsx + commit-dialog.tsx + ... + pages/ + dashboard.tsx + projects.tsx + sessions.tsx + ... +``` + +### Frontend — After +``` +src/ + api/ + tool-types.ts + ssh-keys.ts + git-repositories.ts + sessions.ts + components/ + features/ + git/ + GitToolbar.tsx + FileBrowser.tsx + FileEditor.tsx + CommitDialog.tsx + MergeDialog.tsx + WorkspaceSidebar.tsx + dashboard/ + DashboardSummary.tsx + ActiveSessionsList.tsx + ProjectsSection.tsx + QuickCreateForm.tsx + RecentSessionsSection.tsx + project/ + RepositoriesSettingsTab.tsx + tool-workshop/ + ToolTypesTab.tsx + ProtectedRoute.tsx + AppShell.tsx + pages/ + DashboardPage.tsx + ProjectsPage.tsx + SessionsPage.tsx + ... +``` + +## Risks / Trade-offs + +**[Risk] Import cycles during extraction** → **Mitigation:** Extract schemas first (no service dependencies), then services, then thin routers last. Use TYPE_CHECKING guards. + +**[Risk] Merge conflicts with in-flight features** → **Mitigation:** Coordinate timing. This refactor should be the only large change on `dev` while it's in progress. Freeze other backend work. + +**[Risk] Frontend renaming breaks imports** → **Mitigation:** Use `git mv` for renames so git tracks history. Update all imports in a single commit. + +**[Risk] Missing re-export in docker/__init__.py breaks consumers** → **Mitigation:** After splitting, run a full import test across all backend files. Add any missing re-exports. + +## Migration Plan + +1. **Phase 1: Schemas** — Extract all Pydantic models into `schemas/`. Update imports in API routers. No logic changes. +2. **Phase 2: Services** — Split `docker.py`, extract `instance_lifecycle.py`, extract `config_profiles.py`. Update imports. +3. **Phase 3: Auth** — Add `get_current_user`, migrate routers that need the full user object. +4. **Phase 4: Frontend** — Rename files, move components, update imports. +5. **Phase 5: Verification** — Run full test suite, typecheck, build. diff --git a/openspec/changes/backend-frontend-refactoring/proposal.md b/openspec/changes/backend-frontend-refactoring/proposal.md new file mode 100644 index 0000000..076653a --- /dev/null +++ b/openspec/changes/backend-frontend-refactoring/proposal.md @@ -0,0 +1,52 @@ +## Why + +The `main` branch was previously merged into `dev` (commit `b6f89f9`) bringing a large structural refactoring: schema extraction, service splitting, auth dependency pattern changes, and frontend reorganization. This merge was later overwritten when `dev` was reset to a pre-merge clean state (`51a399c`). + +We have since forward-ported all behavioral features (built-in tool type seeding, config profile defaults, unique constraints, SSH key mounting, terminal backend, etc.) onto the clean `dev` base. **The codebase now works functionally but lacks the structural cleanliness of the refactoring.** + +Monolithic files make the backend harder to navigate, test, and maintain: +- `api/tool_instances.py` is ~3000 lines (mixing HTTP handling with Docker orchestration) +- `api/config_profiles.py` is ~1000 lines (mixing HTTP handling with business logic) +- `services/docker.py` is a monolith of ~600 lines covering compose, container, and tunnel logic +- Frontend API files use inconsistent naming (`tool_types.ts` vs `tool-types.ts`) +- Frontend components are flat in `components/` instead of organized by feature domain + +## What Changes + +Redo the structural refactoring from `b6f89f9`, **adapted to current dev reality**: + +1. **Backend schema extraction** — Extract Pydantic request/response schemas from API routers into `apps/api/src/schemas/` +2. **Backend service splits** — Split `services/docker.py` into `services/docker/` package; extract `services/instance_lifecycle.py` and `services/config_profiles.py` +3. **Auth dependency refactor** — Change from `get_current_user_id` + manual `_get_user` calls to `get_current_user` dependency returning `User` directly +4. **Frontend reorganization** — Move components into `features/` directories; rename API files to kebab-case + +**No behavioral changes.** This is a pure structural refactor. All existing endpoints, models, and UI flows remain identical. + +## Capabilities + +### New Capabilities +- None (pure refactor) + +### Modified Capabilities +- `backend-structure`: Cleaner module boundaries, smaller files, separated concerns +- `frontend-structure`: Feature-organized components, consistent file naming + +## Impact + +- **Backend**: `apps/api/src/schemas/*` (new), `apps/api/src/services/docker/*` (new package), `apps/api/src/services/instance_lifecycle.py` (new), `apps/api/src/services/config_profiles.py` (new) +- **Backend**: `apps/api/src/api/*.py` (reduced in size, imports change) +- **Backend**: `apps/api/src/auth/dependencies.py` (new `get_current_user`) +- **Frontend**: `apps/web/src/components/features/*` (new directories), `apps/web/src/api/*` (renamed to kebab-case) +- **Frontend**: `apps/web/src/pages/*` (renamed to `*Page.tsx`) + +## Exclusions (Already Done) + +The following behavioral features from `b6f89f9` are **already present** in current `dev` and out of scope for this refactor: +- Built-in tool type seeding (`src/seeds/builtin_tool_types.py`) +- Config profile default management (endpoints + `UserConfig` properties) +- Config profile unique constraint (`uq_config_profiles_user_name`) +- SSH key mounting in instance lifecycle +- Terminal backend (WebSocket, session management) +- Tunnel URL regex fix +- Session auto-numbering +- Config profile resolver (`services/config_profile_resolver.py`) diff --git a/openspec/changes/backend-frontend-refactoring/spec.md b/openspec/changes/backend-frontend-refactoring/spec.md new file mode 100644 index 0000000..8c58fba --- /dev/null +++ b/openspec/changes/backend-frontend-refactoring/spec.md @@ -0,0 +1,75 @@ +## Scope + +This change is a **pure structural refactoring** of the backend and frontend codebase. No API contracts, database schemas, or user-facing behaviors change. + +### In Scope + +1. **Schema extraction** (`apps/api/src/schemas/`) + - Extract Pydantic models from API routers into dedicated schema modules + - Schemas to extract: `tool_type`, `tool_instance`, `config_profile`, `user`, `user_config`, `project`, `ssh_key`, `git_repository`, `health` + - Each API router imports schemas from `src.schemas.*` instead of defining inline + +2. **Docker service package** (`apps/api/src/services/docker/`) + - Split `services/docker.py` monolith into focused modules: + - `docker/compose.py` — compose file generation, modification, validation + - `docker/container.py` — container lifecycle (status, IP, network, logs) + - `docker/config_staging.py` — config file staging for instances + - `docker/tunnel.py` — tunnel URL extraction (moved from `services/tunnel.py`) + - `docker/__init__.py` — re-exports for backward compatibility + - Update all imports across the backend + +3. **Instance lifecycle extraction** (`apps/api/src/services/instance_lifecycle.py`) + - Extract instance creation, start, stop, restart, and deletion logic from `api/tool_instances.py` + - The API router becomes thin: validates auth, calls service, returns response + - Service functions are async and receive `AsyncSession`, models, and raw parameters + +4. **Config profile service extraction** (`apps/api/src/services/config_profiles.py`) + - Extract business logic from `api/config_profiles.py`: CRUD helpers, validation, default profile management + - API router delegates to service functions + +5. **Auth dependency refactor** (`apps/api/src/auth/dependencies.py`) + - Add `get_current_user` dependency that returns a `User` model directly + - Update API routers to use `user: User = Depends(get_current_user)` where the full user object is needed + - Keep `get_current_user_id` for endpoints that only need the ID + +6. **Frontend reorganization** + - Move components into `features/` directories by domain: + - `features/git/` — CommitDialog, FileBrowser, FileEditor, GitToolbar, MergeDialog, WorkspaceSidebar + - `features/dashboard/` — ActiveSessionsList, DashboardSummary, ProjectsSection, QuickCreateForm, RecentSessionsSection + - `features/project/` — RepositoriesSettingsTab + - `features/tool-workshop/` — ToolTypesTab (already exists) + - Rename API files from snake_case to kebab-case: + - `tool_types.ts` → `tool-types.ts` + - `ssh_keys.ts` → `ssh-keys.ts` + - `git_repositories.ts` → `git-repositories.ts` + - Rename page files to `*Page.tsx`: + - `dashboard.tsx` → `DashboardPage.tsx` + - `projects.tsx` → `ProjectsPage.tsx` + - etc. + +### Out of Scope + +- Any new features or behavioral changes +- Database schema changes (no migrations) +- API contract changes (same endpoints, same request/response shapes) +- Frontend UI behavior changes (same components, same interactions) +- Removing or modifying the `ConfigProfileInclude` model (already exists as a relation) +- Extracting `ConfigMount` into a separate model (current dev uses JSON arrays; this is a schema decision, not a refactor) +- Changes to `tool_definition_manifest` or `workspace` models + +## Acceptance Criteria + +1. All existing tests pass without modification (behavior unchanged) +2. All existing API endpoints return identical responses for identical requests +3. All frontend pages render identically +4. `docker compose up` starts successfully +5. Backend `ruff check` passes +6. Frontend `npm run typecheck` passes +7. Frontend `npm run build` passes +8. File sizes are reduced: no API router > 500 lines, no service > 400 lines + +## Preconditions + +- Current `dev` branch is stable and all behavioral forward-ports are complete +- All legacy `ToolConfig`/`ConfigFolder` code has been removed +- Database migrations are at a single head diff --git a/openspec/changes/backend-frontend-refactoring/tasks.md b/openspec/changes/backend-frontend-refactoring/tasks.md new file mode 100644 index 0000000..d22e5ea --- /dev/null +++ b/openspec/changes/backend-frontend-refactoring/tasks.md @@ -0,0 +1,118 @@ +## 1. Backend Schema Extraction + +- [ ] 1.1 Create `apps/api/src/schemas/__init__.py` with re-exports +- [ ] 1.2 Extract `schemas/tool_type.py` from `api/tool_types.py` (ToolTypeCreate, ToolTypeUpdate, validation schemas) +- [ ] 1.3 Extract `schemas/tool_instance.py` from `api/tool_instances.py` (CreateInstanceRequest, StartInstanceRequest, SessionItemResponse, SessionListResponse) +- [ ] 1.4 Extract `schemas/config_profile.py` from `api/config_profiles.py` (ConfigProfileCreate, ConfigProfileUpdate, ConfigProfileResponse, DefaultProfilesUpdate, ValidateGitUrlRequest, ValidateGitUrlResponse, GitMountItem, MountItem) +- [ ] 1.5 Extract `schemas/health.py` from `api/health.py` (DatabaseHealth, DiskHealth, HealthChecks, HealthResponse, DatabaseHealthResponse) +- [ ] 1.6 Extract `schemas/user.py` from `api/users.py` (UserProfileResponse, UserProfileUpdate) +- [ ] 1.7 Extract `schemas/user_config.py` from `api/user_config.py` (any request/response schemas) +- [ ] 1.8 Extract `schemas/project.py` from `api/projects.py` (any request/response schemas) +- [ ] 1.9 Extract `schemas/ssh_key.py` from `api/ssh_keys.py` (any request/response schemas) +- [ ] 1.10 Extract `schemas/git_repository.py` from `api/git_repositories.py` (any request/response schemas) +- [ ] 1.11 Update all API routers to import schemas from `src.schemas.*` instead of defining inline +- [ ] 1.12 Verify `py_compile` and `ruff` pass on all schema files + +## 2. Docker Service Package Split + +- [ ] 2.1 Create `apps/api/src/services/docker/__init__.py` with re-exports for backward compatibility +- [ ] 2.2 Create `apps/api/src/services/docker/compose.py` from `services/docker.py`: + - `render_compose_template`, `write_compose_file`, `_modify_compose_file`, `_ensure_container_name_in_compose`, `_ensure_web_bind_address`, `_ensure_backend_network_in_compose`, `_sanitize_compose_file`, `sort_volumes_by_specificity` +- [ ] 2.3 Create `apps/api/src/services/docker/container.py` from `services/docker.py`: + - `get_container_status`, `get_container_logs`, `get_container_id`, `wait_for_container_running`, `get_container_ip_on_network`, `is_container_on_network`, `connect_container_to_network`, `get_backend_network_name` +- [ ] 2.4 Create `apps/api/src/services/docker/config_staging.py` from `services/docker.py`: + - `write_env_file`, `write_config_files`, `ensure_instance_directory` +- [ ] 2.5 Create `apps/api/src/services/docker/tunnel.py` from `services/tunnel.py`: + - `extract_tunnel_url` (move tunnel URL regex extraction here), `start_tunnel`, `stop_tunnel`, `check_tunnel_health`, `recreate_tunnel` +- [ ] 2.6 Remove `apps/api/src/services/docker.py` after verifying all imports updated +- [ ] 2.7 Update `services/tunnel.py` to delegate URL extraction to `docker/tunnel.py` or remove if fully subsumed +- [ ] 2.8 Update all consumers (`api/tool_instances.py`, `services/instance_lifecycle.py`, etc.) to import from `services.docker` package +- [ ] 2.9 Verify `py_compile` and `ruff` pass + +## 3. Instance Lifecycle Extraction + +- [ ] 3.1 Create `apps/api/src/services/instance_lifecycle.py`: + - Extract `create_new_instance`, `start_existing_instance`, `stop_existing_instance`, `restart_existing_instance`, `delete_existing_instance` from `api/tool_instances.py` + - Extract helper functions: `_prepare_manifest_instance`, `_modify_compose_file`, `_ensure_container_name_in_compose`, `_ensure_web_bind_address`, `_ensure_backend_network_in_compose`, `_sanitize_compose_file`, `_resolve_git_mounts`, `_clone_git_repo`, etc. +- [ ] 3.2 Thin `api/tool_instances.py` to ~300 lines: + - HTTP routing, auth validation, request parsing + - Delegate to `instance_lifecycle.py` service functions +- [ ] 3.3 Move `sessions_router` from `api/tool_instances.py` to `api/users.py` or keep as separate `api/sessions.py` (align with `b6f89f9` pattern) +- [ ] 3.4 Verify all instance endpoints (create, start, stop, restart, delete, list, get) still work +- [ ] 3.5 Verify `py_compile` and `ruff` pass + +## 4. Config Profile Service Extraction + +- [ ] 4.1 Create `apps/api/src/services/config_profiles.py`: + - Extract `get_owned_profile`, `check_duplicate_name`, `profile_to_dict`, `_validate_default_profiles`, `get_default_profiles`, `set_default_profiles`, `get_default_profile_for_tool_type`, `get_or_create_user_config`, `list_includes_for_profile`, `list_mounts_for_profile` from `api/config_profiles.py` + - Add cycle detection helpers (`_detect_cycle`, `validate_includes_no_cycle`) +- [ ] 4.2 Thin `api/config_profiles.py` to ~200 lines: + - HTTP routing, request parsing + - Delegate to `services/config_profiles.py` +- [ ] 4.3 Verify all config profile endpoints (CRUD, includes, defaults, preview, validate-git-url) still work +- [ ] 4.4 Verify `py_compile` and `ruff` pass + +## 5. Auth Dependency Refactor + +- [ ] 5.1 Add `get_current_user` to `apps/api/src/auth/dependencies.py`: + - Decode session cookie, look up user in DB, return `User` model + - Raise 401 if missing/invalid session or user not found +- [ ] 5.2 Migrate `api/users.py` to use `get_current_user` instead of `get_current_user_id` + `_get_user` +- [ ] 5.3 Migrate `api/tool_instances.py` sessions_router to use `get_current_user` where appropriate +- [ ] 5.4 Migrate other routers incrementally (dashboard, projects, etc.) where the full user object is needed +- [ ] 5.5 Keep `get_current_user_id` for endpoints that only need the ID +- [ ] 5.6 Verify `py_compile` and `ruff` pass + +## 6. Frontend Reorganization + +- [ ] 6.1 Rename API files to kebab-case: + - `tool_types.ts` → `tool-types.ts` + - `ssh_keys.ts` → `ssh-keys.ts` + - `git_repositories.ts` → `git-repositories.ts` + - Update all imports in pages and components +- [ ] 6.2 Rename page files to `*Page.tsx`: + - `dashboard.tsx` → `DashboardPage.tsx` + - `projects.tsx` → `ProjectsPage.tsx` + - `sessions.tsx` → `SessionsPage.tsx` + - `settings.tsx` → `SettingsPage.tsx` + - `ssh-keys.tsx` → `SshKeysPage.tsx` + - `terminal.tsx` → `TerminalPage.tsx` + - `tool-workshop.tsx` → `ToolWorkshopPage.tsx` + - `config-profiles.tsx` → `ConfigProfilesPage.tsx` + - `git-repositories.tsx` → `GitRepositoriesPage.tsx` + - `repo-workspace.tsx` → `RepoWorkspacePage.tsx` + - `workspaces.tsx` → `WorkspacesPage.tsx` + - `workspace-detail.tsx` → `WorkspaceDetailPage.tsx` + - `profile.tsx` → `ProfilePage.tsx` + - `project-settings.tsx` → `ProjectSettingsPage.tsx` + - `git-history.tsx` → `GitHistoryPage.tsx` + - Update `router.tsx` imports +- [ ] 6.3 Move components into `features/` directories: + - Create `components/features/git/` and move git-related components + - Create `components/features/dashboard/` and move dashboard components + - Create `components/features/project/` and move project components + - Update all imports +- [ ] 6.4 Verify `npm run typecheck` passes +- [ ] 6.5 Verify `npm run build` passes + +## 7. Integration and Verification + +- [ ] 7.1 Run backend tests: `docker exec hq-api pytest` +- [ ] 7.2 Run backend typecheck: `pyright` or equivalent +- [ ] 7.3 Run backend lint: `ruff check` +- [ ] 7.4 Run frontend typecheck: `npm run typecheck` +- [ ] 7.5 Run frontend build: `npm run build` +- [ ] 7.6 Run `docker compose up --build` and verify API starts +- [ ] 7.7 Verify key user flows manually: + - Create a tool instance + - Start/stop an instance + - Create a config profile + - Set a default config profile + - View sessions list + - Open terminal +- [ ] 7.8 Verify no 404s or import errors in browser console + +## 8. Documentation + +- [ ] 8.1 Update `AGENTS.md` or backend README with new module structure +- [ ] 8.2 Document the `get_current_user` vs `get_current_user_id` pattern for future contributors