docs: update refactoring spec with submodule architecture

Enforces max 5-10 files per directory using proper subpackages:
- api/tool/, api/config/, api/workspace/, api/user/, api/project/, api/system/
- services/docker/, services/instance/, services/config/, services/git/,
  services/build/, services/terminal/, services/shared/
- models/tool/, models/config/, models/user/, models/project/, models/system/
- schemas/tool/, schemas/config/, schemas/user/, schemas/project/, schemas/system/

Updated design.md module map and tasks.md with 9 phases.

openspec/changes/backend-frontend-refactoring/tasks.md

@@ -1,118 +1,158 @@
-## 1. Backend Schema Extraction
+## Phase 0: Submodule Infrastructure (Create Directories + __init__.py Files)
 
-- [ ] 1.1 Create `apps/api/src/schemas/__init__.py` with re-exports
-- [ ] 1.2 Extract `schemas/tool_type.py` from `api/tool_types.py` (ToolTypeCreate, ToolTypeUpdate, validation schemas)
-- [ ] 1.3 Extract `schemas/tool_instance.py` from `api/tool_instances.py` (CreateInstanceRequest, StartInstanceRequest, SessionItemResponse, SessionListResponse)
-- [ ] 1.4 Extract `schemas/config_profile.py` from `api/config_profiles.py` (ConfigProfileCreate, ConfigProfileUpdate, ConfigProfileResponse, DefaultProfilesUpdate, ValidateGitUrlRequest, ValidateGitUrlResponse, GitMountItem, MountItem)
-- [ ] 1.5 Extract `schemas/health.py` from `api/health.py` (DatabaseHealth, DiskHealth, HealthChecks, HealthResponse, DatabaseHealthResponse)
-- [ ] 1.6 Extract `schemas/user.py` from `api/users.py` (UserProfileResponse, UserProfileUpdate)
-- [ ] 1.7 Extract `schemas/user_config.py` from `api/user_config.py` (any request/response schemas)
-- [ ] 1.8 Extract `schemas/project.py` from `api/projects.py` (any request/response schemas)
-- [ ] 1.9 Extract `schemas/ssh_key.py` from `api/ssh_keys.py` (any request/response schemas)
-- [ ] 1.10 Extract `schemas/git_repository.py` from `api/git_repositories.py` (any request/response schemas)
-- [ ] 1.11 Update all API routers to import schemas from `src.schemas.*` instead of defining inline
-- [ ] 1.12 Verify `py_compile` and `ruff` pass on all schema files
+- [ ] 0.1 Create `apps/api/src/schemas/tool/__init__.py`
+- [ ] 0.2 Create `apps/api/src/schemas/config/__init__.py`
+- [ ] 0.3 Create `apps/api/src/schemas/user/__init__.py`
+- [ ] 0.4 Create `apps/api/src/schemas/project/__init__.py`
+- [ ] 0.5 Create `apps/api/src/schemas/system/__init__.py`
+- [ ] 0.6 Create `apps/api/src/api/tool/__init__.py`
+- [ ] 0.7 Create `apps/api/src/api/config/__init__.py`
+- [ ] 0.8 Create `apps/api/src/api/workspace/__init__.py`
+- [ ] 0.9 Create `apps/api/src/api/user/__init__.py`
+- [ ] 0.10 Create `apps/api/src/api/project/__init__.py`
+- [ ] 0.11 Create `apps/api/src/api/system/__init__.py`
+- [ ] 0.12 Create `apps/api/src/services/instance/__init__.py`
+- [ ] 0.13 Create `apps/api/src/services/config/__init__.py`
+- [ ] 0.14 Create `apps/api/src/services/git/__init__.py`
+- [ ] 0.15 Create `apps/api/src/services/build/__init__.py`
+- [ ] 0.16 Create `apps/api/src/services/terminal/__init__.py`
+- [ ] 0.17 Create `apps/api/src/services/shared/__init__.py`
+- [ ] 0.18 Create `apps/api/src/models/tool/__init__.py`
+- [ ] 0.19 Create `apps/api/src/models/config/__init__.py`
+- [ ] 0.20 Create `apps/api/src/models/user/__init__.py`
+- [ ] 0.21 Create `apps/api/src/models/project/__init__.py`
+- [ ] 0.22 Create `apps/api/src/models/system/__init__.py`
 
-## 2. Docker Service Package Split
+## Phase 1: Model Subpackages
 
-- [ ] 2.1 Create `apps/api/src/services/docker/__init__.py` with re-exports for backward compatibility
-- [ ] 2.2 Create `apps/api/src/services/docker/compose.py` from `services/docker.py`:
-  - `render_compose_template`, `write_compose_file`, `_modify_compose_file`, `_ensure_container_name_in_compose`, `_ensure_web_bind_address`, `_ensure_backend_network_in_compose`, `_sanitize_compose_file`, `sort_volumes_by_specificity`
-- [ ] 2.3 Create `apps/api/src/services/docker/container.py` from `services/docker.py`:
-  - `get_container_status`, `get_container_logs`, `get_container_id`, `wait_for_container_running`, `get_container_ip_on_network`, `is_container_on_network`, `connect_container_to_network`, `get_backend_network_name`
-- [ ] 2.4 Create `apps/api/src/services/docker/config_staging.py` from `services/docker.py`:
-  - `write_env_file`, `write_config_files`, `ensure_instance_directory`
-- [ ] 2.5 Create `apps/api/src/services/docker/tunnel.py` from `services/tunnel.py`:
-  - `extract_tunnel_url` (move tunnel URL regex extraction here), `start_tunnel`, `stop_tunnel`, `check_tunnel_health`, `recreate_tunnel`
-- [ ] 2.6 Remove `apps/api/src/services/docker.py` after verifying all imports updated
-- [ ] 2.7 Update `services/tunnel.py` to delegate URL extraction to `docker/tunnel.py` or remove if fully subsumed
-- [ ] 2.8 Update all consumers (`api/tool_instances.py`, `services/instance_lifecycle.py`, etc.) to import from `services.docker` package
-- [ ] 2.9 Verify `py_compile` and `ruff` pass
+- [ ] 1.1 Move `models/tool_type.py` → `models/tool/tool_type.py`
+- [ ] 1.2 Move `models/tool_instance.py` → `models/tool/tool_instance.py`
+- [ ] 1.3 Move `models/tool_definition_manifest.py` → `models/tool/tool_definition_manifest.py`
+- [ ] 1.4 Move `models/config_profile.py` → `models/config/config_profile.py`
+- [ ] 1.5 Move `models/config_include.py` (if exists) → `models/config/config_include.py`
+- [ ] 1.6 Move `models/config_mount.py` (if exists) → `models/config/config_mount.py`
+- [ ] 1.7 Move `models/user.py` → `models/user/user.py`
+- [ ] 1.8 Move `models/user_config.py` → `models/user/user_config.py`
+- [ ] 1.9 Move `models/ssh_key.py` → `models/user/ssh_key.py`
+- [ ] 1.10 Move `models/project.py` → `models/project/project.py`
+- [ ] 1.11 Move `models/git_repository.py` → `models/project/git_repository.py`
+- [ ] 1.12 Move `models/workspace.py` → `models/project/workspace.py`
+- [ ] 1.13 Move `models/health_check.py` → `models/system/health_check.py`
+- [ ] 1.14 Move `models/notification.py` → `models/system/notification.py`
+- [ ] 1.15 Move `models/instance_event.py` → `models/system/instance_event.py`
+- [ ] 1.16 Move `models/terminal_session.py` → `models/system/terminal_session.py`
+- [ ] 1.17 Update `models/__init__.py` to import from subpackages
+- [ ] 1.18 Update all backend imports to use `models.tool.tool_type` etc.
+- [ ] 1.19 Verify `py_compile` and `ruff` pass
 
-## 3. Instance Lifecycle Extraction
+## Phase 2: Schema Extraction + Subpackages
 
-- [ ] 3.1 Create `apps/api/src/services/instance_lifecycle.py`:
-  - Extract `create_new_instance`, `start_existing_instance`, `stop_existing_instance`, `restart_existing_instance`, `delete_existing_instance` from `api/tool_instances.py`
-  - Extract helper functions: `_prepare_manifest_instance`, `_modify_compose_file`, `_ensure_container_name_in_compose`, `_ensure_web_bind_address`, `_ensure_backend_network_in_compose`, `_sanitize_compose_file`, `_resolve_git_mounts`, `_clone_git_repo`, etc.
-- [ ] 3.2 Thin `api/tool_instances.py` to ~300 lines:
-  - HTTP routing, auth validation, request parsing
-  - Delegate to `instance_lifecycle.py` service functions
-- [ ] 3.3 Move `sessions_router` from `api/tool_instances.py` to `api/users.py` or keep as separate `api/sessions.py` (align with `b6f89f9` pattern)
-- [ ] 3.4 Verify all instance endpoints (create, start, stop, restart, delete, list, get) still work
-- [ ] 3.5 Verify `py_compile` and `ruff` pass
+- [ ] 2.1 Extract `schemas/tool/tool_type.py` from `api/tool_types.py`
+- [ ] 2.2 Extract `schemas/tool/tool_instance.py` from `api/tool_instances.py`
+- [ ] 2.3 Extract `schemas/config/config_profile.py` from `api/config_profiles.py`
+- [ ] 2.4 Extract `schemas/system/health.py` from `api/health.py`
+- [ ] 2.5 Extract `schemas/user/user.py` from `api/users.py`
+- [ ] 2.6 Extract `schemas/user/user_config.py` from `api/user_config.py`
+- [ ] 2.7 Extract `schemas/project/project.py` from `api/projects.py`
+- [ ] 2.8 Extract `schemas/project/ssh_key.py` from `api/ssh_keys.py`
+- [ ] 2.9 Extract `schemas/project/git_repository.py` from `api/git_repositories.py`
+- [ ] 2.10 Update all API routers to import schemas from `src.schemas.*`
+- [ ] 2.11 Verify `py_compile` and `ruff` pass
 
-## 4. Config Profile Service Extraction
+## Phase 3: Docker Service Package Split
 
-- [ ] 4.1 Create `apps/api/src/services/config_profiles.py`:
-  - Extract `get_owned_profile`, `check_duplicate_name`, `profile_to_dict`, `_validate_default_profiles`, `get_default_profiles`, `set_default_profiles`, `get_default_profile_for_tool_type`, `get_or_create_user_config`, `list_includes_for_profile`, `list_mounts_for_profile` from `api/config_profiles.py`
-  - Add cycle detection helpers (`_detect_cycle`, `validate_includes_no_cycle`)
-- [ ] 4.2 Thin `api/config_profiles.py` to ~200 lines:
-  - HTTP routing, request parsing
-  - Delegate to `services/config_profiles.py`
-- [ ] 4.3 Verify all config profile endpoints (CRUD, includes, defaults, preview, validate-git-url) still work
-- [ ] 4.4 Verify `py_compile` and `ruff` pass
+- [ ] 3.1 Create `services/docker/__init__.py` with re-exports
+- [ ] 3.2 Create `services/docker/compose.py` from `services/docker.py`
+- [ ] 3.3 Create `services/docker/container.py` from `services/docker.py`
+- [ ] 3.4 Create `services/docker/config_staging.py` from `services/docker.py`
+- [ ] 3.5 Create `services/docker/tunnel.py` from `services/tunnel.py`
+- [ ] 3.6 Remove `services/docker.py` after verifying imports
+- [ ] 3.7 Update `services/tunnel.py` or remove if subsumed
+- [ ] 3.8 Update all consumers to import from `services.docker`
+- [ ] 3.9 Verify `py_compile` and `ruff` pass
 
-## 5. Auth Dependency Refactor
+## Phase 4: Service Subpackages
 
-- [ ] 5.1 Add `get_current_user` to `apps/api/src/auth/dependencies.py`:
-  - Decode session cookie, look up user in DB, return `User` model
-  - Raise 401 if missing/invalid session or user not found
-- [ ] 5.2 Migrate `api/users.py` to use `get_current_user` instead of `get_current_user_id` + `_get_user`
-- [ ] 5.3 Migrate `api/tool_instances.py` sessions_router to use `get_current_user` where appropriate
-- [ ] 5.4 Migrate other routers incrementally (dashboard, projects, etc.) where the full user object is needed
-- [ ] 5.5 Keep `get_current_user_id` for endpoints that only need the ID
-- [ ] 5.6 Verify `py_compile` and `ruff` pass
+- [ ] 4.1 Move `services/lifecycle_hooks.py` → `services/instance/lifecycle_hooks.py`
+- [ ] 4.2 Move `services/health_monitor.py` → `services/instance/health_monitor.py`
+- [ ] 4.3 Move `services/event_bus.py` → `services/instance/event_bus.py`
+- [ ] 4.4 Move `services/config_profile_resolver.py` → `services/config/config_profile_resolver.py`
+- [ ] 4.5 Move `services/clone.py` → `services/git/clone.py`
+- [ ] 4.6 Move `services/git_operations.py` → `services/git/git_operations.py`
+- [ ] 4.7 Move `services/git_service.py` → `services/git/git_service.py`
+- [ ] 4.8 Move `services/docker_build.py` → `services/build/docker_build.py`
+- [ ] 4.9 Move `services/manifest_compiler.py` → `services/build/manifest_compiler.py`
+- [ ] 4.10 Move `services/terminal_manager.py` → `services/terminal/terminal_manager.py`
+- [ ] 4.11 Move `services/terminal_session.py` → `services/terminal/terminal_session.py`
+- [ ] 4.12 Move `services/tunnel.py` → `services/shared/tunnel.py`
+- [ ] 4.13 Move `services/notification_service.py` → `services/shared/notification_service.py`
+- [ ] 4.14 Move `services/file_service.py` → `services/shared/file_service.py`
+- [ ] 4.15 Move `services/permission_fixer.py` → `services/shared/permission_fixer.py`
+- [ ] 4.16 Move `services/readiness_probe.py` → `services/shared/readiness_probe.py`
+- [ ] 4.17 Move `services/ssh_keys.py` → `services/shared/ssh_keys.py`
+- [ ] 4.18 Move `services/workspace_manager.py` → `services/shared/workspace_manager.py`
+- [ ] 4.19 Move `services/correlation.py` → `services/shared/correlation.py`
+- [ ] 4.20 Extract `services/instance/instance_lifecycle.py` from `api/tool_instances.py`
+- [ ] 4.21 Extract `services/config/config_profiles.py` from `api/config_profiles.py`
+- [ ] 4.22 Update all imports across the backend
+- [ ] 4.23 Verify `py_compile` and `ruff` pass
 
-## 6. Frontend Reorganization
+## Phase 5: API Router Subpackages
 
-- [ ] 6.1 Rename API files to kebab-case:
-  - `tool_types.ts` → `tool-types.ts`
-  - `ssh_keys.ts` → `ssh-keys.ts`
-  - `git_repositories.ts` → `git-repositories.ts`
-  - Update all imports in pages and components
-- [ ] 6.2 Rename page files to `*Page.tsx`:
-  - `dashboard.tsx` → `DashboardPage.tsx`
-  - `projects.tsx` → `ProjectsPage.tsx`
-  - `sessions.tsx` → `SessionsPage.tsx`
-  - `settings.tsx` → `SettingsPage.tsx`
-  - `ssh-keys.tsx` → `SshKeysPage.tsx`
-  - `terminal.tsx` → `TerminalPage.tsx`
-  - `tool-workshop.tsx` → `ToolWorkshopPage.tsx`
-  - `config-profiles.tsx` → `ConfigProfilesPage.tsx`
-  - `git-repositories.tsx` → `GitRepositoriesPage.tsx`
-  - `repo-workspace.tsx` → `RepoWorkspacePage.tsx`
-  - `workspaces.tsx` → `WorkspacesPage.tsx`
-  - `workspace-detail.tsx` → `WorkspaceDetailPage.tsx`
-  - `profile.tsx` → `ProfilePage.tsx`
-  - `project-settings.tsx` → `ProjectSettingsPage.tsx`
-  - `git-history.tsx` → `GitHistoryPage.tsx`
-  - Update `router.tsx` imports
-- [ ] 6.3 Move components into `features/` directories:
-  - Create `components/features/git/` and move git-related components
-  - Create `components/features/dashboard/` and move dashboard components
-  - Create `components/features/project/` and move project components
-  - Update all imports
-- [ ] 6.4 Verify `npm run typecheck` passes
-- [ ] 6.5 Verify `npm run build` passes
+- [ ] 5.1 Move `api/tool_instances.py` → `api/tool/tool_instances.py`
+- [ ] 5.2 Move `api/tool_types.py` → `api/tool/tool_types.py`
+- [ ] 5.3 Move `api/tool_definitions.py` → `api/tool/tool_definitions.py`
+- [ ] 5.4 Move `api/tool_types_validation.py` → `api/tool/tool_types_validation.py`
+- [ ] 5.5 Move sessions_router from `api/tool_instances.py` → `api/tool/sessions.py`
+- [ ] 5.6 Move `api/config_profiles.py` → `api/config/config_profiles.py`
+- [ ] 5.7 Move `api/user_config.py` → `api/config/user_config.py`
+- [ ] 5.8 Move `api/workspaces.py` → `api/workspace/workspaces.py`
+- [ ] 5.9 Move `api/workspace_files.py` → `api/workspace/workspace_files.py`
+- [ ] 5.10 Move `api/workspace_git.py` → `api/workspace/workspace_git.py`
+- [ ] 5.11 Move `api/workspace_instances.py` → `api/workspace/workspace_instances.py`
+- [ ] 5.12 Move `api/users.py` → `api/user/users.py`
+- [ ] 5.13 Move `api/auth.py` → `api/user/auth.py`
+- [ ] 5.14 Move `api/ssh_keys.py` → `api/user/ssh_keys.py`
+- [ ] 5.15 Move `api/projects.py` → `api/project/projects.py`
+- [ ] 5.16 Move `api/git_repositories.py` → `api/project/git_repositories.py`
+- [ ] 5.17 Move `api/health.py` → `api/system/health.py`
+- [ ] 5.18 Move `api/events.py` → `api/system/events.py`
+- [ ] 5.19 Move `api/notifications.py` → `api/system/notifications.py`
+- [ ] 5.20 Move `api/dashboard.py` → `api/system/dashboard.py`
+- [ ] 5.21 Move `api/terminal.py` → `api/system/terminal.py`
+- [ ] 5.22 Move `api/instance_proxy.py` → `api/system/instance_proxy.py`
+- [ ] 5.23 Update `main.py` to import from subpackages
+- [ ] 5.24 Update all cross-router imports
+- [ ] 5.25 Verify `py_compile` and `ruff` pass
 
-## 7. Integration and Verification
+## Phase 6: Auth Dependency Refactor
 
-- [ ] 7.1 Run backend tests: `docker exec hq-api pytest`
-- [ ] 7.2 Run backend typecheck: `pyright` or equivalent
-- [ ] 7.3 Run backend lint: `ruff check`
-- [ ] 7.4 Run frontend typecheck: `npm run typecheck`
-- [ ] 7.5 Run frontend build: `npm run build`
-- [ ] 7.6 Run `docker compose up --build` and verify API starts
-- [ ] 7.7 Verify key user flows manually:
-  - Create a tool instance
-  - Start/stop an instance
-  - Create a config profile
-  - Set a default config profile
-  - View sessions list
-  - Open terminal
-- [ ] 7.8 Verify no 404s or import errors in browser console
+- [ ] 6.1 Add `get_current_user` to `auth/dependencies.py`
+- [ ] 6.2 Migrate `api/user/users.py` to use `get_current_user`
+- [ ] 6.3 Migrate `api/tool/sessions.py` to use `get_current_user`
+- [ ] 6.4 Migrate other routers incrementally
+- [ ] 6.5 Verify `py_compile` and `ruff` pass
 
-## 8. Documentation
+## Phase 7: Frontend Reorganization
 
-- [ ] 8.1 Update `AGENTS.md` or backend README with new module structure
-- [ ] 8.2 Document the `get_current_user` vs `get_current_user_id` pattern for future contributors
+- [ ] 7.1 Rename API files to kebab-case
+- [ ] 7.2 Rename page files to `*Page.tsx`
+- [ ] 7.3 Move components into `features/` directories
+- [ ] 7.4 Update `router.tsx`
+- [ ] 7.5 Verify `npm run typecheck` passes
+- [ ] 7.6 Verify `npm run build` passes
+
+## Phase 8: Integration and Verification
+
+- [ ] 8.1 Run backend tests: `docker exec hq-api pytest`
+- [ ] 8.2 Run backend lint: `ruff check`
+- [ ] 8.3 Run frontend typecheck: `npm run typecheck`
+- [ ] 8.4 Run frontend build: `npm run build`
+- [ ] 8.5 Run `docker compose up --build` and verify API starts
+- [ ] 8.6 Verify key user flows manually
+- [ ] 8.7 Verify no 404s or import errors in browser console
+
+## Phase 9: Documentation
+
+- [ ] 9.1 Update `AGENTS.md` with new module structure
+- [ ] 9.2 Document `get_current_user` vs `get_current_user_id` pattern