headquarter/openspec/changes/backend-frontend-refactoring/design.md

## Context

Current `dev` has all behavioral features from the overwritten `main` merge, but the code structure is pre-refactor:
- Monolithic `api/tool_instances.py` (~3000 lines)
- Monolithic `api/config_profiles.py` (~1000 lines)
- Monolithic `services/docker.py`
- No `schemas/` directory
- Flat frontend component structure with inconsistent naming

The `b6f89f9` merge from `main` had a clean refactoring that we need to redo, but adapted to our current reality.

## Goals / Non-Goals

**Goals:**
- Extract Pydantic schemas from API routers into `src/schemas/`
- Split `services/docker.py` into `services/docker/` package
- Extract instance lifecycle logic from `api/tool_instances.py` into `services/instance_lifecycle.py`
- Extract config profile business logic from `api/config_profiles.py` into `services/config_profiles.py`
- Add `get_current_user` auth dependency and migrate routers that need the full user object
- Reorganize frontend components into `features/` directories
- Standardize frontend API file naming to kebab-case
- Standardize frontend page naming to `*Page.tsx`

**Non-Goals:**
- Changing any API request/response shapes
- Changing any database schemas
- Adding new features
- Modifying frontend component behavior or styling
- Converting `ConfigProfile` mounts from JSON to relation tables (out of scope — would require migration)

## Decisions

### 0. Submodule Rule: Max 5–10 Files Per Directory
**Decision:** Every directory that functions as a Python module must contain at most 5–10 `.py` files. When a module grows beyond this, split it into a package with submodules.
**Rationale:** Prevents monolithic directories, makes navigation predictable, and keeps cognitive load bounded.

### 1. Schema Extraction: Domain Subpackages
**Decision:** Extract Pydantic models into `schemas/` subpackages by domain:
- `schemas/tool/` — tool_type.py, tool_instance.py
- `schemas/config/` — config_profile.py
- `schemas/user/` — user.py, user_config.py
- `schemas/project/` — project.py, git_repository.py, ssh_key.py
- `schemas/system/` — health.py
**Rationale:** Keeps schemas close to their domain. Each subpackage has ≤5 files.

### 2. API Router Subpackages
**Decision:** Split `api/` into domain subpackages:
- `api/tool/` — tool_instances.py, tool_types.py, tool_definitions.py, tool_types_validation.py, sessions.py
- `api/config/` — config_profiles.py, user_config.py
- `api/workspace/` — workspaces.py, workspace_files.py, workspace_git.py, workspace_instances.py
- `api/user/` — users.py, auth.py, ssh_keys.py
- `api/project/` — projects.py, git_repositories.py
- `api/system/` — health.py, events.py, notifications.py, dashboard.py, terminal.py, instance_proxy.py
**Rationale:** `api/` currently has ~22 files. Splitting into 6 subpackages keeps each at 2–6 files.

### 3. Service Subpackages
**Decision:** Split `services/` into subpackages:
- `services/docker/` — compose.py, container.py, config_staging.py, tunnel.py, __init__.py
- `services/instance/` — instance_lifecycle.py, lifecycle_hooks.py, health_monitor.py, event_bus.py
- `services/config/` — config_profile_resolver.py, config_profiles.py
- `services/git/` — clone.py, git_operations.py, git_service.py
- `services/build/` — docker_build.py, manifest_compiler.py
- `services/terminal/` — terminal_manager.py, terminal_session.py
- `services/shared/` — tunnel.py, notification_service.py, file_service.py, permission_fixer.py, readiness_probe.py, ssh_keys.py, workspace_manager.py, correlation.py
**Rationale:** `services/` currently has ~20 files. Subpackages keep each at ≤8 files.

### 4. Model Subpackages
**Decision:** Split `models/` into subpackages:
- `models/tool/` — tool_type.py, tool_instance.py, tool_definition_manifest.py
- `models/config/` — config_profile.py, config_include.py, config_mount.py
- `models/user/` — user.py, user_config.py, ssh_key.py
- `models/project/` — project.py, git_repository.py, workspace.py
- `models/system/` — health_check.py, notification.py, instance_event.py, terminal_session.py
- `models/base.py` stays at root
**Rationale:** `models/` currently has ~15 files. Subpackages keep each at ≤4 files.

### 5. Docker Service Split: Functional Boundaries
**Decision:** Split by responsibility:
- `compose.py` — compose file generation, modification, port injection, network injection
- `container.py` — container status, IP lookup, network connect, logs
- `config_staging.py` — staging config files into instance directories
- `tunnel.py` — extracting tunnel URLs from cloudflared output
**Rationale:** Each module has a single reason to change. `docker.py` mixed compose logic with container runtime queries.

### 6. Instance Lifecycle: Service Receives Raw Params, Not Request Objects
**Decision:** Service functions receive model instances and primitive parameters, not FastAPI request objects.
**Example:** `create_instance(session, user, project, repo, tool_type, data: CreateInstanceRequest)` → service extracts fields.
**Rationale:** Keeps service layer independent of HTTP framework. Easier to test.

### 7. Auth Pattern: Gradual Migration, Not Big Bang
**Decision:** Add `get_current_user` alongside existing `get_current_user_id`. Migrate routers incrementally.
**Rationale:** Reduces risk. Endpoints that only need the ID can keep the old pattern.

### 8. Frontend Naming: Align with `b6f89f9` Conventions
**Decision:** Use kebab-case for API files, PascalCase for page files with `Page` suffix, `features/` for component directories.
**Rationale:** Matches the `b6f89f9` structure that was already reviewed and accepted.

## Module Map

### Backend — Before
```
api/                        (~22 .py files)
  tool_instances.py         (~3000 lines) — HTTP + Docker + Git + Lifecycle
  config_profiles.py        (~1000 lines) — HTTP + Validation + Defaults
  tool_types.py             (~500 lines)  — HTTP + Schemas
  health.py                 (~150 lines)  — HTTP + Schemas
  users.py                  (~100 lines)  — HTTP + Schemas
  ...
services/                   (~20 .py files)
  docker.py                 (~600 lines)  — Compose + Container + Tunnel
models/                     (~15 .py files)
  config_profile.py
  tool_instance.py
  ...
```

### Backend — After
```
schemas/                    (5 subpackages, ≤5 files each)
  tool/
    __init__.py
    tool_type.py
    tool_instance.py
  config/
    __init__.py
    config_profile.py
  user/
    __init__.py
    user.py
    user_config.py
  project/
    __init__.py
    project.py
    git_repository.py
    ssh_key.py
  system/
    __init__.py
    health.py

api/                        (6 subpackages, 2–6 files each)
  tool/
    __init__.py
    tool_instances.py       (~300 lines)  — HTTP routing only
    tool_types.py           (~250 lines)  — HTTP + validation
    tool_definitions.py
    tool_types_validation.py
    sessions.py
  config/
    __init__.py
    config_profiles.py      (~200 lines)  — HTTP routing only
    user_config.py
  workspace/
    __init__.py
    workspaces.py
    workspace_files.py
    workspace_git.py
    workspace_instances.py
  user/
    __init__.py
    users.py                (~60 lines)   — HTTP only
    auth.py
    ssh_keys.py
  project/
    __init__.py
    projects.py
    git_repositories.py
  system/
    __init__.py
    health.py               (~80 lines)   — HTTP only
    events.py
    notifications.py
    dashboard.py
    terminal.py
    instance_proxy.py

services/                   (7 subpackages, ≤8 files each)
  docker/
    __init__.py             (~40 lines)   — Re-exports
    compose.py              (~240 lines)  — Compose generation
    container.py            (~120 lines)  — Container queries
    config_staging.py       (~80 lines)   — File staging
    tunnel.py               (~150 lines)  — Tunnel URL extraction
  instance/
    __init__.py
    instance_lifecycle.py   (~420 lines)  — Create/Start/Stop/Restart/Delete
    lifecycle_hooks.py
    health_monitor.py
    event_bus.py
  config/
    __init__.py
    config_profile_resolver.py
    config_profiles.py      (~300 lines)  — CRUD + Defaults
  git/
    __init__.py
    clone.py
    git_operations.py
    git_service.py
  build/
    __init__.py
    docker_build.py
    manifest_compiler.py
  terminal/
    __init__.py
    terminal_manager.py
    terminal_session.py
  shared/
    __init__.py
    tunnel.py
    notification_service.py
    file_service.py
    permission_fixer.py
    readiness_probe.py
    ssh_keys.py
    workspace_manager.py
    correlation.py

models/                     (5 subpackages + base.py)
  tool/
    __init__.py
    tool_type.py
    tool_instance.py
    tool_definition_manifest.py
  config/
    __init__.py
    config_profile.py
    config_include.py
    config_mount.py
  user/
    __init__.py
    user.py
    user_config.py
    ssh_key.py
  project/
    __init__.py
    project.py
    git_repository.py
    workspace.py
  system/
    __init__.py
    health_check.py
    notification.py
    instance_event.py
    terminal_session.py
  base.py                   (stays at root)
```

### Frontend — Before
```
src/
  api/
    tool_types.ts
    ssh_keys.ts
    git_repositories.ts
    sessions.ts
  components/
    git-toolbar.tsx
    file-editor.tsx
    commit-dialog.tsx
    ...
  pages/
    dashboard.tsx
    projects.tsx
    sessions.tsx
    ...
```

### Frontend — After
```
src/
  api/
    tool-types.ts
    ssh-keys.ts
    git-repositories.ts
    sessions.ts
  components/
    features/
      git/
        GitToolbar.tsx
        FileBrowser.tsx
        FileEditor.tsx
        CommitDialog.tsx
        MergeDialog.tsx
        WorkspaceSidebar.tsx
      dashboard/
        DashboardSummary.tsx
        ActiveSessionsList.tsx
        ProjectsSection.tsx
        QuickCreateForm.tsx
        RecentSessionsSection.tsx
      project/
        RepositoriesSettingsTab.tsx
      tool-workshop/
        ToolTypesTab.tsx
    ProtectedRoute.tsx
    AppShell.tsx
  pages/
    DashboardPage.tsx
    ProjectsPage.tsx
    SessionsPage.tsx
    ...
```

## Risks / Trade-offs

**[Risk] Import cycles during extraction** → **Mitigation:** Extract schemas first (no service dependencies), then services, then thin routers last. Use TYPE_CHECKING guards.

**[Risk] Merge conflicts with in-flight features** → **Mitigation:** Coordinate timing. This refactor should be the only large change on `dev` while it's in progress. Freeze other backend work.

**[Risk] Frontend renaming breaks imports** → **Mitigation:** Use `git mv` for renames so git tracks history. Update all imports in a single commit.

**[Risk] Missing re-export in docker/__init__.py breaks consumers** → **Mitigation:** After splitting, run a full import test across all backend files. Add any missing re-exports.

## Migration Plan

1. **Phase 1: Schemas** — Extract all Pydantic models into `schemas/`. Update imports in API routers. No logic changes.
2. **Phase 2: Services** — Split `docker.py`, extract `instance_lifecycle.py`, extract `config_profiles.py`. Update imports.
3. **Phase 3: Auth** — Add `get_current_user`, migrate routers that need the full user object.
4. **Phase 4: Frontend** — Rename files, move components, update imports.
5. **Phase 5: Verification** — Run full test suite, typecheck, build.