refactor: centralize types and extract seed data (Task 1.1)

- Create types/ directory with centralized domain types: session, tool-instance, tool-type, git-repository, config-folder, tool-config, project, user, api-response - Remove inline type definitions from API modules; re-export from types/ for backward compatibility - Update state/sessions.tsx to import Session from types/session.ts - Update all consumer components/pages to import from types/ - Extract seed_builtin_tool_types from main.py to seeds/builtin_tool_types.py - Create types/index.ts barrel export Quality gates: tsc (pass), eslint (pass), Python syntax (pass)
2026-06-02 18:56:54 +00:00
parent d894cd9723
commit ee1fa6bee5
36 changed files with 3003 additions and 435 deletions
@@ -0,0 +1,532 @@
+# Repo Restructure — Exploration Report
+
+**Project:** Headquarter (full-stack workspace platform)
+**Date:** 2026-06-02
+**Scope:** Comprehensive codebase audit for structural refactoring
+
+---
+
+## 1. Directory Structure
+
+### Root Layout
+```
+/workspace
+├── apps/
+│   ├── web/          # React 18 + Vite frontend
+│   └── api/          # Python FastAPI + SQLAlchemy backend
+├── e2e/              # Playwright tests
+├── docs/             # (not heavily populated)
+└── openspec/         # OpenSpec changes
+```
+
+### Frontend (`apps/web/src/`)
+```
+src/
+├── api/              # 13 API modules (~1,200 LOC total)
+│   ├── client.ts
+│   ├── dashboard.ts
+│   ├── git_repositories.ts
+│   ├── profile.ts
+│   ├── projects.ts
+│   ├── sessions.ts
+│   ├── settings.ts
+│   ├── ssh_keys.ts
+│   ├── terminal.ts
+│   ├── tool_configs.ts
+│   ├── tool_types.ts
+│   ├── config_folders.ts
+│   └── config_profiles.ts
+├── components/       # 16 components (~2,100 LOC total)
+│   ├── app-shell.tsx
+│   ├── code-editor.tsx
+│   ├── commit-dialog.tsx
+│   ├── commit-panel.tsx
+│   ├── file-editor.tsx
+│   ├── git-toolbar.tsx
+│   ├── icon.tsx
+│   ├── instance-list.tsx
+│   ├── merge-dialog.tsx
+│   ├── protected-route.tsx
+│   ├── protected-route.test.tsx
+│   ├── repository-create-dialog.tsx
+│   ├── settings-tab-layout.tsx
+│   ├── syntax-highlighter.tsx
+│   ├── workspace-header.tsx
+│   └── repositories-settings-tab.tsx
+├── hooks/            # 1 hook
+│   └── use-theme.ts
+├── pages/            # 15 pages (~3,500 LOC total)
+│   ├── dashboard.tsx
+│   ├── git-history.tsx
+│   ├── git-repositories.tsx
+│   ├── placeholder.tsx
+│   ├── profile.tsx
+│   ├── project-settings.tsx
+│   ├── projects.tsx
+│   ├── repo-workspace.tsx
+│   ├── settings.tsx
+│   ├── ssh-keys.tsx
+│   ├── terminal.tsx
+│   ├── tool-configs.tsx
+│   ├── tool-types.tsx
+│   └── tool-workshop.tsx
+├── state/            # 2 context providers
+│   ├── auth.tsx
+│   └── sessions.tsx
+├── types/            # 2 type modules
+│   └── terminal.ts
+├── utils/            # 3 utilities
+│   ├── icons.ts
+│   ├── language.ts
+│   └── terminal-protocol.ts
+├── styles.css        # 1 massive stylesheet (2,844 lines)
+├── router.tsx        # Route definitions
+├── main.tsx          # Entry point
+└── types.ts          # Shared domain types
+```
+
+### Backend (`apps/api/`)
+```
+apps/api/
+├── src/
+│   ├── main.py                 # App entry point (~287 lines)
+│   ├── config.py               # Pydantic settings (~128 lines)
+│   ├── database.py             # SQLAlchemy setup (~114 lines)
+│   ├── logging_config.py       # Middleware & logging (~92 lines)
+│   ├── auth/
+│   │   ├── session.py
+│   │   └── dependencies.py
+│   ├── api/                    # 15 routers
+│   │   ├── auth.py
+│   │   ├── config_folders.py
+│   │   ├── config_profiles.py
+│   │   ├── dashboard.py
+│   │   ├── git_repositories.py  # ~900+ lines
+│   │   ├── health.py
+│   │   ├── instance_proxy.py
+│   │   ├── projects.py
+│   │   ├── ssh_keys.py
+│   │   ├── terminal.py
+│   │   ├── tool_configs.py
+│   │   ├── tool_instances.py    # ~1,463 lines — CRITICAL
+│   │   ├── tool_types.py
+│   │   ├── user_config.py
+│   │   └── users.py
+│   ├── models/                 # SQLAlchemy models
+│   ├── services/               # Business logic
+│   │   ├── docker.py           # ~457+ lines
+│   │   ├── docker_build.py
+│   │   ├── git_control.py
+│   │   ├── git_files.py
+│   │   ├── git_history.py
+│   │   ├── git_url_parser.py
+│   │   ├── profile_resolver.py
+│   │   └── readiness_probe.py
+│   ├── utils/                  # Additional utilities
+│   └── scripts/
+│       └── seed.py
+├── alembic/versions/           # 14+ migrations
+└── tests/
+    ├── conftest.py
+    ├── unit/
+    └── integration/
+```
+
+---
+
+## 2. File Sizes — Files Over 200 Lines
+
+### 🔴 CRITICAL — Over 400 Lines (Must Split)
+
+| File | Lines | Issue |
+|------|-------|-------|
+| `apps/web/src/styles.css` | **2,844** | Single stylesheet for entire app; mixes layout, components, pages, syntax highlighting, and themes |
+| `apps/api/src/api/tool_instances.py` | **1,463** | Monolithic router: CRUD, Docker orchestration, tunneling, proxying, config resolution, readiness probes |
+| `apps/api/src/api/git_repositories.py` | **~900+** | Combined file browsing, Git control (branch/commit/merge/push/pull), URL parsing, history |
+| `apps/api/src/services/docker.py` | **~457+** | Docker compose, container management, tunneling, config folder staging all in one |
+
+### 🟡 WARNING — Over 200 Lines (Should Split)
+
+| File | Lines | Issue |
+|------|-------|-------|
+| `apps/web/src/pages/tool-workshop.tsx` | **~700+** | 3-tab admin page with inline forms for tool types, configs, AND folders |
+| `apps/web/src/pages/sessions.tsx` | **~668** | Sessions page with create form, active/recent lists, inline confirmations |
+| `apps/web/src/pages/repo-workspace.tsx` | **~394** | Page + FileBrowser component + mixed data loading |
+| `apps/web/src/components/instance-list.tsx` | **~388** | Instance CRUD + health checks + create dialog |
+| `apps/web/src/pages/tool-types.tsx` | **~380** | Tool types list + create/edit dialog inline |
+| `apps/web/src/pages/tool-configs.tsx` | **~354** | Tool configs list + create/edit dialog inline |
+| `apps/web/src/hooks/use-terminal-connection.ts` | **~439** | WS lifecycle, ping-pong, reconnection, local echo, resize debouncing |
+| `apps/web/src/pages/dashboard.tsx` | **~338** | Summary cards, session lists, quick-create form, recent sessions |
+| `apps/web/src/components/terminal.tsx` | **~309** | Terminal chrome + xterm lifecycle + resize observer |
+| `apps/web/src/pages/git-history.tsx` | **~233** | Commit list + detail panel with inline formatting |
+| `apps/web/src/api/git_repositories.ts` | **~245** | API functions + types (reasonable, but types should move) |
+| `apps/web/src/pages/projects.tsx` | **~206** | List + create dialog + delete confirmation |
+| `apps/api/src/main.py` | **~287** | Router registration + startup logic + seeding + error handlers |
+| `apps/api/src/api/config_profiles.py` | **~877** | Config profiles CRUD + complex resolution logic |
+| `apps/api/src/api/tool_types.py` | **~616** | Tool types CRUD + compose/dockerfile validation |
+| `apps/api/src/api/config_folders.py` | **~372** | Config folders CRUD |
+
+---
+
+## 3. Frontend Module Analysis
+
+### Components (16 files, ~2,100 LOC, avg ~131 LOC)
+**Too large:**
+- `git-toolbar.tsx` (~268) — mixes git ops, branch creation form, merge dialog trigger, status summary
+- `file-editor.tsx` (~241) — view/edit/commit workflow
+- `instance-list.tsx` (~388) — instance CRUD + health + create dialog
+- `terminal.tsx` (~309) — terminal chrome + xterm lifecycle
+
+**Well-sized:**
+- `workspace-header.tsx` (~48)
+- `protected-route.tsx` (~19)
+- `icon.tsx` (~165)
+
+### Pages (15 files, ~3,500 LOC, avg ~233 LOC)
+**All pages are too large.** Every page mixes:
+- Data fetching (useEffect + API calls)
+- Local state management (useState for forms, dialogs, loading)
+- UI rendering (JSX)
+
+**Worst offenders:**
+- `tool-workshop.tsx` (~700) — 3 completely different admin interfaces in one file
+- `sessions.tsx` (~668) — create form + active/recent lists + confirmations
+- `repo-workspace.tsx` (~394) — contains `FileBrowser` component inline
+- `dashboard.tsx` (~338) — summary, active sessions, projects list, quick-create form
+
+### Hooks (3 files)
+- `use-theme.ts` (~23) — fine
+- `use-terminal-connection.ts` (~439) — too large; mixes WS lifecycle, ping-pong, reconnection, echo, resize
+
+### API Modules (13 files, ~1,200 LOC)
+- Well-organized by domain
+- **Inconsistency:** Some define types inline (`api/sessions.ts` defines `ToolInstance`, `Session`), others in separate `types.ts`
+- `api/client.ts` — centralized Axios instance with auth interceptor. Good pattern.
+
+### State/Context (2 files)
+- `auth.tsx` (~63) — well-sized
+- `sessions.tsx` (~44) — well-sized
+
+### Styles (1 file, 2,844 lines) — CRITICAL
+**`styles.css` is the biggest problem in the frontend.** It contains:
+- CSS variables / design tokens
+- Global resets
+- Layout (shell, nav, content grid)
+- Page styles (home, settings, git-history, repo-workspace)
+- Component styles (cards, buttons, dialogs, forms, file-tree, editor)
+- Syntax highlighting overrides
+- Responsive media queries scattered throughout
+
+### Types
+- `src/types.ts` — core domain types (SessionUser, Project)
+- `src/types/terminal.ts` — terminal-specific WebSocket protocol types
+- **Problem:** API modules also export their own types (`ToolInstance`, `Session`, `GitRepository`, etc.) causing duplication and confusion. `Session` is defined in BOTH `api/sessions.ts` and `state/sessions.tsx`.
+
+### Utils
+- `icons.ts` (~180) — icon name mapping
+- `language.ts` (~90) — file extension → language detection
+- `terminal-protocol.ts` (~76) — WS message encoding/decoding + type guards
+
+### Router
+- `router.tsx` (~58) — clean and readable
+
+### Tests
+- `components/protected-route.test.tsx` (~49)
+- `api/tool_types.test.ts` (~227)
+- `api/config_folders.test.ts` (~131)
+- `pages/dashboard.test.tsx` (~81)
+- `pages/projects.test.tsx` (~174) — failing tests (React Router context issue)
+- `pages/tool-workshop.test.tsx` (~527)
+- `hooks/use-terminal-connection.test.ts` (~339)
+- **Massive gaps:** No tests for most pages, hooks, state providers, or components
+
+---
+
+## 4. Backend Module Analysis
+
+### Entry Points
+- `src/main.py` (~287) — FastAPI app setup, CORS, middleware, exception handlers, startup events, seeding, router mounting
+- **Problem:** Seed data (builtin tool types) is hardcoded here (~100 lines of compose templates). Should be in `seeds/` or `services/seed_data.py`.
+
+### Routers/Endpoints (15 files)
+**Organization:** One router per domain — good structure in theory, but files are too large.
+
+**`tool_instances.py` (1,463 lines)** — The worst offender. Contains:
+- Pydantic request/response models
+- Helper functions: `_modify_compose_file`, `_apply_resolved_profile`, `_get_user`, `_get_owned_project`, `_sanitize_name`, `_generate_instance_name`
+- Endpoints: create, list, get, start, stop, restart, delete, logs, recreate-tunnel, health-check, proxy
+- Inline Docker orchestration logic (should be in services)
+- Inline config resolution (should use service layer)
+
+**`git_repositories.py` (~900+ lines)** — Contains:
+- Repository CRUD
+- File browsing endpoints
+- Git control endpoints (branch, checkout, commit, fetch, pull, push, merge)
+- URL parsing endpoint
+
+**`config_profiles.py` (~877 lines)** — Contains:
+- Config profile CRUD
+- Complex profile resolution logic
+- Config folder/application logic
+
+### Models
+- Located in `src/models/` — one file per entity
+- Clean separation, well-sized
+
+### Services/Business Logic
+- `docker.py` (~456) — Docker compose, container ops, tunneling, config file staging. Too large.
+- `docker_build.py` (~69) — Image building
+- `git_control.py` (~295) — Git operations
+- `git_files.py` (~439) — File tree, read, write
+- `git_history.py` (~382) — Commit history, graph, diff
+- `git_url_parser.py` (~228) — URL parsing and validation
+- `profile_resolver.py` (~251) — Config profile resolution
+- `readiness_probe.py` (~66) — Container health probes
+- `terminal_manager.py` (~193) — Terminal session lifecycle
+- `terminal_session.py` (~162) — Individual terminal session handling
+
+### Database/ORM
+- `database.py` (~116) — Engine, session factory, init with alembic subprocess
+- `config.py` (~143) — Pydantic settings with env var resolution
+- Alembic migrations in `alembic/versions/` — 14+ migration files
+
+---
+
+## 5. Coupling and Dependency Patterns
+
+### Frontend High-Coupling Files
+
+**`repo-workspace.tsx`** imports from:
+- `react-router-dom` (params, search params)
+- `../api/client` (direct apiClient usage)
+- `../api/git_repositories`
+- `../components/commit-panel`
+- `../components/file-editor`
+- `../components/git-toolbar`
+- `../components/instance-list`
+- `../components/workspace-header`
+- `../api/tool_types`
+
+**`dashboard.tsx`** imports from:
+- `../api/dashboard`, `../api/sessions`, `../api/projects`, `../api/git_repositories`, `../api/tool_types`, `../api/settings`
+- `../types`, `../components/icon`
+
+**`tool-workshop.tsx`** imports from:
+- `../api/tool_types`, `../api/tool_configs`, `../api/config_folders`
+- Manages 3 separate entity forms with ~20 useState variables each
+
+### Circular Dependencies
+- **No obvious circular imports detected**, but `Session` type is duplicated between `api/sessions.ts` and `state/sessions.tsx`, creating conceptual circularity.
+
+### Business Logic Mixed with UI
+- **Every page component** contains API calls directly in `useEffect`
+- Form validation logic is inline in page components
+- `repo-workspace.tsx` defines `FileBrowser` as an inner component — cannot be tested or reused independently
+
+### API Call Patterns
+- **Mostly centralized** in `api/` modules — good
+- **Exception:** `repo-workspace.tsx`, `file-editor.tsx`, `project-settings.tsx` use `apiClient` directly instead of domain API modules
+- **Exception:** `app-shell.tsx` calls `getUserSessions()` directly
+
+---
+
+## 6. Naming Inconsistencies
+
+### File Naming Conventions
+
+| Location | Convention | Examples | Issues |
+|----------|-----------|----------|--------|
+| `pages/` | mostly kebab-case | `git-history.tsx`, `repo-workspace.tsx` | `projects.tsx`, `profile.tsx`, `settings.tsx`, `dashboard.tsx` are NOT kebab-case |
+| `components/` | kebab-case | `app-shell.tsx`, `protected-route.tsx` | `repositories-settings-tab.tsx` (long but consistent) |
+| `api/` | snake_case | `tool_configs.ts`, `git_repositories.ts` | Mixes with frontend convention |
+| `utils/` | kebab-case | `terminal-protocol.ts` | Good |
+| `hooks/` | camelCase | `useTheme.ts` would be standard, but file is `use-theme.ts` | Actually kebab-case, which is fine but inconsistent with React convention |
+| Backend routers | snake_case | `tool_instances.py`, `git_repositories.py` | Consistent within backend |
+| Backend services | snake_case | `docker.py`, `profile_resolver.py` | Consistent |
+
+### Component vs File Naming
+- Component `ProtectedRoute` → file `protected-route.tsx` ✅
+- Component `AppShell` → file `app-shell.tsx` ✅
+- Component `GitHistoryPage` → file `git-history.tsx` ❌ (should be `GitHistoryPage` in `git-history-page.tsx` OR component renamed to `GitHistory`)
+- Component `RepoWorkspace` → file `repo-workspace.tsx` ❌ (same issue)
+- Page components use `Page` suffix inconsistently: `ProjectsPage`, `GitHistoryPage`, but `RepoWorkspace` has no `Page` suffix
+
+### Function/Variable Naming
+- Frontend: camelCase consistently
+- Backend: snake_case consistently
+- **API types:** Backend uses `snake_case` fields; frontend types mirror this (`default_ssh_key_id`, `tool_type_name`). Good for API alignment.
+
+---
+
+## 7. Quality Signals
+
+### TODO/FIXME Comments
+- Only **2 TODOs** found:
+  - `apps/api/src/utils/git_history.py:188-189`: `# TODO: extract committer separately` (appears twice)
+
+This is surprisingly low — suggests either good maintenance or lack of inline documentation.
+
+### Dead Code / Unused Exports
+- `dashboard.tsx` exports `HomePage as DashboardPage` — dual naming is confusing
+- `src/types.ts` exports `SessionPayload` which is only used in auth context
+- Several CSS classes in `styles.css` may be unused (hard to verify without build analysis)
+
+### Duplicate Logic
+- **Backend auth checks:** `_get_user()` and `_get_owned_project()` are duplicated in nearly every router file (`tool_instances.py`, `git_repositories.py`, `ssh_keys.py`, etc.)
+- **Frontend loading/error patterns:** Identical `status: "loading" | "ready" | "error"` state + retry button pattern copied in ~8 page components
+- **Frontend form dialogs:** Create/edit/delete confirmation pattern repeated in `projects.tsx`, `tool-types.tsx`, `tool-configs.tsx`, `ssh-keys.tsx`
+
+### Test Coverage Gaps
+- **Frontend:** 7 test files, but many pages and components untested
+- **Backend:** Unit tests for `git_url_parser.py`, `migration_metadata.py`, `profile_resolver.py`, `readiness_probe.py`, `docker_build.py`, `terminal_manager.py`, `terminal_session.py`; integration tests via `conftest.py`
+- **E2E tests** only cover login flow (`e2e/tests/login.spec.ts`)
+
+---
+
+## Recommendations
+
+### Target Directory Structure
+
+#### Frontend (`apps/web/src/`)
+```
+src/
+├── api/                    # Keep — centralized API layer
+│   ├── client.ts
+│   ├── __mocks__/          # Add: mock API responses for tests
+│   └── {domain}/           # Group by domain
+│       ├── index.ts        # Re-exports
+│       ├── types.ts        # Domain types ONLY
+│       └── api.ts          # API functions
+├── components/             # Generic UI components
+│   ├── ui/                 # Primitive components (Button, Card, Dialog, Input)
+│   ├── layout/             # AppShell, Navigation, Header
+│   └── features/           # Domain-specific components
+│       ├── git/
+│       ├── project/
+│       ├── session/
+│       └── settings/
+├── hooks/                  # Custom hooks
+│   ├── use-theme.ts
+│   ├── use-auth.ts         # Extract from state/auth.tsx?
+│   └── use-api-query.ts    # NEW: reusable data fetching
+├── pages/                  # Route entry points ONLY
+│   ├── dashboard/
+│   │   └── page.tsx
+│   ├── projects/
+│   │   ├── page.tsx
+│   │   ├── project-list.tsx
+│   │   └── create-project-dialog.tsx
+│   └── ...
+├── state/                  # Keep contexts
+├── styles/
+│   ├── tokens.css          # CSS variables only
+│   ├── global.css          # Resets + base styles
+│   ├── components/         # Component styles
+│   └── pages/              # Page-specific styles
+├── types/                  # Centralize ALL shared types
+│   └── index.ts
+└── utils/
+```
+
+#### Backend (`apps/api/src/`)
+```
+src/
+├── main.py                 # Router mounting + middleware ONLY
+├── config.py
+├── database.py
+├── logging_config.py
+├── auth/
+├── api/
+│   └── v1/                 # Versioned routes
+│       ├── __init__.py
+│       ├── auth.py
+│       ├── projects/
+│       │   ├── __init__.py
+│       │   ├── router.py
+│       │   └── dependencies.py
+│       ├── repositories/
+│       │   ├── __init__.py
+│       │   ├── router.py     # CRUD only
+│       │   ├── files.py      # File browsing
+│       │   └── git.py        # Git control operations
+│       ├── instances/
+│       │   ├── __init__.py
+│       │   ├── router.py     # CRUD + lifecycle
+│       │   ├── compose.py    # Compose file generation
+│       │   ├── tunnel.py     # Cloudflare tunnel ops
+│       │   └── proxy.py      # HTTP proxy
+│       └── ...
+├── models/
+├── schemas/                # NEW: Pydantic schemas separate from routers
+├── services/
+│   ├── docker/
+│   │   ├── __init__.py
+│   │   ├── compose.py      # Extract from docker.py
+│   │   ├── container.py    # Container lifecycle
+│   │   ├── tunnel.py       # Cloudflare tunneling
+│   │   └── config.py       # Config file staging
+│   └── git/
+│       ├── control.py
+│       ├── files.py
+│       └── history.py
+├── seeds/                  # NEW: Seed data
+│   └── builtin_tool_types.py
+└── tests/
+```
+
+---
+
+### Files That MUST Be Split
+
+1. **`apps/web/src/styles.css`** → Split into 5-8 files by concern
+2. **`apps/api/src/api/tool_instances.py`** → Split into router + compose service + tunnel service + proxy service
+3. **`apps/api/src/api/git_repositories.py`** → Split into repository CRUD router + file router + git control router
+4. **`apps/api/src/services/docker.py`** → Split into compose, container, tunnel, config staging modules
+5. **`apps/web/src/pages/tool-workshop.tsx`** → Split into 3 page tabs or feature components
+6. **`apps/web/src/pages/repo-workspace.tsx`** → Extract `FileBrowser` to `components/features/git/file-browser.tsx`
+7. **`apps/web/src/pages/sessions.tsx`** → Extract create form, active list, recent list
+8. **`apps/web/src/hooks/use-terminal-connection.ts`** → Extract WS manager, echo handler, resize debouncer
+
+---
+
+### Naming Convention to Standardize On
+
+| Layer | Convention | Example |
+|-------|-----------|---------|
+| React components (files) | PascalCase matching component | `GitHistoryPage.tsx` |
+| React hooks (files) | camelCase | `useTheme.ts` |
+| Utility modules | kebab-case | `terminal-protocol.ts` |
+| API modules | kebab-case | `tool-configs.ts` |
+| Backend routers | snake_case | `tool_instances.py` |
+| Backend services | snake_case | `profile_resolver.py` |
+| CSS modules | kebab-case matching component | `git-history-page.module.css` |
+
+---
+
+### Order of Migration (First → Last)
+
+**Phase 1: Safe Foundations (low risk)**
+1. Extract shared types to `src/types/index.ts` (remove duplication)
+2. Create `src/hooks/use-api-query.ts` for reusable data fetching
+3. Extract `FileBrowser` from `repo-workspace.tsx`
+4. Move seed data from `main.py` to `seeds/builtin_tool_types.py`
+
+**Phase 2: Style System (medium risk, high reward)**
+5. Split `styles.css` into design tokens + component modules
+6. Introduce CSS modules or Tailwind utility extraction for component styles
+
+**Phase 3: Backend Decomposition (medium risk)**
+7. Extract `_get_user` and `_get_owned_project` to `auth/dependencies.py` or `api/dependencies.py`
+8. Split `tool_instances.py` into router + services
+9. Split `git_repositories.py` into CRUD + files + git control routers
+10. Split `services/docker.py` into focused modules
+
+**Phase 4: Frontend Page Decomposition (higher risk — touches UX)**
+11. Split `tool-workshop.tsx` into feature components
+12. Split `dashboard.tsx` into summary/session/project sections
+13. Split `sessions.tsx` into create-form + lists
+14. Split `settings.tsx` — move `GeneralSettingsTab` to its own file
+
+**Phase 5: Testing & Polish**
+15. Add tests for extracted components
+16. Add backend integration tests for refactored routers