diff --git a/docs/README.md b/docs/README.md index d2c565e..5fd9106 100644 --- a/docs/README.md +++ b/docs/README.md @@ -26,6 +26,7 @@ User guides for each feature: - [Repositories](features/repositories.md) - Git repository management - [Workspace](features/workspace.md) - Repository workspace - [Git History](features/git-history.md) - History visualization +- [Web Terminal](features/terminal.md) - Interactive terminal for tool instances - [Authentication](features/auth.md) - Login and user management - [Settings](features/settings.md) - User preferences - [Tool Types](features/tool-types.md) - Development tool management diff --git a/docs/architecture/backend.md b/docs/architecture/backend.md index db82501..58aaefe 100644 --- a/docs/architecture/backend.md +++ b/docs/architecture/backend.md @@ -13,16 +13,16 @@ The Headquarter backend is built with **FastAPI** and follows a layered architec │ Middleware: CORS → Request Logging → Exception Logging │ ├─────────────────────────────────────────────────────────────┤ │ API Layer (src/api/) │ -│ ┌─────────┐ ┌──────────┐ ┌────────┐ ┌──────────┐ │ -│ │ Auth │ │ Projects │ │ Users │ │ Git │ │ -│ │ Routes │ │ Routes │ │ Routes │ │ Repos │ │ -│ └────┬────┘ └────┬─────┘ └───┬────┘ └────┬─────┘ │ -├───────┼───────────┼───────────┼───────────┼─────────────────┤ -│ │ │ │ │ │ -│ Auth │ Project │ User │ Git │ │ -│ Layer │ Service │ Service │ Service │ │ -│ │ │ │ │ │ -├───────┴───────────┴───────────┴───────────┴─────────────────┤ +│ ┌─────────┐ ┌─────────┐ ┌────────┐ ┌──────────┐ │ +│ │ Auth │ │Terminal │ │Projects│ │ Git │ │ +│ │ Routes │ │ WS │ │ Routes │ │ Repos │ │ +│ └────┬────┘ └────┬────┘ └───┬────┘ └────┬─────┘ │ +├───────┼───────────┼──────────┼───────────┼──────────────────┤ +│ │ │ │ │ │ +│ Auth │ Terminal │ Project │ Git │ │ +│ Layer │ Manager │ Service │ Service │ │ +│ │ + Session│ │ │ │ +├───────┴───────────┴──────────┴───────────┴──────────────────┤ │ Data Layer │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ Models │ │ Database │ │ Config │ │ @@ -37,6 +37,7 @@ The Headquarter backend is built with **FastAPI** and follows a layered architec src/ ├── api/ # API Routes │ ├── auth.py # Authentication endpoints +│ ├── terminal.py # WebSocket terminal endpoint │ ├── projects.py # Project endpoints │ ├── git_repositories.py # Repository endpoints │ ├── users.py # User endpoints @@ -55,6 +56,11 @@ src/ │ ├── tool_type.py # Tool type model │ ├── ssh_key.py # SSH key model │ └── user_config.py # User config model +├── services/ # Business Logic +│ ├── terminal_manager.py # Terminal session manager +│ ├── terminal_session.py # PTY + docker exec session +│ ├── docker.py # Docker operations +│ └── profile_resolver.py # Profile resolution ├── utils/ # Utilities │ ├── git_url_parser.py # URL parsing │ ├── git_files.py # Git file operations @@ -64,6 +70,65 @@ src/ └── main.py # Application entry point ``` +## Terminal System + +The terminal system provides interactive shell access to running tool instances via WebSocket. + +### Architecture + +``` +Client (WebSocket) + ↕ +terminal.py (FastAPI WS endpoint) + ├─ Auth validation (session cookie) + ├─ Instance ownership check + ├─ Session lifecycle (create / monitor / cleanup) + └─ Echo state detection (termios) + ↕ +TerminalManager + ├─ create_session() → spawns TerminalSession + ├─ _read_loop() → batches PTY output → WebSocket + ├─ _write_loop() → WebSocket input → PTY + └─ _heartbeat_loop() → closes idle connections (60s) + ↕ +TerminalSession + ├─ start() → pty.openpty() + docker exec + ├─ read_output() → select.select() + os.read() + ├─ write_input() → os.write() to PTY master + ├─ resize() → TIOCSWINSZ ioctl + └─ check_echo_state() → termios.ECHO flag +``` + +### Protocol + +**Binary frames**: Raw terminal I/O (hot path) +**Text (JSON) frames**: Control messages + +**Control messages:** + +| Direction | Type | Purpose | +|-----------|------|---------| +| Client → Server | `ping` | Heartbeat (every 15s idle) | +| Server → Client | `pong` | Heartbeat response | +| Client → Server | `resize` | Terminal dimensions changed | +| Server → Client | `set_echo_state` | Enable/disable local echo | +| Server → Client | `session_ended` | Container process exited | + +### Message Batching + +The read loop batches small PTY reads into single WebSocket frames: +- Buffer accumulates data for up to 16ms +- Flushed immediately when no new data is available +- Reduces WebSocket frame overhead for rapid output + +### Reconnect Behavior + +The server cannot resume a `docker exec` PTY across connections. On reconnect: +1. Old session is terminated +2. New `docker exec` is spawned +3. Client restores scrollback from `sessionStorage` +4. New shell appears seamlessly to the user + ## Layers ### 1. API Layer (`src/api/`) diff --git a/docs/architecture/frontend.md b/docs/architecture/frontend.md index fab9dd9..f105a93 100644 --- a/docs/architecture/frontend.md +++ b/docs/architecture/frontend.md @@ -26,22 +26,26 @@ apps/web/src/ │ ├── ssh_keys.ts # SSH key API │ ├── tool_types.ts # Tool type API │ ├── users.ts # User API +│ ├── sessions.ts # Tool instance sessions API │ └── settings.ts # Settings API ├── components/ # Reusable components │ ├── app-shell.tsx # Main app layout +│ ├── terminal.tsx # xterm.js terminal component │ ├── protected-route.tsx # Auth guard │ └── [more...] ├── context/ # React contexts │ └── auth.tsx # Auth state management ├── hooks/ # Custom hooks │ ├── use-auth.ts # Auth hook -│ └── use-theme.ts # Theme hook +│ ├── use-theme.ts # Theme hook +│ └── use-terminal-connection.ts # Terminal WebSocket lifecycle ├── pages/ # Page components (routes) │ ├── dashboard.tsx # Dashboard │ ├── projects.tsx # Project list │ ├── repo-workspace.tsx # Repository workspace │ ├── git-history.tsx # Git history │ ├── git-repositories.tsx # Repository management +│ ├── terminal.tsx # Web terminal │ ├── profile.tsx # User profile │ ├── settings.tsx # User settings │ ├── tool-types.tsx # Tool types @@ -164,6 +168,7 @@ interface AuthState { } /> } /> } /> +} /> } /> } /> } /> @@ -269,12 +274,64 @@ test('renders file list', () => { 4. **Caching**: Browser caches API responses (ETags) 5. **Optimistic UI**: Immediate feedback before API response +## Terminal Architecture + +The web terminal is the most complex component in the frontend. It bridges a browser-based terminal emulator with a server-side PTY session. + +### Component Stack + +``` +TerminalPage (route) +└── TerminalComponent + ├── Status bar (connection state, latency, actions) + ├── Session-ended overlay (reconnect / go back) + ├── Reconnect banner (spinner + countdown) + └── xterm.js (terminal emulator) + ├── FitAddon (auto-resize to container) + ├── SerializeAddon (scrollback serialization) + └── WebLinksAddon (clickable URLs) +``` + +### Connection Hook + +`useTerminalConnection` manages the full WebSocket lifecycle: + +``` +CONNECTING + → onopen → CONNECTED → heartbeat every 15s + → onclose (unexpected) → RECONNECTING + → backoff: 1s → 2s → 4s → 8s → 16s → 30s max + → up to 10 attempts + → onopen → restore scrollback → CONNECTED + → onclose (expected) → DISCONNECTED +``` + +**Key behaviors:** +- **Local echo**: Printable ASCII chars appear instantly; server echo is deduplicated +- **Resize**: Debounced 200ms, throttled to 1 message per 500ms +- **Scrollback**: Serialized to `sessionStorage` on disconnect, restored on reconnect +- **Keyboard**: `Ctrl+Shift+R` triggers manual reconnect + +### Data Flow + +``` +User types 'a' + → xterm onData event + → useTerminalConnection.sendInput('a') + → local echo writes 'a' to xterm immediately + → WebSocket sends 'a' to server + → server PTY echoes 'a' back + → client receives 'a' via binary frame + → deduplicates against pending echo buffer + → (no-op if matched, or writes remaining chars) +``` + ## Future Improvements - [ ] Add React Query for server state management - [ ] Implement virtual scrolling for large file trees - [ ] Add service worker for offline support -- [ ] Implement real-time updates (WebSocket) +- [x] Implement real-time updates (WebSocket) — Terminal done - [ ] Add error boundary components ## Development Workflow diff --git a/docs/features/terminal.md b/docs/features/terminal.md new file mode 100644 index 0000000..d9fdf06 --- /dev/null +++ b/docs/features/terminal.md @@ -0,0 +1,216 @@ +# Web Terminal + +## Overview + +The web terminal provides an interactive shell session inside running tool instances directly from your browser. It uses xterm.js to render a full terminal emulator connected via WebSocket to a PTY-backed docker exec session. + +The terminal is designed to feel as close to a local terminal as possible, with features for network resilience, low-latency typing, and session continuity. + +## How to Use + +### Opening a Terminal + +1. Navigate to a **project** and select a **repository** +2. Go to the repository **workspace** +3. Start or select a **tool instance** that supports the terminal interface +4. Click the **"Open Terminal"** button + +The terminal opens in full-page mode with a status bar at the top. + +### Terminal Layout + +``` +┌─────────────────────────────────────────────┐ +│ ● Connected [Reconnect] [×] │ +├─────────────────────────────────────────────┤ +│ │ +│ user@container:~$ ls -la │ +│ total 128 │ +│ drwxr-xr-x 5 user user 4096 May 27 10:00 │ +│ ... │ +│ │ +└─────────────────────────────────────────────┘ +``` + +**Status bar (top):** +- **Connection dot** — color indicates connection health +- **Status text** — shows current state and latency +- **Reconnect button** — appears when disconnected +- **Close button** — returns to the previous page + +### Connection States + +| Indicator | Meaning | Action | +|-----------|---------|--------| +| 🟡 **Yellow dot** + "Connecting..." | Opening WebSocket | Wait or check network | +| 🟢 **Green dot** + "Connected" | Healthy connection (<100ms) | Ready to use | +| 🟡 **Yellow dot** + "Slow (150ms)" | Elevated latency | Connection usable but laggy | +| 🟡 **Yellow dot** + "Reconnecting (2)" | Connection lost, retrying | Wait for auto-reconnect | +| ⚪ **Gray dot** + "Disconnected" | Max retries exceeded | Click Reconnect or refresh | + +**Hover the status dot** to see the current round-trip latency in milliseconds. + +### Typing + +Type normally as you would in a local terminal. The terminal supports: + +- **Printable characters** appear instantly (local echo) +- **Special keys** (Tab, Enter, Ctrl+C, arrow keys) are sent to the server +- **Password prompts** automatically suppress local echo +- **Unicode** input and output + +### Reconnecting + +The terminal **automatically reconnects** if the WebSocket drops: + +- Brief disconnects (WiFi hiccups, proxy timeouts) are recovered within 1–5 seconds +- Up to **10 reconnection attempts** with exponential backoff +- **Scrollback is preserved** across reconnects +- A visual divider (`--- Reconnected ---`) separates old and new output + +**Manual reconnect:** +- Click the **Reconnect** button in the status bar +- Or press **Ctrl+Shift+R** anywhere in the terminal page + +### Session Ended + +When the container process exits (e.g., you run `exit` or the container stops), the terminal shows an overlay: + +``` +┌─────────────────────────┐ +│ Session Ended │ +│ The container process │ +│ has exited. │ +│ │ +│ [Reconnect] [Go Back] │ +└─────────────────────────┘ +``` + +- **Reconnect** — spawns a new shell session in the same container +- **Go Back** — returns to the workspace page + +## Keyboard Shortcuts + +| Shortcut | Action | +|----------|--------| +| `Ctrl+Shift+R` | Force reconnect (bypasses backoff) | +| Standard terminal shortcuts | `Ctrl+C`, `Ctrl+D`, `Ctrl+L`, Tab completion, etc. | + +## Technical Details + +### WebSocket Protocol + +The terminal communicates over a binary WebSocket with mixed JSON control messages. + +**Connection:** +``` +ws://api.example.com/ws/tool-instances/{instance_id}/terminal +``` + +**Binary frames** carry raw terminal I/O. **Text (JSON) frames** carry control messages: + +**Client → Server:** +- `{"type":"ping","id":n}` — heartbeat ping +- `{"type":"resize","cols":120,"rows":40}` — terminal resize +- Raw bytes — keystroke input + +**Server → Client:** +- `{"type":"pong","id":n}` — heartbeat response +- `{"type":"status","status":"connected"}` — session ready +- `{"type":"set_echo_state","enabled":false}` — disable local echo +- `{"type":"session_ended","reason":"process_exit"}` — session ended +- Raw bytes — terminal output + +### Architecture + +``` +Browser Backend +┌──────────────────────┐ ┌─────────────────────────────┐ +│ TerminalComponent │ │ terminal.py (WS endpoint) │ +│ ├─ xterm.js │◄───────►│ ├─ auth + session mgmt │ +│ ├─ FitAddon │ WS │ └─ echo state detection │ +│ ├─ SerializeAddon │ │ │ +│ └─ useTerminalConn. │ │ TerminalManager │ +│ ├─ heartbeat │ │ ├─ read_loop (batching) │ +│ ├─ reconnect │ │ ├─ write_loop │ +│ ├─ local echo │ │ └─ heartbeat_loop │ +│ └─ resize throttle│ │ │ +│ │ │ TerminalSession │ +│ sessionStorage │ │ ├─ PTY + docker exec │ +│ (scrollback backup) │ │ └─ termios echo detection │ +└──────────────────────┘ └─────────────────────────────┘ +``` + +### Reconnect Behavior + +On disconnect: +1. The client serializes terminal scrollback to `sessionStorage` +2. Backoff timer starts (1s, 2s, 4s, 8s, 16s, then caps at 30s) +3. On reconnect, scrollback is restored + divider line +4. A new `docker exec` session is spawned transparently + +**Note:** The underlying docker exec PTY is not resumable. Reconnect creates a new shell, but scrollback continuity makes this transparent. + +### Performance + +- **Local echo** makes printable characters appear in <1ms +- **Message batching** on the backend reduces WebSocket frame overhead +- **Resize debouncing** (200ms) + throttling (500ms) prevents server spam +- **Heartbeat interval** is 15s to balance detection speed with server load + +## Troubleshooting + +### "Connecting..." stays yellow + +**Issue:** WebSocket cannot open +**Check:** +1. Is the API server running? +2. Is the tool instance in "running" status? +3. Check browser console for connection errors +4. Verify the `VITE_API_BASE_URL` points to the correct API + +### "Reconnecting" loops forever + +**Issue:** Max reconnection attempts exceeded +**Check:** +1. Is the container still running? (`docker ps`) +2. Did the container crash or get stopped? +3. Check server logs for `Terminal session error` + +### Typing feels slow + +**Issue:** High latency or no local echo +**Check:** +1. Hover the status dot — latency >100ms is shown as "Slow" +2. Local echo only works for printable ASCII characters +3. Password prompts intentionally disable echo +4. Very high latency may indicate a congested network + +### Terminal is blank after reconnect + +**Issue:** Scrollback not restored +**Check:** +1. `sessionStorage` may have been cleared (new browser session) +2. The scrollback cap is 10,000 lines — very long sessions may truncate +3. Browser privacy settings may block `sessionStorage` + +### "Session Ended" immediately + +**Issue:** Container process exits right away +**Check:** +1. The container's default command may have finished +2. Check the tool type's Docker Compose template +3. Some tools (like one-off scripts) are not meant for persistent terminal sessions + +## Configuration + +No additional configuration is required. The terminal adapts automatically to: +- Browser window size (via ResizeObserver) +- System light/dark theme preference +- Network conditions (reconnect backoff) + +## Related Features + +- [Workspace](workspace.md) — Open the terminal from the repository workspace +- [Tool Types](tool-types.md) — Configure which tools expose a terminal interface +- [SSH Keys](ssh-keys.md) — Manage SSH keys for repository access from within the terminal