feat: responsive web terminal with auto-reconnect, heartbeat, and local echo
Implements a resilient, responsive web terminal that survives network blips, provides instant typing feedback, and restores scrollback on reconnect. Backend changes: - Add heartbeat tracking (15s ping interval, 60s idle timeout) - Add message batching (16ms flush window) for efficient I/O - Add termios echo detection and set_echo_state control messages - Add graceful session_ended notification before close - Add ping/pong protocol support Frontend changes: - Rewrite TerminalComponent with status bar, connection indicator, session-ended overlay, reconnect banner, and ResizeObserver - Add useTerminalConnection hook with: - Exponential backoff auto-reconnect (1s → 30s max, 10 attempts) - Heartbeat/ping-pong with latency tracking - Local echo for printable ASCII with server deduplication - Resize debounce (200ms) + throttle (500ms) - Scrollback serialization via xterm-addon-serialize - Ctrl+Shift+R manual reconnect shortcut - Add WebSocket protocol types and encoding utilities - Add xterm-addon-serialize dependency Tests: - 16 backend unit tests (TerminalSession + TerminalManager) - 13 frontend hook tests (connection lifecycle, reconnect, resize, scrollback, callbacks) Quality gates: - Frontend typecheck: clean - Frontend lint: clean - Frontend tests: 48 passed - Backend unit tests: 101 passed - Backend ruff: clean SDD artifacts: openspec/changes/responsive-terminal/
This commit is contained in:
@@ -0,0 +1,371 @@
|
||||
# Design: Responsive Web Terminal
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ BROWSER │
|
||||
│ ┌──────────────┐ ┌─────────────────┐ ┌──────────────────────────┐ │
|
||||
│ │ TerminalPage │ │ TerminalComponent │ │ TerminalConnection │ │
|
||||
│ │ (router) │◄──│ (xterm.js + UI) │◄──│ (WS + heartbeat + echo) │ │
|
||||
│ └──────────────┘ └─────────────────┘ └──────────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌─────┴─────┐ ┌──────┴──────┐ │
|
||||
│ │ xterm.js │ │ sessionStorage│ │
|
||||
│ │ + addons │ │ (scrollback) │ │
|
||||
│ └───────────┘ └───────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
│ WebSocket
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ FASTAPI │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────────┐ │
|
||||
│ │ terminal.py │ │ TerminalManager │ │ TerminalSession │ │
|
||||
│ │ (WS endpoint) │◄──│ (session mgmt) │◄──│ (PTY + docker exec) │ │
|
||||
│ └──────────────────┘ └──────────────────┘ └─────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────┴────┐ │
|
||||
│ │ docker │ │
|
||||
│ │ exec │ │
|
||||
│ └─────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Connection State Machine
|
||||
|
||||
### Client State Machine
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ IDLE │
|
||||
└──────┬──────┘
|
||||
│ mount
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ CONNECTING │◄────────────────────────┐
|
||||
└──────┬──────┘ │
|
||||
│ onopen │
|
||||
▼ │
|
||||
┌─────────────────────────┐ │
|
||||
│ CONNECTED │ │
|
||||
│ (heartbeat active) │ │
|
||||
└──────┬──────────┬───────┘ │
|
||||
│ │ │
|
||||
onclose/ │ │ ping timeout │
|
||||
onerror │ │ │
|
||||
▼ ▼ │
|
||||
┌─────────────────────────┐ │
|
||||
│ RECONNECTING │───────────────────┘
|
||||
│ (backoff: 1→2→4→8→30s) │ onopen (success)
|
||||
└──────┬──────────────────┘
|
||||
│ max retries (10)
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ DISCONNECTED │
|
||||
│ (manual reconnect │
|
||||
│ or navigate away) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
### Server State Machine (per session)
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ PENDING │
|
||||
└──────┬──────┘
|
||||
│ ws.accept()
|
||||
▼
|
||||
┌─────────────┐
|
||||
┌────►│ ACTIVE │◄────┐
|
||||
│ │ (I/O loops │ │
|
||||
│ │ + heartbeat) │
|
||||
│ └──────┬──────┘ │
|
||||
│ │ │
|
||||
│ ws close│ new ws │
|
||||
│ ▼ │
|
||||
│ ┌─────────────┐ │
|
||||
└─────┤ CLOSED ├──────┘
|
||||
│ (cleanup) │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
## Protocol Specification
|
||||
|
||||
### Message Types
|
||||
|
||||
All control messages are JSON text frames. Raw terminal I/O uses binary frames.
|
||||
|
||||
#### Client → Server
|
||||
|
||||
| Type | Payload | When |
|
||||
|------|---------|------|
|
||||
| `ping` | `{ id: number }` | Every 15s of inactivity |
|
||||
| `pong` | `{ id: number }` | Response to server ping |
|
||||
| `resize` | `{ cols: number, rows: number }` | Terminal size changes (debounced) |
|
||||
| `input` | `{ data: string }` | User keystrokes (base64-encoded) |
|
||||
|
||||
#### Server → Client
|
||||
|
||||
| Type | Payload | When |
|
||||
|------|---------|------|
|
||||
| `pong` | `{ id: number }` | Response to client ping |
|
||||
| `status` | `{ status: "connected" \| "reconnected" }` | After auth + session ready |
|
||||
| `set_echo_state` | `{ enabled: boolean }` | When PTY echo flag changes |
|
||||
| `session_ended` | `{ reason: string }` | When container process exits |
|
||||
|
||||
### Binary Frame Convention
|
||||
|
||||
- **Client → Server:** Raw UTF-8 bytes of user input. No wrapping.
|
||||
- **Server → Client:** Raw bytes from PTY master read. No wrapping.
|
||||
|
||||
This avoids the current Blob→ArrayBuffer async conversion and JSON parsing overhead for the hot path.
|
||||
|
||||
## Frontend Design
|
||||
|
||||
### New Files
|
||||
|
||||
```
|
||||
apps/web/src/
|
||||
├── components/
|
||||
│ └── terminal.tsx (rewrite: state machine + reconnect)
|
||||
├── hooks/
|
||||
│ └── use-terminal-connection.ts (NEW: WS lifecycle, heartbeat, reconnect)
|
||||
├── utils/
|
||||
│ └── terminal-protocol.ts (NEW: message encoding/decoding)
|
||||
└── types/
|
||||
└── terminal.ts (NEW: protocol types)
|
||||
```
|
||||
|
||||
### `useTerminalConnection` Hook
|
||||
|
||||
Responsibilities:
|
||||
1. **WebSocket lifecycle:** Open, close, reconnect with backoff
|
||||
2. **Heartbeat:** Send ping every 15s, expect pong within 5s
|
||||
3. **Local echo:** Write printable chars to xterm immediately, deduplicate server echo
|
||||
4. **Resize:** Debounce resize events, send JSON control message
|
||||
5. **Scrollback:** Serialize on disconnect, restore on reconnect
|
||||
6. **State reporting:** Expose `status`, `latency`, `attempt` to UI
|
||||
|
||||
```typescript
|
||||
interface TerminalConnectionState {
|
||||
status: "connecting" | "connected" | "reconnecting" | "disconnected";
|
||||
attempt: number;
|
||||
latency: number | null; // last RTT in ms
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
interface TerminalConnection {
|
||||
state: TerminalConnectionState;
|
||||
sendInput: (data: string) => void;
|
||||
sendResize: (cols: number, rows: number) => void;
|
||||
reconnect: () => void; // manual, bypasses backoff
|
||||
onData: (callback: (data: Uint8Array) => void) => void;
|
||||
onControl: (callback: (msg: ServerControlMessage) => void) => void;
|
||||
}
|
||||
```
|
||||
|
||||
### Local Echo Algorithm
|
||||
|
||||
```
|
||||
1. User types character c
|
||||
2. IF c is printable ASCII AND echo is enabled:
|
||||
a. Write c to xterm immediately
|
||||
b. Add c to "pending echo" buffer
|
||||
c. Send c to server via WebSocket
|
||||
3. ELSE (control char, arrow, escape sequence):
|
||||
a. Send c to server only
|
||||
b. Do NOT write to xterm
|
||||
4. When server sends data:
|
||||
a. For each char in server data:
|
||||
- IF char matches head of "pending echo" buffer:
|
||||
→ Pop from buffer (deduplication)
|
||||
- ELSE:
|
||||
→ Write char to xterm
|
||||
b. If "pending echo" buffer grows > 100 chars (stale):
|
||||
→ Flush buffer to xterm (server echo was lost)
|
||||
```
|
||||
|
||||
### Scrollback Serialization
|
||||
|
||||
```
|
||||
ON disconnect:
|
||||
1. buffer = xterm.serialize({ scrollback: 10000 })
|
||||
2. sessionStorage.setItem(`hq-terminal-${instanceId}`, buffer)
|
||||
|
||||
ON reconnect:
|
||||
1. buffer = sessionStorage.getItem(`hq-terminal-${instanceId}`)
|
||||
2. IF buffer:
|
||||
xterm.write(buffer)
|
||||
xterm.write("\r\n\x1b[90m--- Reconnected ---\x1b[0m\r\n")
|
||||
3. sessionStorage.removeItem(`hq-terminal-${instanceId}`)
|
||||
```
|
||||
|
||||
### Resize Debouncing
|
||||
|
||||
Use `ResizeObserver` on the terminal container instead of `window.resize`:
|
||||
|
||||
```typescript
|
||||
const resizeObserver = new ResizeObserver(
|
||||
debounce((entries) => {
|
||||
fitAddon.fit();
|
||||
sendResize(term.cols, term.rows);
|
||||
}, 200)
|
||||
);
|
||||
```
|
||||
|
||||
Rate limit: max 1 resize message per 500ms.
|
||||
|
||||
## Backend Design
|
||||
|
||||
### Modified Files
|
||||
|
||||
```
|
||||
apps/api/src/
|
||||
├── api/terminal.py (modify: ping/pong, session_ended)
|
||||
├── services/terminal_manager.py (rewrite: heartbeat tracking, batching)
|
||||
└── services/terminal_session.py (modify: batching read, echo detection)
|
||||
```
|
||||
|
||||
### TerminalManager Changes
|
||||
|
||||
**Heartbeat tracking:**
|
||||
- Track `last_ping_at` per session
|
||||
- Background task: if `last_ping_at` is older than 60s, close the WebSocket
|
||||
|
||||
**Message batching in read_loop:**
|
||||
```python
|
||||
async def _read_loop(self, session, websocket):
|
||||
buffer = bytearray()
|
||||
last_flush = time.monotonic()
|
||||
|
||||
while session.is_alive() and not session._closed:
|
||||
data = await session.read_output()
|
||||
if data:
|
||||
buffer.extend(data)
|
||||
|
||||
now = time.monotonic()
|
||||
if buffer and (now - last_flush >= 0.016 or not data):
|
||||
await websocket.send_bytes(bytes(buffer))
|
||||
buffer.clear()
|
||||
last_flush = now
|
||||
elif not data:
|
||||
await asyncio.sleep(0.001)
|
||||
```
|
||||
|
||||
**Reconnect support:**
|
||||
- When a new WebSocket connects for the same instance, terminate the old session and spawn a new one
|
||||
- This is the docker exec limitation — we cannot resume a PTY, only replace it
|
||||
|
||||
### TerminalSession Changes
|
||||
|
||||
**Echo state detection:**
|
||||
```python
|
||||
import termios
|
||||
|
||||
def _detect_echo_state(self) -> bool:
|
||||
if self._master_fd is None:
|
||||
return True
|
||||
try:
|
||||
attrs = termios.tcgetattr(self._master_fd)
|
||||
return bool(attrs[3] & termios.ECHO)
|
||||
except:
|
||||
return True
|
||||
```
|
||||
|
||||
Call `_detect_echo_state()` after each resize and periodically (every 1s) during active I/O. Send `set_echo_state` to client when it changes.
|
||||
|
||||
**Batch-friendly read:**
|
||||
- Change `read_output()` to use `asyncio.wait_for(select, timeout)` instead of blocking `select.select` with 0.1s timeout
|
||||
- Return immediately when data is available, sleep briefly when not
|
||||
|
||||
### Terminal Endpoint Changes
|
||||
|
||||
- Accept `ping` messages, respond with `pong`
|
||||
- On session end (process exit), send `session_ended` before closing with code 1000
|
||||
- Distinguish between container exit (friendly) and error (unexpected)
|
||||
|
||||
## Data Flow: Typing with Local Echo
|
||||
|
||||
```
|
||||
User presses 'a'
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ onData handler │──► xterm.write('a') [instant feedback]
|
||||
│ │──► pendingEcho.push('a')
|
||||
│ │──► ws.send(binary 'a')
|
||||
└─────────────────┘
|
||||
│
|
||||
▼ (network)
|
||||
┌─────────────────┐
|
||||
│ TerminalSession │──► os.write(master_fd, b'a')
|
||||
│ │──► docker exec PTY echoes 'a' back
|
||||
│ │──► os.read(master_fd) → b'a'
|
||||
└─────────────────┘
|
||||
│
|
||||
▼ (WebSocket)
|
||||
┌─────────────────┐
|
||||
│ onMessage │──► data = b'a'
|
||||
│ (binary frame) │──► IF data[0] == pendingEcho[0]:
|
||||
│ │ pendingEcho.shift() // dedup
|
||||
│ │ ELSE:
|
||||
│ │ xterm.write(data)
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow: Reconnection
|
||||
|
||||
```
|
||||
WebSocket closes (code 1006)
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ ConnectionState │──► status = "reconnecting"
|
||||
│ │──► attempt = 1
|
||||
│ │──► scrollback = xterm.serialize()
|
||||
│ │──► sessionStorage.setItem(key, scrollback)
|
||||
│ │──► schedule reconnect in 1s
|
||||
└─────────────────┘
|
||||
│
|
||||
▼ (1s later)
|
||||
┌─────────────────┐
|
||||
│ Reconnect │──► new WebSocket(url)
|
||||
│ │──► onopen: send scrollback from storage
|
||||
│ │──► xterm.write(restored + divider)
|
||||
│ │──► status = "connected"
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Component Responsibilities
|
||||
|
||||
| Component | Responsibilities |
|
||||
|-----------|-----------------|
|
||||
| `TerminalPage` | Routing, layout, back button |
|
||||
| `TerminalComponent` | xterm.js lifecycle, addons, theme, status bar UI |
|
||||
| `useTerminalConnection` | WebSocket, heartbeat, reconnect, local echo, resize |
|
||||
| `terminal-protocol` | Encode/decode control messages, base64 helper |
|
||||
| `terminal.py` (API) | Auth, WebSocket accept, route control messages |
|
||||
| `TerminalManager` | Session lifecycle, heartbeat tracking, read/write loops |
|
||||
| `TerminalSession` | PTY + docker exec, echo detection, batching read |
|
||||
|
||||
## Tradeoffs
|
||||
|
||||
| Decision | Option A (Chosen) | Option B | Why A |
|
||||
|----------|-------------------|----------|-------|
|
||||
| **Reconnect strategy** | Exponential backoff, max 30s | Instant reconnect with no backoff | Backoff prevents server overload during outages |
|
||||
| **Local echo scope** | Printable ASCII only | All characters | Control chars/escapes need server-side processing (shell state) |
|
||||
| **Scrollback storage** | `sessionStorage` (tab-scoped) | `localStorage` (persistent) | Privacy: terminal may contain secrets |
|
||||
| **Scrollback cap** | 10,000 lines | Unlimited | Memory safety; 10K lines covers typical session |
|
||||
| **Heartbeat interval** | 15s client → server | 5s | Balance between detection speed and server load |
|
||||
| **Binary vs text I/O** | Binary frames for raw data | JSON-wrapped base64 | Binary is ~33% more efficient, zero parse overhead |
|
||||
| **Resize trigger** | ResizeObserver on container | window.resize | Container-level is more accurate for flex layouts |
|
||||
| **Echo detection** | Server inspects PTY termios | Client guesses from input | Server is authoritative; client cannot know shell state |
|
||||
| **New docker exec on reconnect** | Accept limitation | Implement persistent session | PTY resumption across connections is extremely complex; scrollback continuity is the pragmatic fix |
|
||||
|
||||
## Quality Gates
|
||||
|
||||
- `cd apps/web && npm run typecheck` — TypeScript compiles
|
||||
- `cd apps/web && npm run lint` — ESLint passes
|
||||
- `cd apps/web && npm test` — Vitest passes (new tests for protocol + hook)
|
||||
- `make test` — Backend pytest passes
|
||||
- Manual test: disconnect/reconnect, type latency, resize, container exit
|
||||
@@ -0,0 +1,59 @@
|
||||
# Explore: Responsive Web Terminal
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The current web terminal feels sluggish and fragile compared to a local terminal session. Key pain points:
|
||||
|
||||
1. **No reconnection** — A brief network hiccup kills the terminal. Users must navigate away and back.
|
||||
2. **No heartbeat** — Half-open connections stall silently. No way to know if the terminal is alive.
|
||||
3. **High input latency** — Every keystroke round-trips to the server before appearing on screen. No local echo.
|
||||
4. **Inefficient I/O path** — Backend `select` polling with 0.1s timeout, 4096-byte reads, busy-wait sleep(0.01). Frontend receives Blob and converts to ArrayBuffer asynchronously.
|
||||
5. **No scrollback persistence** — Reconnect starts with a blank terminal. Session history is lost.
|
||||
6. **Rudimentary resize** — Fires on every window resize event with no debouncing.
|
||||
7. **No connection quality feedback** — Binary status (connected/disconnected). No latency or health indicator.
|
||||
8. **No graceful container exit handling** — Process death closes WebSocket with a generic error.
|
||||
|
||||
## Current Architecture
|
||||
|
||||
### Frontend
|
||||
- `apps/web/src/components/terminal.tsx` — xterm.js v5.3.0 with FitAddon and WebLinksAddon
|
||||
- WebSocket to `/ws/tool-instances/{instance_id}/terminal`
|
||||
- Receives Blob (binary) and string (JSON control) messages
|
||||
- Sends raw bytes for input, JSON for resize
|
||||
- Basic status: connecting | connected | disconnected | error
|
||||
|
||||
### Backend
|
||||
- `apps/api/src/api/terminal.py` — FastAPI WebSocket endpoint, auth, session lifecycle
|
||||
- `apps/api/src/services/terminal_manager.py` — Manages TerminalSession, read/write loops
|
||||
- `apps/api/src/services/terminal_session.py` — PTY-based `docker exec` with `select` I/O
|
||||
- Protocol: raw bytes for terminal I/O, JSON for resize control messages
|
||||
|
||||
### Gaps vs. Local Terminal Feel
|
||||
|
||||
| Aspect | Local Terminal | Current Web Terminal |
|
||||
|--------|---------------|----------------------|
|
||||
| Keystroke feedback | Immediate (kernel TTY) | Round-trip (~50-200ms) |
|
||||
| Network resilience | N/A (local) | Dies on any disconnect |
|
||||
| Scrollback | Persistent | Lost on reconnect |
|
||||
| Resize | Instant | Undebounced, may spam |
|
||||
| Health visibility | Always local | Binary connected/disconnected |
|
||||
| Large output | Buffered by kernel | Select polling, 4KB chunks |
|
||||
|
||||
## Opportunities
|
||||
|
||||
- **WebSocket reconnection with exponential backoff** and session token for continuity
|
||||
- **Heartbeat/ping-pong** to detect half-open connections within seconds
|
||||
- **Local echo optimization** for printable characters (with server-side authoritative sync)
|
||||
- **Message batching** on backend to reduce WebSocket frame overhead
|
||||
- **Scrollback serialization** via xterm-addon-serialize to restore on reconnect
|
||||
- **Resize debouncing** to avoid flooding the server
|
||||
- **Connection quality indicator** (latency, jitter) in the terminal chrome
|
||||
- **Graceful handling** of container exit with clear user messaging
|
||||
|
||||
## Risks
|
||||
|
||||
- Adding heartbeat may increase server load with many concurrent terminals
|
||||
- Local echo requires careful handling of password prompts and special modes
|
||||
- Reconnecting to a docker exec PTY is not natively resumable — new `docker exec` on reconnect
|
||||
- xterm-addon-serialize may be large for very long sessions
|
||||
- Changes touch both frontend and backend — cross-stack coordination needed
|
||||
@@ -0,0 +1,77 @@
|
||||
# Proposal: Responsive Web Terminal
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The web terminal in Headquarter feels sluggish and fragile compared to a local terminal session. Users experience high input latency (every keystroke round-trips to the server before appearing), lose their session on any network blip, and have no visibility into connection health. This makes the terminal the weakest part of the workspace experience, especially for users on slower or unstable networks.
|
||||
|
||||
## User Stories
|
||||
|
||||
### US-1: Network Resilience
|
||||
> As a developer working on a laptop with WiFi,
|
||||
> I want the terminal to survive brief disconnections (up to ~30 seconds),
|
||||
> so that a network hiccup does not kill my running process and scrollback.
|
||||
|
||||
### US-2: Responsive Typing
|
||||
> As a developer typing commands or code in the terminal,
|
||||
> I want keystrokes to appear on screen instantly,
|
||||
> so that the terminal feels like a local TTY and not a remote typewriter.
|
||||
|
||||
### US-3: Session Continuity
|
||||
> As a developer who accidentally refreshed the page,
|
||||
> I want my terminal scrollback and state to be restored on reconnect,
|
||||
> so that I do not lose context of what I was doing.
|
||||
|
||||
### US-4: Connection Health Visibility
|
||||
> As a developer on a slow or congested network,
|
||||
> I want to see clear feedback about connection quality and reconnection attempts,
|
||||
> so that I understand whether lag is from the server, the container, or my network.
|
||||
|
||||
### US-5: Graceful Container Exit
|
||||
> As a developer whose container process has finished,
|
||||
> I want to see a clear message explaining what happened and options to reconnect or go back,
|
||||
> so that I am not confused by a generic "Connection closed" error.
|
||||
|
||||
## Success Metrics
|
||||
|
||||
| Metric | Current | Target |
|
||||
|--------|---------|--------|
|
||||
| Time-to-reconnect after disconnect | ∞ (must navigate away) | < 5 seconds |
|
||||
| Typing latency (median) | ~100-300ms | < 50ms perceived |
|
||||
| Scrollback lost on reconnect | 100% | 0% (restored from serialization) |
|
||||
| Silent connection stalls detected | 0% | 100% within 10 seconds |
|
||||
| User confusion on container exit | High | Low (clear messaging) |
|
||||
|
||||
## Scope
|
||||
|
||||
### In Scope
|
||||
- WebSocket auto-reconnection with exponential backoff
|
||||
- Heartbeat/ping-pong protocol between client and server
|
||||
- Local echo for printable characters (with server authoritative sync)
|
||||
- Resize debouncing to avoid server spam
|
||||
- Scrollback serialization via xterm-addon-serialize on disconnect
|
||||
- Scrollback restoration on reconnect
|
||||
- Connection quality indicator (latency, status) in terminal chrome
|
||||
- Graceful container exit handling with user-friendly messaging
|
||||
- Backend message batching for large output bursts
|
||||
|
||||
### Out of Scope (for this change)
|
||||
- Full terminal session recording/playback
|
||||
- Multi-user collaborative terminal sessions
|
||||
- Terminal session persistence across server restarts
|
||||
- Clipboard integration improvements (separate feature)
|
||||
- Terminal search/find (separate feature)
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|------|-----------|--------|------------|
|
||||
| Heartbeat increases server load with many terminals | Medium | Medium | Use 15s heartbeat interval; skip during idle periods |
|
||||
| Local echo breaks password prompts | Medium | High | Disable local echo when terminal is in "no echo" mode; server sends echo-state control messages |
|
||||
| Scrollback serialization is large for long sessions | Low | Medium | Cap serialization at 10,000 lines; compress before send |
|
||||
| Reconnect spawns new docker exec = new shell | Certain | Low | Accept as limitation; focus on scrollback continuity and clear messaging |
|
||||
| Cross-stack changes introduce regressions | Medium | High | Comprehensive test coverage; fresh review before merge |
|
||||
|
||||
## Approval
|
||||
|
||||
- [ ] Approved
|
||||
- [ ] Needs revision
|
||||
@@ -0,0 +1,153 @@
|
||||
# Spec: Responsive Web Terminal
|
||||
|
||||
## Overview
|
||||
|
||||
Upgrade the web terminal from a fragile single-shot WebSocket into a resilient, responsive terminal that survives network blips, provides instant typing feedback, restores scrollback on reconnect, and gives users clear visibility into connection health.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### AC-1: WebSocket Auto-Reconnection
|
||||
|
||||
**GIVEN** a terminal is connected to a running instance
|
||||
**WHEN** the WebSocket disconnects (network hiccup, server restart, proxy timeout)
|
||||
**THEN** the client automatically reconnects with exponential backoff (1s, 2s, 4s, 8s, max 30s)
|
||||
**AND** the user sees a reconnection indicator showing attempt count and next retry time
|
||||
**AND** after successful reconnection, the terminal scrollback is restored
|
||||
**AND** a new `docker exec` session is spawned transparently
|
||||
|
||||
**Test:** Disconnect WiFi for 5s, verify reconnect and scrollback intact.
|
||||
|
||||
### AC-2: Heartbeat / Ping-Pong Protocol
|
||||
|
||||
**GIVEN** a terminal connection is established
|
||||
**WHEN** 15 seconds pass with no data exchanged
|
||||
**THEN** the client sends a `ping` control message
|
||||
**AND** the server responds with a `pong` within 5 seconds
|
||||
**AND** if no `pong` is received within 5 seconds, the client treats the connection as dead and begins reconnection
|
||||
**AND** the server closes WebSockets that have not sent any message (including ping) for 60 seconds
|
||||
|
||||
**Test:** Block server responses with firewall rule, verify connection declared dead within 20s and reconnection starts.
|
||||
|
||||
### AC-3: Local Echo for Reduced Typing Latency
|
||||
|
||||
**GIVEN** the terminal is in a normal interactive shell
|
||||
**WHEN** the user types printable ASCII characters
|
||||
**THEN** they appear on screen immediately (local echo) without waiting for the server round-trip
|
||||
**AND** when the server sends the authoritative echo back, the client reconciles (deduplicates)
|
||||
**AND** when the server sends a `set_echo_state` control message with `enabled: false` (e.g., for password prompts), local echo is disabled
|
||||
**AND** when `set_echo_state` with `enabled: true` is received, local echo is re-enabled
|
||||
|
||||
**Test:** Type `echo hello` — characters appear instantly. Run `sudo` — local echo stops during password prompt.
|
||||
|
||||
### AC-4: Resize Debouncing
|
||||
|
||||
**GIVEN** the user is resizing the browser window
|
||||
**WHEN** the terminal dimensions change
|
||||
**THEN** resize events are debounced by 200ms
|
||||
**AND** only the final dimensions after the user stops resizing are sent to the server
|
||||
**AND** at most one resize message is sent per 500ms
|
||||
|
||||
**Test:** Rapidly resize window 10 times in 1s — verify only 1-2 resize messages sent.
|
||||
|
||||
### AC-5: Scrollback Serialization and Restoration
|
||||
|
||||
**GIVEN** a terminal has been in use with output history
|
||||
**WHEN** a disconnect occurs
|
||||
**THEN** the client serializes the terminal buffer (via xterm-addon-serialize, capped at 10,000 lines)
|
||||
**AND** stores it in `sessionStorage` under key `hq-terminal-{instance_id}`
|
||||
**AND** on successful reconnection, the serialized content is written back into the terminal before new output
|
||||
**AND** a visual divider line indicates "--- Reconnected ---" between old and new output
|
||||
|
||||
**Test:** Run `ls -la` 50 times, disconnect, reconnect — verify all output visible with divider.
|
||||
|
||||
### AC-6: Connection Quality Indicator
|
||||
|
||||
**GIVEN** the terminal is connected
|
||||
**THEN** the status bar shows:
|
||||
- Green dot + "Connected" when healthy (latency < 100ms)
|
||||
- Yellow dot + "Slow" when latency is 100-500ms
|
||||
- Red dot + "Reconnecting (N)" during reconnection attempts
|
||||
- Gray dot + "Disconnected" when permanently disconnected (max retries exceeded)
|
||||
**AND** hovering the status dot shows a tooltip with round-trip latency (ms) and jitter
|
||||
**AND** the indicator updates every 5 seconds
|
||||
|
||||
**Test:** Use network throttling in dev tools to simulate slow connection, verify indicator changes.
|
||||
|
||||
### AC-7: Graceful Container Exit
|
||||
|
||||
**GIVEN** a terminal session is active
|
||||
**WHEN** the container process exits (shell terminates, container stops)
|
||||
**THEN** the terminal shows a clear message: "Session ended. The container process has exited."
|
||||
**AND** a "Reconnect" button is shown to spawn a new session
|
||||
**AND** a "Go Back" button navigates to the previous page
|
||||
**AND** the WebSocket closes with code 1000 (normal) instead of an error code
|
||||
|
||||
**Test:** Run `exit` in the terminal, verify friendly message and buttons appear.
|
||||
|
||||
### AC-8: Backend Message Batching
|
||||
|
||||
**GIVEN** a container process is producing output rapidly
|
||||
**WHEN** the backend PTY produces multiple small reads within a single event loop tick
|
||||
**THEN** the backend batches them into a single WebSocket binary frame
|
||||
**AND** batching does not add more than 16ms of latency
|
||||
**AND** the batch is flushed immediately when no new data is available
|
||||
|
||||
**Test:** Run `yes | head -n 10000` and measure WebSocket frame count vs. current implementation.
|
||||
|
||||
### AC-9: Keyboard Shortcut for Reconnect
|
||||
|
||||
**GIVEN** the terminal is disconnected
|
||||
**WHEN** the user presses `Ctrl+Shift+R`
|
||||
**THEN** an immediate reconnection attempt is triggered (bypassing backoff)
|
||||
|
||||
**Test:** Disconnect terminal, press `Ctrl+Shift+R`, verify immediate reconnect attempt.
|
||||
|
||||
## API / Protocol Changes
|
||||
|
||||
### WebSocket Control Messages (JSON)
|
||||
|
||||
```typescript
|
||||
// Client → Server
|
||||
type ClientMessage =
|
||||
| { type: "ping"; id: number }
|
||||
| { type: "pong"; id: number }
|
||||
| { type: "resize"; cols: number; rows: number }
|
||||
| { type: "input"; data: string } // base64-encoded bytes
|
||||
|
||||
// Server → Client
|
||||
type ServerMessage =
|
||||
| { type: "pong"; id: number }
|
||||
| { type: "status"; status: "connected" | "reconnected" }
|
||||
| { type: "set_echo_state"; enabled: boolean }
|
||||
| { type: "session_ended"; reason: "process_exit" | "container_stop" | "timeout" }
|
||||
```
|
||||
|
||||
### Binary Frames
|
||||
|
||||
- Raw terminal output from server → client: binary WebSocket frame (no wrapping)
|
||||
- Raw terminal input from client → server: binary WebSocket frame (no wrapping)
|
||||
- Control messages (resize, ping, etc.): text JSON frames
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Frontend
|
||||
- `xterm-addon-serialize` — scrollback serialization
|
||||
- `xterm-addon-webgl` (optional) — GPU rendering for smoother feel
|
||||
|
||||
### Backend
|
||||
- No new Python dependencies required
|
||||
- Uses existing `asyncio`, `fastapi`, `websockets`
|
||||
|
||||
## Non-Functional Requirements
|
||||
|
||||
- **Latency:** Perceived typing latency < 50ms for local echo characters
|
||||
- **Reconnection time:** < 5 seconds for transient disconnects
|
||||
- **Memory:** Scrollback serialization capped at 10,000 lines (~2-5MB worst case)
|
||||
- **Server load:** Heartbeat interval 15s; max 4 pings/minute per terminal
|
||||
- **Browser support:** Chrome 90+, Firefox 88+, Safari 14+ (all support required WebSocket features)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should we add a "full screen" button to the terminal chrome? (Nice-to-have, out of scope for this change)
|
||||
2. Should scrollback be persisted across full page reloads (via `localStorage`) or only during session (`sessionStorage`)? — **Decision:** Use `sessionStorage` to avoid leaking sensitive data.
|
||||
3. Should the server echo-state detection be automatic (TIOCGWINSZ / stty inspection) or manual (client tells server)? — **Decision:** Server detects via PTY state inspection; sends `set_echo_state` to client.
|
||||
@@ -0,0 +1,213 @@
|
||||
# Tasks: Responsive Web Terminal
|
||||
|
||||
## Review Workload Forecast
|
||||
|
||||
| Task | Estimated Lines | Stack | Risk |
|
||||
|------|----------------|-------|------|
|
||||
| T1: Protocol types + utilities | ~120 | Frontend | Low |
|
||||
| T2: Backend heartbeat + batching | ~200 | Backend | Medium |
|
||||
| T3: Backend echo detection + graceful exit | ~150 | Backend | Medium |
|
||||
| T4: useTerminalConnection hook | ~280 | Frontend | High |
|
||||
| T5: TerminalComponent rewrite | ~250 | Frontend | High |
|
||||
| T6: Frontend tests | ~180 | Frontend | Low |
|
||||
| T7: Backend tests | ~120 | Backend | Low |
|
||||
| **Total** | **~1,300** | | |
|
||||
|
||||
**Review recommendation:** This exceeds the 400-line budget. Split into **3 chained PRs**:
|
||||
1. **PR-1 (Backend foundation):** T1 protocol types + T2 heartbeat/batching + T3 echo/exit + T7 backend tests (~590 lines)
|
||||
2. **PR-2 (Frontend connection):** T4 useTerminalConnection hook + T6 frontend hook tests (~460 lines)
|
||||
3. **PR-3 (Terminal UI + integration):** T5 TerminalComponent rewrite + page integration + remaining tests (~250 lines)
|
||||
|
||||
---
|
||||
|
||||
## Task T1: Protocol Types and Utilities
|
||||
|
||||
**Files:**
|
||||
- `apps/web/src/types/terminal.ts` (new)
|
||||
- `apps/web/src/utils/terminal-protocol.ts` (new)
|
||||
- `apps/web/package.json` (add `xterm-addon-serialize`)
|
||||
|
||||
**Description:**
|
||||
Define TypeScript types for all WebSocket control messages. Implement encode/decode helpers that distinguish binary frames (raw terminal I/O) from JSON text frames (control messages). Add base64 encoding for the `input` control message type. Install `xterm-addon-serialize` dependency.
|
||||
|
||||
**Acceptance:**
|
||||
- All message types from the design spec are represented as TypeScript types
|
||||
- `encodeControlMessage` and `decodeControlMessage` functions handle JSON serialization
|
||||
- `isControlMessage` helper correctly identifies text vs binary frames
|
||||
- `npm install` completes without lockfile conflicts
|
||||
|
||||
**Depends on:** None
|
||||
**Estimated:** 2 hours
|
||||
|
||||
---
|
||||
|
||||
## Task T2: Backend Heartbeat and Message Batching
|
||||
|
||||
**Files:**
|
||||
- `apps/api/src/services/terminal_manager.py`
|
||||
- `apps/api/src/api/terminal.py`
|
||||
|
||||
**Description:**
|
||||
Rewrite `TerminalManager` read loop to batch small reads into single WebSocket frames (max 16ms buffering). Add heartbeat tracking: server records `last_client_message_at` timestamp, and a background task closes WebSockets idle for 60s. Update `terminal.py` endpoint to accept `ping` control messages and respond with `pong`. Handle binary input frames (not just text JSON).
|
||||
|
||||
**Acceptance:**
|
||||
- Backend sends batched binary frames; `yes | head -n 10000` produces fewer WebSocket frames than before
|
||||
- Server responds to `ping` with matching `pong` within 100ms
|
||||
- Server closes idle connections after 60s of no client messages
|
||||
- Backend accepts both binary and text WebSocket frames for input
|
||||
- `make test` passes (existing backend tests still green)
|
||||
|
||||
**Depends on:** None
|
||||
**Estimated:** 3 hours
|
||||
|
||||
---
|
||||
|
||||
## Task T3: Backend Echo Detection and Graceful Exit
|
||||
|
||||
**Files:**
|
||||
- `apps/api/src/services/terminal_session.py`
|
||||
- `apps/api/src/services/terminal_manager.py`
|
||||
- `apps/api/src/api/terminal.py`
|
||||
|
||||
**Description:**
|
||||
Add `termios` PTY inspection to detect ECHO flag state changes. Send `set_echo_state` control messages to client when echo toggles. Detect container process exit (returncode set) and send `session_ended` JSON message before closing WebSocket with code 1000. Distinguish between normal process exit, container stop, and unexpected errors.
|
||||
|
||||
**Acceptance:**
|
||||
- Running `stty -echo` in terminal triggers `set_echo_state: false` message
|
||||
- Running `stty echo` triggers `set_echo_state: true` message
|
||||
- Running `exit` in shell sends `session_ended: { reason: "process_exit" }` then closes with code 1000
|
||||
- Stopping container sends `session_ended: { reason: "container_stop" }`
|
||||
- Unexpected errors still close with code 4000 and error message
|
||||
|
||||
**Depends on:** T2
|
||||
**Estimated:** 2.5 hours
|
||||
|
||||
---
|
||||
|
||||
## Task T4: useTerminalConnection Hook
|
||||
|
||||
**Files:**
|
||||
- `apps/web/src/hooks/use-terminal-connection.ts` (new)
|
||||
|
||||
**Description:**
|
||||
Implement the core connection hook with: WebSocket lifecycle (open/close/reconnect with exponential backoff), heartbeat (send ping every 15s, timeout after 5s), local echo (write printable ASCII to xterm immediately, deduplicate server echo), resize debouncing (200ms, max 1/500ms), scrollback serialization on disconnect, scrollback restoration on reconnect, connection quality tracking (latency, jitter), manual reconnect bypass.
|
||||
|
||||
**Acceptance:**
|
||||
- Hook exposes `state`, `sendInput`, `sendResize`, `reconnect`, `onData`, `onControl`
|
||||
- Reconnect backoff: 1s, 2s, 4s, 8s, then max 30s
|
||||
- Max 10 reconnection attempts before giving up
|
||||
- Local echo works for printable ASCII; disabled when echo state is false
|
||||
- Pending echo buffer deduplicates server echo correctly
|
||||
- Pending echo buffer flushes to terminal if it grows > 100 chars
|
||||
- Resize sends at most 1 message per 500ms
|
||||
- `Ctrl+Shift+R` triggers immediate reconnect when disconnected
|
||||
- Scrollback serialized to `sessionStorage` on disconnect, restored on reconnect with divider
|
||||
|
||||
**Depends on:** T1
|
||||
**Estimated:** 4 hours
|
||||
|
||||
---
|
||||
|
||||
## Task T5: TerminalComponent Rewrite
|
||||
|
||||
**Files:**
|
||||
- `apps/web/src/components/terminal.tsx` (rewrite)
|
||||
- `apps/web/src/pages/terminal.tsx` (minor)
|
||||
- `apps/web/src/styles.css` (add terminal status styles)
|
||||
|
||||
**Description:**
|
||||
Rewrite `TerminalComponent` to use `useTerminalConnection`. Integrate xterm.js with the hook's `onData` and `onControl` callbacks. Add status bar with connection quality indicator (green/yellow/red/gray dot, latency tooltip, attempt counter). Add reconnect overlay when disconnected. Wire xterm `onData` to hook's `sendInput`. Use `ResizeObserver` for container-level resize detection. Apply xterm-addon-serialize for scrollback. Update page to pass instance ID and handle close.
|
||||
|
||||
**Acceptance:**
|
||||
- Terminal renders and connects on mount
|
||||
- Status bar shows correct dot color based on connection state
|
||||
- Hovering dot shows latency tooltip
|
||||
- Reconnect overlay appears when max retries exceeded
|
||||
- ResizeObserver triggers fit + resize message (debounced)
|
||||
- Theme colors adapt to dark/light mode
|
||||
- Close button works
|
||||
|
||||
**Depends on:** T4
|
||||
**Estimated:** 3 hours
|
||||
|
||||
---
|
||||
|
||||
## Task T6: Frontend Tests
|
||||
|
||||
**Files:**
|
||||
- `apps/web/src/utils/terminal-protocol.test.ts` (new)
|
||||
- `apps/web/src/hooks/use-terminal-connection.test.ts` (new)
|
||||
|
||||
**Description:**
|
||||
Write Vitest tests for protocol utilities (encode/decode all message types, base64 round-trip, frame type detection). Write tests for the connection hook using a mock WebSocket server (or manual mock). Test: reconnect backoff timing, heartbeat timeout detection, local echo deduplication, resize throttling, scrollback serialization round-trip.
|
||||
|
||||
**Acceptance:**
|
||||
- Protocol tests cover all message types and edge cases
|
||||
- Hook tests cover connection lifecycle without real WebSocket
|
||||
- All tests pass: `cd apps/web && npm test`
|
||||
- Coverage for new code > 80%
|
||||
|
||||
**Depends on:** T1, T4
|
||||
**Estimated:** 3 hours
|
||||
|
||||
---
|
||||
|
||||
## Task T7: Backend Tests
|
||||
|
||||
**Files:**
|
||||
- `apps/api/tests/unit/test_terminal_session.py` (new)
|
||||
- `apps/api/tests/unit/test_terminal_manager.py` (new)
|
||||
|
||||
**Description:**
|
||||
Write pytest unit tests for `TerminalSession` (PTY creation, resize, echo detection, process exit detection). Write tests for `TerminalManager` (session creation, batching logic, heartbeat tracking). Use mocks for `os`, `pty`, `termios`, and `asyncio` where appropriate.
|
||||
|
||||
**Acceptance:**
|
||||
- TerminalSession tests: start, resize, write, read, echo detection, close
|
||||
- TerminalManager tests: create session, read loop batching, heartbeat timeout
|
||||
- All tests pass: `make test`
|
||||
|
||||
**Depends on:** T2, T3
|
||||
**Estimated:** 2.5 hours
|
||||
|
||||
---
|
||||
|
||||
## Task Order and Dependencies
|
||||
|
||||
```
|
||||
T1 ──► T4 ──► T5 ──► PR-3 (Frontend UI)
|
||||
│
|
||||
└──► T6 (Frontend tests)
|
||||
|
||||
T2 ──► T3 ──► PR-1 (Backend foundation)
|
||||
│
|
||||
└──► T7 (Backend tests)
|
||||
```
|
||||
|
||||
**Parallel work possible:**
|
||||
- T1 and T2 can be done in parallel (no dependencies)
|
||||
- T3 and T4 can be done in parallel (T3 depends on T2, T4 depends on T1)
|
||||
- T5 depends on T4
|
||||
- T6 depends on T4
|
||||
- T7 depends on T3
|
||||
|
||||
## Chained PR Plan
|
||||
|
||||
### PR-1: Backend Foundation
|
||||
**Scope:** T1 (protocol types only) + T2 + T3 + T7
|
||||
**Files touched:** `apps/api/src/services/terminal_manager.py`, `apps/api/src/services/terminal_session.py`, `apps/api/src/api/terminal.py`, new test files, `apps/web/src/types/terminal.ts`, `apps/web/src/utils/terminal-protocol.ts`
|
||||
**Estimated diff:** ~590 lines
|
||||
**Review focus:** Protocol correctness, heartbeat logic, batching efficiency
|
||||
|
||||
### PR-2: Frontend Connection Hook
|
||||
**Scope:** T4 + T6
|
||||
**Files touched:** `apps/web/src/hooks/use-terminal-connection.ts`, new test files
|
||||
**Estimated diff:** ~460 lines
|
||||
**Review focus:** State machine correctness, local echo algorithm, reconnection logic
|
||||
|
||||
### PR-3: Terminal UI Integration
|
||||
**Scope:** T5
|
||||
**Files touched:** `apps/web/src/components/terminal.tsx`, `apps/web/src/pages/terminal.tsx`, `apps/web/src/styles.css`
|
||||
**Estimated diff:** ~250 lines
|
||||
**Review focus:** UX, accessibility, visual polish, integration with hook
|
||||
|
||||
**Note:** PR-2 and PR-3 can be developed in parallel if PR-1's protocol types are stable. The hook can be tested against mock protocol types before the backend is merged.
|
||||
Reference in New Issue
Block a user