feat: responsive web terminal with auto-reconnect, heartbeat, and local echo
Implements a resilient, responsive web terminal that survives network blips, provides instant typing feedback, and restores scrollback on reconnect. Backend changes: - Add heartbeat tracking (15s ping interval, 60s idle timeout) - Add message batching (16ms flush window) for efficient I/O - Add termios echo detection and set_echo_state control messages - Add graceful session_ended notification before close - Add ping/pong protocol support Frontend changes: - Rewrite TerminalComponent with status bar, connection indicator, session-ended overlay, reconnect banner, and ResizeObserver - Add useTerminalConnection hook with: - Exponential backoff auto-reconnect (1s → 30s max, 10 attempts) - Heartbeat/ping-pong with latency tracking - Local echo for printable ASCII with server deduplication - Resize debounce (200ms) + throttle (500ms) - Scrollback serialization via xterm-addon-serialize - Ctrl+Shift+R manual reconnect shortcut - Add WebSocket protocol types and encoding utilities - Add xterm-addon-serialize dependency Tests: - 16 backend unit tests (TerminalSession + TerminalManager) - 13 frontend hook tests (connection lifecycle, reconnect, resize, scrollback, callbacks) Quality gates: - Frontend typecheck: clean - Frontend lint: clean - Frontend tests: 48 passed - Backend unit tests: 101 passed - Backend ruff: clean SDD artifacts: openspec/changes/responsive-terminal/
This commit is contained in:
@@ -0,0 +1,371 @@
|
||||
# Design: Responsive Web Terminal
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ BROWSER │
|
||||
│ ┌──────────────┐ ┌─────────────────┐ ┌──────────────────────────┐ │
|
||||
│ │ TerminalPage │ │ TerminalComponent │ │ TerminalConnection │ │
|
||||
│ │ (router) │◄──│ (xterm.js + UI) │◄──│ (WS + heartbeat + echo) │ │
|
||||
│ └──────────────┘ └─────────────────┘ └──────────────────────────┘ │
|
||||
│ │ │ │
|
||||
│ ┌─────┴─────┐ ┌──────┴──────┐ │
|
||||
│ │ xterm.js │ │ sessionStorage│ │
|
||||
│ │ + addons │ │ (scrollback) │ │
|
||||
│ └───────────┘ └───────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
│ WebSocket
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ FASTAPI │
|
||||
│ ┌──────────────────┐ ┌──────────────────┐ ┌─────────────────────┐ │
|
||||
│ │ terminal.py │ │ TerminalManager │ │ TerminalSession │ │
|
||||
│ │ (WS endpoint) │◄──│ (session mgmt) │◄──│ (PTY + docker exec) │ │
|
||||
│ └──────────────────┘ └──────────────────┘ └─────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────┴────┐ │
|
||||
│ │ docker │ │
|
||||
│ │ exec │ │
|
||||
│ └─────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Connection State Machine
|
||||
|
||||
### Client State Machine
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ IDLE │
|
||||
└──────┬──────┘
|
||||
│ mount
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ CONNECTING │◄────────────────────────┐
|
||||
└──────┬──────┘ │
|
||||
│ onopen │
|
||||
▼ │
|
||||
┌─────────────────────────┐ │
|
||||
│ CONNECTED │ │
|
||||
│ (heartbeat active) │ │
|
||||
└──────┬──────────┬───────┘ │
|
||||
│ │ │
|
||||
onclose/ │ │ ping timeout │
|
||||
onerror │ │ │
|
||||
▼ ▼ │
|
||||
┌─────────────────────────┐ │
|
||||
│ RECONNECTING │───────────────────┘
|
||||
│ (backoff: 1→2→4→8→30s) │ onopen (success)
|
||||
└──────┬──────────────────┘
|
||||
│ max retries (10)
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ DISCONNECTED │
|
||||
│ (manual reconnect │
|
||||
│ or navigate away) │
|
||||
└─────────────────────────┘
|
||||
```
|
||||
|
||||
### Server State Machine (per session)
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ PENDING │
|
||||
└──────┬──────┘
|
||||
│ ws.accept()
|
||||
▼
|
||||
┌─────────────┐
|
||||
┌────►│ ACTIVE │◄────┐
|
||||
│ │ (I/O loops │ │
|
||||
│ │ + heartbeat) │
|
||||
│ └──────┬──────┘ │
|
||||
│ │ │
|
||||
│ ws close│ new ws │
|
||||
│ ▼ │
|
||||
│ ┌─────────────┐ │
|
||||
└─────┤ CLOSED ├──────┘
|
||||
│ (cleanup) │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
## Protocol Specification
|
||||
|
||||
### Message Types
|
||||
|
||||
All control messages are JSON text frames. Raw terminal I/O uses binary frames.
|
||||
|
||||
#### Client → Server
|
||||
|
||||
| Type | Payload | When |
|
||||
|------|---------|------|
|
||||
| `ping` | `{ id: number }` | Every 15s of inactivity |
|
||||
| `pong` | `{ id: number }` | Response to server ping |
|
||||
| `resize` | `{ cols: number, rows: number }` | Terminal size changes (debounced) |
|
||||
| `input` | `{ data: string }` | User keystrokes (base64-encoded) |
|
||||
|
||||
#### Server → Client
|
||||
|
||||
| Type | Payload | When |
|
||||
|------|---------|------|
|
||||
| `pong` | `{ id: number }` | Response to client ping |
|
||||
| `status` | `{ status: "connected" \| "reconnected" }` | After auth + session ready |
|
||||
| `set_echo_state` | `{ enabled: boolean }` | When PTY echo flag changes |
|
||||
| `session_ended` | `{ reason: string }` | When container process exits |
|
||||
|
||||
### Binary Frame Convention
|
||||
|
||||
- **Client → Server:** Raw UTF-8 bytes of user input. No wrapping.
|
||||
- **Server → Client:** Raw bytes from PTY master read. No wrapping.
|
||||
|
||||
This avoids the current Blob→ArrayBuffer async conversion and JSON parsing overhead for the hot path.
|
||||
|
||||
## Frontend Design
|
||||
|
||||
### New Files
|
||||
|
||||
```
|
||||
apps/web/src/
|
||||
├── components/
|
||||
│ └── terminal.tsx (rewrite: state machine + reconnect)
|
||||
├── hooks/
|
||||
│ └── use-terminal-connection.ts (NEW: WS lifecycle, heartbeat, reconnect)
|
||||
├── utils/
|
||||
│ └── terminal-protocol.ts (NEW: message encoding/decoding)
|
||||
└── types/
|
||||
└── terminal.ts (NEW: protocol types)
|
||||
```
|
||||
|
||||
### `useTerminalConnection` Hook
|
||||
|
||||
Responsibilities:
|
||||
1. **WebSocket lifecycle:** Open, close, reconnect with backoff
|
||||
2. **Heartbeat:** Send ping every 15s, expect pong within 5s
|
||||
3. **Local echo:** Write printable chars to xterm immediately, deduplicate server echo
|
||||
4. **Resize:** Debounce resize events, send JSON control message
|
||||
5. **Scrollback:** Serialize on disconnect, restore on reconnect
|
||||
6. **State reporting:** Expose `status`, `latency`, `attempt` to UI
|
||||
|
||||
```typescript
|
||||
interface TerminalConnectionState {
|
||||
status: "connecting" | "connected" | "reconnecting" | "disconnected";
|
||||
attempt: number;
|
||||
latency: number | null; // last RTT in ms
|
||||
error: string | null;
|
||||
}
|
||||
|
||||
interface TerminalConnection {
|
||||
state: TerminalConnectionState;
|
||||
sendInput: (data: string) => void;
|
||||
sendResize: (cols: number, rows: number) => void;
|
||||
reconnect: () => void; // manual, bypasses backoff
|
||||
onData: (callback: (data: Uint8Array) => void) => void;
|
||||
onControl: (callback: (msg: ServerControlMessage) => void) => void;
|
||||
}
|
||||
```
|
||||
|
||||
### Local Echo Algorithm
|
||||
|
||||
```
|
||||
1. User types character c
|
||||
2. IF c is printable ASCII AND echo is enabled:
|
||||
a. Write c to xterm immediately
|
||||
b. Add c to "pending echo" buffer
|
||||
c. Send c to server via WebSocket
|
||||
3. ELSE (control char, arrow, escape sequence):
|
||||
a. Send c to server only
|
||||
b. Do NOT write to xterm
|
||||
4. When server sends data:
|
||||
a. For each char in server data:
|
||||
- IF char matches head of "pending echo" buffer:
|
||||
→ Pop from buffer (deduplication)
|
||||
- ELSE:
|
||||
→ Write char to xterm
|
||||
b. If "pending echo" buffer grows > 100 chars (stale):
|
||||
→ Flush buffer to xterm (server echo was lost)
|
||||
```
|
||||
|
||||
### Scrollback Serialization
|
||||
|
||||
```
|
||||
ON disconnect:
|
||||
1. buffer = xterm.serialize({ scrollback: 10000 })
|
||||
2. sessionStorage.setItem(`hq-terminal-${instanceId}`, buffer)
|
||||
|
||||
ON reconnect:
|
||||
1. buffer = sessionStorage.getItem(`hq-terminal-${instanceId}`)
|
||||
2. IF buffer:
|
||||
xterm.write(buffer)
|
||||
xterm.write("\r\n\x1b[90m--- Reconnected ---\x1b[0m\r\n")
|
||||
3. sessionStorage.removeItem(`hq-terminal-${instanceId}`)
|
||||
```
|
||||
|
||||
### Resize Debouncing
|
||||
|
||||
Use `ResizeObserver` on the terminal container instead of `window.resize`:
|
||||
|
||||
```typescript
|
||||
const resizeObserver = new ResizeObserver(
|
||||
debounce((entries) => {
|
||||
fitAddon.fit();
|
||||
sendResize(term.cols, term.rows);
|
||||
}, 200)
|
||||
);
|
||||
```
|
||||
|
||||
Rate limit: max 1 resize message per 500ms.
|
||||
|
||||
## Backend Design
|
||||
|
||||
### Modified Files
|
||||
|
||||
```
|
||||
apps/api/src/
|
||||
├── api/terminal.py (modify: ping/pong, session_ended)
|
||||
├── services/terminal_manager.py (rewrite: heartbeat tracking, batching)
|
||||
└── services/terminal_session.py (modify: batching read, echo detection)
|
||||
```
|
||||
|
||||
### TerminalManager Changes
|
||||
|
||||
**Heartbeat tracking:**
|
||||
- Track `last_ping_at` per session
|
||||
- Background task: if `last_ping_at` is older than 60s, close the WebSocket
|
||||
|
||||
**Message batching in read_loop:**
|
||||
```python
|
||||
async def _read_loop(self, session, websocket):
|
||||
buffer = bytearray()
|
||||
last_flush = time.monotonic()
|
||||
|
||||
while session.is_alive() and not session._closed:
|
||||
data = await session.read_output()
|
||||
if data:
|
||||
buffer.extend(data)
|
||||
|
||||
now = time.monotonic()
|
||||
if buffer and (now - last_flush >= 0.016 or not data):
|
||||
await websocket.send_bytes(bytes(buffer))
|
||||
buffer.clear()
|
||||
last_flush = now
|
||||
elif not data:
|
||||
await asyncio.sleep(0.001)
|
||||
```
|
||||
|
||||
**Reconnect support:**
|
||||
- When a new WebSocket connects for the same instance, terminate the old session and spawn a new one
|
||||
- This is the docker exec limitation — we cannot resume a PTY, only replace it
|
||||
|
||||
### TerminalSession Changes
|
||||
|
||||
**Echo state detection:**
|
||||
```python
|
||||
import termios
|
||||
|
||||
def _detect_echo_state(self) -> bool:
|
||||
if self._master_fd is None:
|
||||
return True
|
||||
try:
|
||||
attrs = termios.tcgetattr(self._master_fd)
|
||||
return bool(attrs[3] & termios.ECHO)
|
||||
except:
|
||||
return True
|
||||
```
|
||||
|
||||
Call `_detect_echo_state()` after each resize and periodically (every 1s) during active I/O. Send `set_echo_state` to client when it changes.
|
||||
|
||||
**Batch-friendly read:**
|
||||
- Change `read_output()` to use `asyncio.wait_for(select, timeout)` instead of blocking `select.select` with 0.1s timeout
|
||||
- Return immediately when data is available, sleep briefly when not
|
||||
|
||||
### Terminal Endpoint Changes
|
||||
|
||||
- Accept `ping` messages, respond with `pong`
|
||||
- On session end (process exit), send `session_ended` before closing with code 1000
|
||||
- Distinguish between container exit (friendly) and error (unexpected)
|
||||
|
||||
## Data Flow: Typing with Local Echo
|
||||
|
||||
```
|
||||
User presses 'a'
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ onData handler │──► xterm.write('a') [instant feedback]
|
||||
│ │──► pendingEcho.push('a')
|
||||
│ │──► ws.send(binary 'a')
|
||||
└─────────────────┘
|
||||
│
|
||||
▼ (network)
|
||||
┌─────────────────┐
|
||||
│ TerminalSession │──► os.write(master_fd, b'a')
|
||||
│ │──► docker exec PTY echoes 'a' back
|
||||
│ │──► os.read(master_fd) → b'a'
|
||||
└─────────────────┘
|
||||
│
|
||||
▼ (WebSocket)
|
||||
┌─────────────────┐
|
||||
│ onMessage │──► data = b'a'
|
||||
│ (binary frame) │──► IF data[0] == pendingEcho[0]:
|
||||
│ │ pendingEcho.shift() // dedup
|
||||
│ │ ELSE:
|
||||
│ │ xterm.write(data)
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow: Reconnection
|
||||
|
||||
```
|
||||
WebSocket closes (code 1006)
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ ConnectionState │──► status = "reconnecting"
|
||||
│ │──► attempt = 1
|
||||
│ │──► scrollback = xterm.serialize()
|
||||
│ │──► sessionStorage.setItem(key, scrollback)
|
||||
│ │──► schedule reconnect in 1s
|
||||
└─────────────────┘
|
||||
│
|
||||
▼ (1s later)
|
||||
┌─────────────────┐
|
||||
│ Reconnect │──► new WebSocket(url)
|
||||
│ │──► onopen: send scrollback from storage
|
||||
│ │──► xterm.write(restored + divider)
|
||||
│ │──► status = "connected"
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## Component Responsibilities
|
||||
|
||||
| Component | Responsibilities |
|
||||
|-----------|-----------------|
|
||||
| `TerminalPage` | Routing, layout, back button |
|
||||
| `TerminalComponent` | xterm.js lifecycle, addons, theme, status bar UI |
|
||||
| `useTerminalConnection` | WebSocket, heartbeat, reconnect, local echo, resize |
|
||||
| `terminal-protocol` | Encode/decode control messages, base64 helper |
|
||||
| `terminal.py` (API) | Auth, WebSocket accept, route control messages |
|
||||
| `TerminalManager` | Session lifecycle, heartbeat tracking, read/write loops |
|
||||
| `TerminalSession` | PTY + docker exec, echo detection, batching read |
|
||||
|
||||
## Tradeoffs
|
||||
|
||||
| Decision | Option A (Chosen) | Option B | Why A |
|
||||
|----------|-------------------|----------|-------|
|
||||
| **Reconnect strategy** | Exponential backoff, max 30s | Instant reconnect with no backoff | Backoff prevents server overload during outages |
|
||||
| **Local echo scope** | Printable ASCII only | All characters | Control chars/escapes need server-side processing (shell state) |
|
||||
| **Scrollback storage** | `sessionStorage` (tab-scoped) | `localStorage` (persistent) | Privacy: terminal may contain secrets |
|
||||
| **Scrollback cap** | 10,000 lines | Unlimited | Memory safety; 10K lines covers typical session |
|
||||
| **Heartbeat interval** | 15s client → server | 5s | Balance between detection speed and server load |
|
||||
| **Binary vs text I/O** | Binary frames for raw data | JSON-wrapped base64 | Binary is ~33% more efficient, zero parse overhead |
|
||||
| **Resize trigger** | ResizeObserver on container | window.resize | Container-level is more accurate for flex layouts |
|
||||
| **Echo detection** | Server inspects PTY termios | Client guesses from input | Server is authoritative; client cannot know shell state |
|
||||
| **New docker exec on reconnect** | Accept limitation | Implement persistent session | PTY resumption across connections is extremely complex; scrollback continuity is the pragmatic fix |
|
||||
|
||||
## Quality Gates
|
||||
|
||||
- `cd apps/web && npm run typecheck` — TypeScript compiles
|
||||
- `cd apps/web && npm run lint` — ESLint passes
|
||||
- `cd apps/web && npm test` — Vitest passes (new tests for protocol + hook)
|
||||
- `make test` — Backend pytest passes
|
||||
- Manual test: disconnect/reconnect, type latency, resize, container exit
|
||||
Reference in New Issue
Block a user