Files
headquarter/openspec/changes/instance-health-monitoring/tasks.md
T
Fusion d5f9df33b7 feat(frontend): update sessions page for enhanced health monitoring
- Add new status badges: starting, probing, unhealthy
- Show tunnel error only when tunnel_status is unreachable
- Show app error badge with status code for error_response
- Add collapsible probe output section for diagnostics
- Update health polling to check all active instances
- Only show Recreate Tunnel button for unreachable tunnels
2026-05-22 21:26:05 +02:00

57 lines
3.4 KiB
Markdown

## 1. Backend - Container Startup Verification
- [x] 1.1 Implement `wait_for_container_running()` in `services/docker.py` - polls `docker ps` until container reaches "running" state or timeout
- [x] 1.2 Implement `get_container_status()` in `services/docker.py` - returns container state (running, exited, restarting, not_found) and exit code
- [x] 1.3 Update `start_instance()` in `api/tool_instances.py` to call startup verification after `docker compose up`
- [x] 1.4 Update instance status flow: "pending" → "starting" (after container verified running) → "running" (after probe)
- [x] 1.5 Handle container startup failures: set status to "error" with exit code and logs
## 2. Backend - Readiness Probe Integration
- [x] 2.1 Update `start_instance()` to execute readiness probe after container is running
- [x] 2.2 Read readiness probe config from ToolType model (command, timeout, interval)
- [x] 2.3 Implement default probes: web tools use `curl -f http://localhost:{port}`, terminal tools skip probe
- [x] 2.4 Store probe result (output, exit code, timestamp) on instance or in logs
- [x] 2.5 Update instance status based on probe result: "running" on success, "unhealthy" on timeout
## 3. Backend - Health Check Enhancement
- [x] 3.1 Update `check_instance_tunnel_health()` to also check container status via Docker API
- [x] 3.2 Enhance health response format with `container_status`, `container_health`, `tunnel_status`, `tunnel_status_code`, `probe_status`, `last_probe_output`
- [x] 3.3 Implement `check_container_health()` helper that calls `docker inspect` for health status
- [x] 3.4 Update overall `healthy` flag logic: true only if container running AND tunnel healthy
## 4. Backend - Smart Tunnel Recovery
- [x] 4.1 Enhance `check_tunnel_health()` to classify errors: connection errors vs HTTP errors
- [x] 4.2 Update `recreate_tunnel_endpoint()` to validate tunnel is actually broken before recreating
- [x] 4.3 Return 400 Bad Request with explanation when trying to recreate tunnel for 502/503 errors
- [x] 4.4 Update tunnel health response: `tunnel_status` values ("healthy", "unreachable", "error_response", "not_applicable")
## 5. Frontend - Status Display
- [x] 5.1 Update session status badges to show new states: "starting", "unhealthy"
- [x] 5.2 Show container error messages when instance fails to start
- [x] 5.3 Display "tunnel error" badge only when `tunnel_status === "unreachable"`
- [x] 5.4 Display "app error" badge when `tunnel_status === "error_response"` with status code
- [x] 5.5 Show "starting..." badge when `container_status === "running"` but `probe_status === "pending"`
## 6. Frontend - Health Polling
- [x] 6.1 Update health polling to use enhanced health endpoint response
- [x] 6.2 Store full health state (container + tunnel) in component state
- [x] 6.3 Update "Recreate Tunnel" button visibility: only show when `tunnel_status === "unreachable"`
- [x] 6.4 Show probe output in a collapsible section for diagnostics
## 7. Testing and Quality Gates
- [ ] 7.1 Test container startup verification with fast-starting container
- [ ] 7.2 Test container startup failure (container exits immediately)
- [ ] 7.3 Test readiness probe success and timeout scenarios
- [ ] 7.4 Test health endpoint with various container states
- [ ] 7.5 Test smart tunnel recovery (connection error vs 502)
- [ ] 7.6 Run backend linting (ruff)
- [ ] 7.7 Run backend type checking (mypy)
- [ ] 7.8 Run frontend type checking (tsc)
- [ ] 7.9 Build frontend and verify no errors