headquarter/openspec/changes/instance-health-monitoring/tasks.md

1. Backend - Container Startup Verification

• 1.1 Implement wait_for_container_running() in services/docker.py - polls docker ps until container reaches "running" state or timeout
• 1.2 Implement get_container_status() in services/docker.py - returns container state (running, exited, restarting, not_found) and exit code
• 1.3 Update start_instance() in api/tool_instances.py to call startup verification after docker compose up
• 1.4 Update instance status flow: "pending" → "starting" (after container verified running) → "running" (after probe)
• 1.5 Handle container startup failures: set status to "error" with exit code and logs

2. Backend - Readiness Probe Integration

• 2.1 Update start_instance() to execute readiness probe after container is running
• 2.2 Read readiness probe config from ToolType model (command, timeout, interval)
• 2.3 Implement default probes: web tools use curl -f http://localhost:{port}, terminal tools skip probe
• 2.4 Store probe result (output, exit code, timestamp) on instance or in logs
• 2.5 Update instance status based on probe result: "running" on success, "unhealthy" on timeout

3. Backend - Health Check Enhancement

• 3.1 Update check_instance_tunnel_health() to also check container status via Docker API
• 3.2 Enhance health response format with container_status, container_health, tunnel_status, tunnel_status_code, probe_status, last_probe_output
• 3.3 Implement check_container_health() helper that calls docker inspect for health status
• 3.4 Update overall healthy flag logic: true only if container running AND tunnel healthy

4. Backend - Smart Tunnel Recovery

• 4.1 Enhance check_tunnel_health() to classify errors: connection errors vs HTTP errors
• 4.2 Update recreate_tunnel_endpoint() to validate tunnel is actually broken before recreating
• 4.3 Return 400 Bad Request with explanation when trying to recreate tunnel for 502/503 errors
• 4.4 Update tunnel health response: tunnel_status values ("healthy", "unreachable", "error_response", "not_applicable")

5. Frontend - Status Display

• 5.1 Update session status badges to show new states: "starting", "unhealthy"
• 5.2 Show container error messages when instance fails to start
• 5.3 Display "tunnel error" badge only when tunnel_status === "unreachable"
• 5.4 Display "app error" badge when tunnel_status === "error_response" with status code
• 5.5 Show "starting..." badge when container_status === "running" but probe_status === "pending"

6. Frontend - Health Polling

• 6.1 Update health polling to use enhanced health endpoint response
• 6.2 Store full health state (container + tunnel) in component state
• 6.3 Update "Recreate Tunnel" button visibility: only show when tunnel_status === "unreachable"
• 6.4 Show probe output in a collapsible section for diagnostics

7. Testing and Quality Gates

• 7.1 Test container startup verification with fast-starting container
• 7.2 Test container startup failure (container exits immediately)
• 7.3 Test readiness probe success and timeout scenarios
• 7.4 Test health endpoint with various container states
• 7.5 Test smart tunnel recovery (connection error vs 502)
• 7.6 Run backend linting (ruff)
• 7.7 Run backend type checking (mypy)
• 7.8 Run frontend type checking (tsc)
• 7.9 Build frontend and verify no errors