feat(frontend): update sessions page for enhanced health monitoring

- Add new status badges: starting, probing, unhealthy
- Show tunnel error only when tunnel_status is unreachable
- Show app error badge with status code for error_response
- Add collapsible probe output section for diagnostics
- Update health polling to check all active instances
- Only show Recreate Tunnel button for unreachable tunnels
This commit is contained in:
Fusion
2026-05-22 21:26:05 +02:00
parent 2a9e57ad0d
commit d5f9df33b7
14 changed files with 779 additions and 49 deletions
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-05-22
@@ -0,0 +1,79 @@
## Context
The current instance management has critical gaps in health monitoring that lead to poor user experience:
1. **Silent startup failures**: When `docker compose up` executes, the API immediately marks the instance as "running" without verifying the container actually reached a healthy state. Containers that crash on startup or fail to bind to their port appear "running" in the UI but serve 502 errors.
2. **Tunnel-only health checks**: The existing health check at `GET /instances/{id}/health` only performs an HTTP HEAD request to the tunnel URL. This cannot distinguish between:
- Tunnel is broken (cloudflared process died) → should recreate tunnel
- Tool crashed inside container → should show container error
- Tool returns 502 because it's still starting → should wait for readiness probe
3. **Unused readiness probes**: The `readiness_probe.py` service was built during the tool-workshop change but is never called during instance startup. Tool types can configure readiness probes (e.g., `curl -f http://localhost:8080/health`) but these are ignored.
4. **Blind auto-recovery**: The frontend shows a "Recreate Tunnel" button when the health check fails, but this recreates the tunnel even when the application itself is returning 502 errors, wasting time and confusing users.
## Goals / Non-Goals
**Goals:**
- Verify containers actually start successfully before marking instances as "running"
- Distinguish container health from tunnel health in monitoring
- Integrate readiness probes into the instance startup flow
- Only recreate tunnels when the tunnel itself is broken, not when the tool returns errors
- Provide clear error messages when instances fail to start
**Non-Goals:**
- Persistent tunnels (keeping temporary cloudflared tunnels)
- Automatic restart of crashed containers (Docker already does this with restart policies)
- Health check WebSocket push (polling is sufficient)
- Changing the Docker compose architecture
## Decisions
**1. Startup verification via Docker API**
- After `docker compose up`, poll `docker ps` for 30 seconds to verify container state transitions to "running"
- If container exits or stays in "restarting" loop, mark instance as "error" with exit code
- Rationale: Direct Docker API check is more reliable than HTTP checks during startup when ports may not be bound yet
**2. Readiness probe as gate to "running" status**
- Instance status flow: `pending``starting` (container up) → `running` (probe passed)
- If probe fails after timeout, status becomes `unhealthy` (not `error` - container is still up)
- Rationale: Distinguishes "container won't start" from "container started but app isn't ready yet"
**3. Container + Tunnel dual health checks**
- Health endpoint returns both `container_status` (from Docker API) and `tunnel_status` (HTTP check)
- Frontend shows different badges: "container unhealthy" vs "tunnel error"
- Rationale: Users need to know if they should wait (app starting) or recreate tunnel
**4. Smart tunnel failure detection**
- Connection errors (ECONNREFUSED, ETIMEDOUT, DNS failure) → tunnel is broken → allow recreate
- HTTP 502/503/504 → application error → show "app error" badge, don't recreate
- HTTP 200-399 → healthy
- Rationale: 502 from the tool means the tunnel is working fine, the tool just isn't responding
**5. Readiness probe configuration from ToolType**
- Use existing `readiness_probe` JSON field on ToolType model
- Default probe for web tools: `curl -f http://localhost:{port}`
- Default probe for terminal tools: none (skip probe, mark running immediately)
- Rationale: Leverages existing infrastructure, provides sensible defaults
## Risks / Trade-offs
**[Risk] Startup polling adds latency** → Mitigation: Poll every 2 seconds with 30 second max timeout. Most containers start in <5 seconds.
**[Risk] Docker API calls from API container** → Mitigation: API container already has Docker CLI access for managing instances. Using `docker ps` is consistent with existing patterns.
**[Risk] False "unhealthy" from slow-starting tools** → Mitigation: 30 second default timeout with configurable override per tool type. Frontend shows "starting..." status during probe.
**[Risk] Probe commands may not exist in container** → Mitigation: Probe failures log stderr. If probe command missing, container still starts but marked as running without probe validation.
## Migration Plan
No database migration needed. This change:
1. Adds new status values ("starting", "unhealthy") to existing `status` enum
2. Uses existing `readiness_probe` column on `tool_types` table
3. Changes health check API response format (adds fields, doesn't remove)
## Open Questions
None.
@@ -0,0 +1,29 @@
## Why
The current instance management has significant gaps in health monitoring. When starting instances, there's no verification that containers actually boot successfully - failures only surface when users try to access broken tunnels. The existing health check only validates tunnel URLs, not container health, leading to false positives where a "healthy" tunnel serves 502 errors from a crashed tool. Additionally, readiness probes exist as unused infrastructure, and auto-recovery blindly recreates tunnels on any HTTP error including legitimate 502s from the application itself.
## What Changes
- **Startup health checks**: Verify containers reach a running state after `docker compose up`, with clear failure messages when containers crash or fail to start
- **Container health checks**: Check container status via Docker API (`docker ps`, `docker inspect`) in addition to tunnel URL checks
- **Readiness probe integration**: Wire the existing `execute_probe()` service into the instance startup flow, using tool type configured probes
- **Smart auto-recovery**: Only recreate tunnels when the tunnel endpoint itself is unreachable (connection refused, timeout, DNS failure), NOT when the tool returns 502/503/504 errors
- **Instance status granularity**: Distinguish between "starting" (container booting), "running" (healthy), "unhealthy" (container up but probe failing), and "error" (failed to start)
## Capabilities
### New Capabilities
- `instance-startup-health`: Container startup verification and failure detection
- `instance-runtime-health`: Continuous health monitoring combining container and tunnel checks
- `readiness-probe-integration`: Tool-type configured readiness probes during instance startup
- `smart-tunnel-recovery`: Context-aware tunnel recreation that distinguishes tunnel failures from application errors
### Modified Capabilities
- `session-management-fixes`: Update health check endpoint to include container status, modify tunnel health logic to be smarter about error codes
## Impact
- **Backend**: `api/tool_instances.py` (start_instance, health check, recreate tunnel), `services/docker.py` (container status checks), `services/readiness_probe.py` (integration into startup flow)
- **Frontend**: `pages/sessions.tsx` (display new status states, show startup errors, smarter health badges)
- **Database**: No schema changes - uses existing `status` field with new state values
- **API**: New response fields in health check endpoint (container_status, probe_result, last_probe_at)
@@ -0,0 +1,57 @@
## ADDED Requirements
### Requirement: Runtime health endpoint
The system SHALL provide a health endpoint that checks both container and tunnel health.
#### Scenario: Full health check
- **GIVEN** a running web-enabled instance
- **WHEN** `GET /instances/{id}/health` is called
- **THEN** the response includes:
- `container_status`: "running", "exited", "restarting", or "not_found"
- `container_health`: "healthy", "unhealthy", or null (if no Docker healthcheck)
- `tunnel_status`: "healthy", "unreachable", or "error_response"
- `tunnel_status_code`: the HTTP status code from the tunnel URL, or null
- `probe_status`: "passed", "failed", "pending", or "not_configured"
- `healthy`: true only if container is running AND tunnel is healthy
#### Scenario: Health check for terminal-only instance
- **GIVEN** a running terminal-only instance
- **WHEN** `GET /instances/{id}/health` is called
- **THEN** the response includes `container_status: "running"`
- **AND** `tunnel_status: "not_applicable"`
- **AND** `healthy: true` if container is running
### Requirement: Continuous health polling
The system SHALL support periodic health checks from the frontend.
#### Scenario: Frontend health polling
- **GIVEN** active instances in the UI
- **WHEN** the frontend polls health every 30 seconds
- **THEN** the health status is displayed as a badge
- **AND** the badge shows "tunnel error" only when tunnel is unreachable
- **AND** the badge shows "app error" when tunnel returns 502/503/504
- **AND** the badge shows "starting" when container is up but probe is pending
### Requirement: Container state synchronization
The system SHALL update instance status when container state changes unexpectedly.
#### Scenario: Container crashes
- **GIVEN** an instance with status "running"
- **WHEN** the container exits (crash or OOM)
- **AND** a health check is performed
- **THEN** the instance status is updated to "error"
- **AND** the container exit code and logs are captured
#### Scenario: Container stopped externally
- **GIVEN** an instance with status "running"
- **WHEN** the container is stopped via docker command outside the system
- **AND** a health check is performed
- **THEN** the instance status is updated to "stopped"
## MODIFIED Requirements
None.
## REMOVED Requirements
None.
@@ -0,0 +1,83 @@
## ADDED Requirements
### Requirement: Container startup verification
The system SHALL verify that containers reach a running state before marking instances as "running".
#### Scenario: Container starts successfully
- **WHEN** `docker compose up` completes
- **THEN** the system polls `docker ps` every 2 seconds for up to 30 seconds
- **AND** when the container state is "running", the instance status becomes "starting"
- **AND** the readiness probe begins execution
#### Scenario: Container fails to start
- **WHEN** `docker compose up` completes
- **AND** the container exits within 30 seconds
- **THEN** the instance status becomes "error"
- **AND** the container exit code is stored in the error message
#### Scenario: Container stays in restarting loop
- **WHEN** `docker compose up` completes
- **AND** the container remains in "restarting" state after 30 seconds
- **THEN** the instance status becomes "error"
- **AND** the error message indicates the container is stuck restarting
### Requirement: Readiness probe execution
The system SHALL execute readiness probes for web-enabled tool instances before marking them as "running".
#### Scenario: Probe succeeds
- **GIVEN** a tool instance with status "starting"
- **AND** the tool type has a readiness probe configured
- **WHEN** the probe command returns exit code 0 within the timeout
- **THEN** the instance status becomes "running"
- **AND** the tunnel is created (for web tools)
#### Scenario: Probe times out
- **GIVEN** a tool instance with status "starting"
- **AND** the tool type has a readiness probe configured
- **WHEN** the probe does not succeed within the configured timeout (default 30s)
- **THEN** the instance status becomes "unhealthy"
- **AND** the tunnel is still created (the container is running)
- **AND** the last probe output is stored for diagnostics
#### Scenario: Terminal tool skips probe
- **GIVEN** a tool instance for a terminal-only tool type
- **WHEN** the container reaches "running" state
- **THEN** the instance status immediately becomes "running"
- **AND** no readiness probe is executed
### Requirement: Container health monitoring
The system SHALL check container health in addition to tunnel health.
#### Scenario: Container is healthy
- **GIVEN** a running instance
- **WHEN** the health endpoint is queried
- **THEN** the response includes `container_status: "running"`
- **AND** the response includes `container_health: "healthy"` if Docker healthcheck exists
#### Scenario: Container has crashed
- **GIVEN** a running instance
- **WHEN** the container exits or is stopped externally
- **AND** the health endpoint is queried
- **THEN** the response includes `container_status: "exited"`
- **AND** the response includes `healthy: false`
- **AND** the instance status in the database is updated to "error"
## MODIFIED Requirements
### Requirement: Status Monitoring
The system SHALL track tool status with startup and health states.
#### Scenario: Status check with health details
- **GIVEN** a tool instance
- **WHEN** status is queried
- **THEN** the real-time container status is returned:
- `pending`: Instance created, container not yet started
- `starting`: Container is running, readiness probe in progress
- `running`: Container is running and probe passed (or terminal tool)
- `unhealthy`: Container is running but probe failed/timed out
- `stopped`: Container was stopped by user
- `error`: Container failed to start or crashed
## REMOVED Requirements
None.
@@ -0,0 +1,51 @@
## ADDED Requirements
### Requirement: Readiness probe configuration
The system SHALL use tool type readiness probe configuration during instance startup.
#### Scenario: Web tool with custom probe
- **GIVEN** a tool type with `readiness_probe` configured as:
- `command: "curl -f http://localhost:8080/api/health"`
- `timeout: 60`
- `interval: 5`
- **WHEN** an instance of this type starts
- **THEN** the system executes the probe command inside the container
- **AND** retries every 5 seconds for up to 60 seconds
- **AND** the instance remains in "starting" status until probe succeeds
#### Scenario: Web tool with default probe
- **GIVEN** a web-enabled tool type with no `readiness_probe` configured
- **WHEN** an instance of this type starts
- **THEN** the system uses the default probe: `curl -f http://localhost:{port}`
- **AND** retries every 2 seconds for up to 30 seconds
#### Scenario: Probe command execution
- **GIVEN** a readiness probe command
- **WHEN** the system executes it inside the container
- **THEN** it runs via `docker exec {container_id} sh -c "{command}"`
- **AND** stdout/stderr are captured for diagnostics
- **AND** exit code 0 indicates success
### Requirement: Probe result storage
The system SHALL store readiness probe results for diagnostics.
#### Scenario: Successful probe logged
- **GIVEN** a readiness probe that succeeds
- **WHEN** the probe returns exit code 0
- **THEN** the success is logged with timestamp
- **AND** the instance status changes to "running"
#### Scenario: Failed probe logged
- **GIVEN** a readiness probe that fails or times out
- **WHEN** the probe reaches timeout
- **THEN** the failure is logged with last stdout/stderr output
- **AND** the instance status changes to "unhealthy"
- **AND** the probe output is available via the health endpoint
## MODIFIED Requirements
None.
## REMOVED Requirements
None.
@@ -0,0 +1,45 @@
## ADDED Requirements
### Requirement: Tunnel failure classification
The system SHALL distinguish tunnel failures from application errors when determining whether to recreate a tunnel.
#### Scenario: Tunnel is broken
- **GIVEN** a running instance with a tunnel URL
- **WHEN** the health check receives one of:
- Connection refused (ECONNREFUSED)
- Connection timeout (ETIMEDOUT)
- DNS resolution failure (ENOTFOUND)
- Empty response
- **THEN** the tunnel status is "unreachable"
- **AND** the frontend shows a "tunnel error" badge
- **AND** the "Recreate Tunnel" button is enabled
#### Scenario: Application returns error
- **GIVEN** a running instance with a tunnel URL
- **WHEN** the health check receives HTTP 502, 503, or 504
- **THEN** the tunnel status is "error_response"
- **AND** the frontend shows an "app error" badge
- **AND** the "Recreate Tunnel" button is NOT shown
- **AND** the status code is displayed for diagnostics
#### Scenario: Application is healthy
- **GIVEN** a running instance with a tunnel URL
- **WHEN** the health check receives HTTP 200-399
- **THEN** the tunnel status is "healthy"
- **AND** no error badge is shown
#### Scenario: Tunnel recreates successfully
- **GIVEN** an instance with a broken tunnel (status "unreachable")
- **WHEN** the user clicks "Recreate Tunnel"
- **THEN** the old cloudflared process is stopped
- **AND** a new cloudflared process is started
- **AND** the instance URL is updated
- **AND** the tunnel status becomes "healthy" (after verification)
## MODIFIED Requirements
None.
## REMOVED Requirements
None.
@@ -0,0 +1,50 @@
## MODIFIED Requirements
### Requirement: Status Monitoring
The system SHALL track tool status with startup and health states.
#### Scenario: Status check with health details
- **GIVEN** a tool instance
- **WHEN** status is queried
- **THEN** the real-time container status is returned:
- `pending`: Instance created, container not yet started
- `starting`: Container is running, readiness probe in progress
- `running`: Container is running and probe passed (or terminal tool)
- `unhealthy`: Container is running but probe failed/timed out
- `stopped`: Container was stopped by user
- `error`: Container failed to start or crashed
## ADDED Requirements
### Requirement: Health check endpoint enhancement
The system SHALL provide detailed health information through the health check endpoint.
#### Scenario: Health check with container and tunnel status
- **GIVEN** a running instance
- **WHEN** `GET /instances/{id}/health` is called
- **THEN** the response includes:
- `healthy`: boolean - overall health
- `container_status`: "running", "exited", "restarting", or "not_found"
- `tunnel_status`: "healthy", "unreachable", "error_response", or "not_applicable"
- `tunnel_status_code`: HTTP status code or null
- `probe_status`: "passed", "failed", "pending", or "not_configured"
- `last_probe_output`: string or null
### Requirement: Smart tunnel recreation
The system SHALL only allow tunnel recreation when the tunnel itself is broken.
#### Scenario: Recreate tunnel for unreachable tunnel
- **GIVEN** an instance with `tunnel_status: "unreachable"`
- **WHEN** the recreate tunnel endpoint is called
- **THEN** the tunnel is recreated
- **AND** the new URL is returned
#### Scenario: Block recreation for application errors
- **GIVEN** an instance with `tunnel_status: "error_response"` (e.g., HTTP 502)
- **WHEN** the recreate tunnel endpoint is called
- **THEN** the request is rejected with 400 Bad Request
- **AND** the error message explains the tunnel is working but the application is returning errors
## REMOVED Requirements
None.
@@ -0,0 +1,56 @@
## 1. Backend - Container Startup Verification
- [x] 1.1 Implement `wait_for_container_running()` in `services/docker.py` - polls `docker ps` until container reaches "running" state or timeout
- [x] 1.2 Implement `get_container_status()` in `services/docker.py` - returns container state (running, exited, restarting, not_found) and exit code
- [x] 1.3 Update `start_instance()` in `api/tool_instances.py` to call startup verification after `docker compose up`
- [x] 1.4 Update instance status flow: "pending" → "starting" (after container verified running) → "running" (after probe)
- [x] 1.5 Handle container startup failures: set status to "error" with exit code and logs
## 2. Backend - Readiness Probe Integration
- [x] 2.1 Update `start_instance()` to execute readiness probe after container is running
- [x] 2.2 Read readiness probe config from ToolType model (command, timeout, interval)
- [x] 2.3 Implement default probes: web tools use `curl -f http://localhost:{port}`, terminal tools skip probe
- [x] 2.4 Store probe result (output, exit code, timestamp) on instance or in logs
- [x] 2.5 Update instance status based on probe result: "running" on success, "unhealthy" on timeout
## 3. Backend - Health Check Enhancement
- [x] 3.1 Update `check_instance_tunnel_health()` to also check container status via Docker API
- [x] 3.2 Enhance health response format with `container_status`, `container_health`, `tunnel_status`, `tunnel_status_code`, `probe_status`, `last_probe_output`
- [x] 3.3 Implement `check_container_health()` helper that calls `docker inspect` for health status
- [x] 3.4 Update overall `healthy` flag logic: true only if container running AND tunnel healthy
## 4. Backend - Smart Tunnel Recovery
- [x] 4.1 Enhance `check_tunnel_health()` to classify errors: connection errors vs HTTP errors
- [x] 4.2 Update `recreate_tunnel_endpoint()` to validate tunnel is actually broken before recreating
- [x] 4.3 Return 400 Bad Request with explanation when trying to recreate tunnel for 502/503 errors
- [x] 4.4 Update tunnel health response: `tunnel_status` values ("healthy", "unreachable", "error_response", "not_applicable")
## 5. Frontend - Status Display
- [x] 5.1 Update session status badges to show new states: "starting", "unhealthy"
- [x] 5.2 Show container error messages when instance fails to start
- [x] 5.3 Display "tunnel error" badge only when `tunnel_status === "unreachable"`
- [x] 5.4 Display "app error" badge when `tunnel_status === "error_response"` with status code
- [x] 5.5 Show "starting..." badge when `container_status === "running"` but `probe_status === "pending"`
## 6. Frontend - Health Polling
- [x] 6.1 Update health polling to use enhanced health endpoint response
- [x] 6.2 Store full health state (container + tunnel) in component state
- [x] 6.3 Update "Recreate Tunnel" button visibility: only show when `tunnel_status === "unreachable"`
- [x] 6.4 Show probe output in a collapsible section for diagnostics
## 7. Testing and Quality Gates
- [ ] 7.1 Test container startup verification with fast-starting container
- [ ] 7.2 Test container startup failure (container exits immediately)
- [ ] 7.3 Test readiness probe success and timeout scenarios
- [ ] 7.4 Test health endpoint with various container states
- [ ] 7.5 Test smart tunnel recovery (connection error vs 502)
- [ ] 7.6 Run backend linting (ruff)
- [ ] 7.7 Run backend type checking (mypy)
- [ ] 7.8 Run frontend type checking (tsc)
- [ ] 7.9 Build frontend and verify no errors