feat(frontend): update sessions page for enhanced health monitoring
- Add new status badges: starting, probing, unhealthy - Show tunnel error only when tunnel_status is unreachable - Show app error badge with status code for error_response - Add collapsible probe output section for diagnostics - Update health polling to check all active instances - Only show Recreate Tunnel button for unreachable tunnels
This commit is contained in:
Fusion
@@ -0,0 +1,57 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Runtime health endpoint
|
||||
The system SHALL provide a health endpoint that checks both container and tunnel health.
|
||||
|
||||
#### Scenario: Full health check
|
||||
- **GIVEN** a running web-enabled instance
|
||||
- **WHEN** `GET /instances/{id}/health` is called
|
||||
- **THEN** the response includes:
|
||||
- `container_status`: "running", "exited", "restarting", or "not_found"
|
||||
- `container_health`: "healthy", "unhealthy", or null (if no Docker healthcheck)
|
||||
- `tunnel_status`: "healthy", "unreachable", or "error_response"
|
||||
- `tunnel_status_code`: the HTTP status code from the tunnel URL, or null
|
||||
- `probe_status`: "passed", "failed", "pending", or "not_configured"
|
||||
- `healthy`: true only if container is running AND tunnel is healthy
|
||||
|
||||
#### Scenario: Health check for terminal-only instance
|
||||
- **GIVEN** a running terminal-only instance
|
||||
- **WHEN** `GET /instances/{id}/health` is called
|
||||
- **THEN** the response includes `container_status: "running"`
|
||||
- **AND** `tunnel_status: "not_applicable"`
|
||||
- **AND** `healthy: true` if container is running
|
||||
|
||||
### Requirement: Continuous health polling
|
||||
The system SHALL support periodic health checks from the frontend.
|
||||
|
||||
#### Scenario: Frontend health polling
|
||||
- **GIVEN** active instances in the UI
|
||||
- **WHEN** the frontend polls health every 30 seconds
|
||||
- **THEN** the health status is displayed as a badge
|
||||
- **AND** the badge shows "tunnel error" only when tunnel is unreachable
|
||||
- **AND** the badge shows "app error" when tunnel returns 502/503/504
|
||||
- **AND** the badge shows "starting" when container is up but probe is pending
|
||||
|
||||
### Requirement: Container state synchronization
|
||||
The system SHALL update instance status when container state changes unexpectedly.
|
||||
|
||||
#### Scenario: Container crashes
|
||||
- **GIVEN** an instance with status "running"
|
||||
- **WHEN** the container exits (crash or OOM)
|
||||
- **AND** a health check is performed
|
||||
- **THEN** the instance status is updated to "error"
|
||||
- **AND** the container exit code and logs are captured
|
||||
|
||||
#### Scenario: Container stopped externally
|
||||
- **GIVEN** an instance with status "running"
|
||||
- **WHEN** the container is stopped via docker command outside the system
|
||||
- **AND** a health check is performed
|
||||
- **THEN** the instance status is updated to "stopped"
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
None.
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,83 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Container startup verification
|
||||
The system SHALL verify that containers reach a running state before marking instances as "running".
|
||||
|
||||
#### Scenario: Container starts successfully
|
||||
- **WHEN** `docker compose up` completes
|
||||
- **THEN** the system polls `docker ps` every 2 seconds for up to 30 seconds
|
||||
- **AND** when the container state is "running", the instance status becomes "starting"
|
||||
- **AND** the readiness probe begins execution
|
||||
|
||||
#### Scenario: Container fails to start
|
||||
- **WHEN** `docker compose up` completes
|
||||
- **AND** the container exits within 30 seconds
|
||||
- **THEN** the instance status becomes "error"
|
||||
- **AND** the container exit code is stored in the error message
|
||||
|
||||
#### Scenario: Container stays in restarting loop
|
||||
- **WHEN** `docker compose up` completes
|
||||
- **AND** the container remains in "restarting" state after 30 seconds
|
||||
- **THEN** the instance status becomes "error"
|
||||
- **AND** the error message indicates the container is stuck restarting
|
||||
|
||||
### Requirement: Readiness probe execution
|
||||
The system SHALL execute readiness probes for web-enabled tool instances before marking them as "running".
|
||||
|
||||
#### Scenario: Probe succeeds
|
||||
- **GIVEN** a tool instance with status "starting"
|
||||
- **AND** the tool type has a readiness probe configured
|
||||
- **WHEN** the probe command returns exit code 0 within the timeout
|
||||
- **THEN** the instance status becomes "running"
|
||||
- **AND** the tunnel is created (for web tools)
|
||||
|
||||
#### Scenario: Probe times out
|
||||
- **GIVEN** a tool instance with status "starting"
|
||||
- **AND** the tool type has a readiness probe configured
|
||||
- **WHEN** the probe does not succeed within the configured timeout (default 30s)
|
||||
- **THEN** the instance status becomes "unhealthy"
|
||||
- **AND** the tunnel is still created (the container is running)
|
||||
- **AND** the last probe output is stored for diagnostics
|
||||
|
||||
#### Scenario: Terminal tool skips probe
|
||||
- **GIVEN** a tool instance for a terminal-only tool type
|
||||
- **WHEN** the container reaches "running" state
|
||||
- **THEN** the instance status immediately becomes "running"
|
||||
- **AND** no readiness probe is executed
|
||||
|
||||
### Requirement: Container health monitoring
|
||||
The system SHALL check container health in addition to tunnel health.
|
||||
|
||||
#### Scenario: Container is healthy
|
||||
- **GIVEN** a running instance
|
||||
- **WHEN** the health endpoint is queried
|
||||
- **THEN** the response includes `container_status: "running"`
|
||||
- **AND** the response includes `container_health: "healthy"` if Docker healthcheck exists
|
||||
|
||||
#### Scenario: Container has crashed
|
||||
- **GIVEN** a running instance
|
||||
- **WHEN** the container exits or is stopped externally
|
||||
- **AND** the health endpoint is queried
|
||||
- **THEN** the response includes `container_status: "exited"`
|
||||
- **AND** the response includes `healthy: false`
|
||||
- **AND** the instance status in the database is updated to "error"
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Status Monitoring
|
||||
The system SHALL track tool status with startup and health states.
|
||||
|
||||
#### Scenario: Status check with health details
|
||||
- **GIVEN** a tool instance
|
||||
- **WHEN** status is queried
|
||||
- **THEN** the real-time container status is returned:
|
||||
- `pending`: Instance created, container not yet started
|
||||
- `starting`: Container is running, readiness probe in progress
|
||||
- `running`: Container is running and probe passed (or terminal tool)
|
||||
- `unhealthy`: Container is running but probe failed/timed out
|
||||
- `stopped`: Container was stopped by user
|
||||
- `error`: Container failed to start or crashed
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,51 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Readiness probe configuration
|
||||
The system SHALL use tool type readiness probe configuration during instance startup.
|
||||
|
||||
#### Scenario: Web tool with custom probe
|
||||
- **GIVEN** a tool type with `readiness_probe` configured as:
|
||||
- `command: "curl -f http://localhost:8080/api/health"`
|
||||
- `timeout: 60`
|
||||
- `interval: 5`
|
||||
- **WHEN** an instance of this type starts
|
||||
- **THEN** the system executes the probe command inside the container
|
||||
- **AND** retries every 5 seconds for up to 60 seconds
|
||||
- **AND** the instance remains in "starting" status until probe succeeds
|
||||
|
||||
#### Scenario: Web tool with default probe
|
||||
- **GIVEN** a web-enabled tool type with no `readiness_probe` configured
|
||||
- **WHEN** an instance of this type starts
|
||||
- **THEN** the system uses the default probe: `curl -f http://localhost:{port}`
|
||||
- **AND** retries every 2 seconds for up to 30 seconds
|
||||
|
||||
#### Scenario: Probe command execution
|
||||
- **GIVEN** a readiness probe command
|
||||
- **WHEN** the system executes it inside the container
|
||||
- **THEN** it runs via `docker exec {container_id} sh -c "{command}"`
|
||||
- **AND** stdout/stderr are captured for diagnostics
|
||||
- **AND** exit code 0 indicates success
|
||||
|
||||
### Requirement: Probe result storage
|
||||
The system SHALL store readiness probe results for diagnostics.
|
||||
|
||||
#### Scenario: Successful probe logged
|
||||
- **GIVEN** a readiness probe that succeeds
|
||||
- **WHEN** the probe returns exit code 0
|
||||
- **THEN** the success is logged with timestamp
|
||||
- **AND** the instance status changes to "running"
|
||||
|
||||
#### Scenario: Failed probe logged
|
||||
- **GIVEN** a readiness probe that fails or times out
|
||||
- **WHEN** the probe reaches timeout
|
||||
- **THEN** the failure is logged with last stdout/stderr output
|
||||
- **AND** the instance status changes to "unhealthy"
|
||||
- **AND** the probe output is available via the health endpoint
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
None.
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,45 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Tunnel failure classification
|
||||
The system SHALL distinguish tunnel failures from application errors when determining whether to recreate a tunnel.
|
||||
|
||||
#### Scenario: Tunnel is broken
|
||||
- **GIVEN** a running instance with a tunnel URL
|
||||
- **WHEN** the health check receives one of:
|
||||
- Connection refused (ECONNREFUSED)
|
||||
- Connection timeout (ETIMEDOUT)
|
||||
- DNS resolution failure (ENOTFOUND)
|
||||
- Empty response
|
||||
- **THEN** the tunnel status is "unreachable"
|
||||
- **AND** the frontend shows a "tunnel error" badge
|
||||
- **AND** the "Recreate Tunnel" button is enabled
|
||||
|
||||
#### Scenario: Application returns error
|
||||
- **GIVEN** a running instance with a tunnel URL
|
||||
- **WHEN** the health check receives HTTP 502, 503, or 504
|
||||
- **THEN** the tunnel status is "error_response"
|
||||
- **AND** the frontend shows an "app error" badge
|
||||
- **AND** the "Recreate Tunnel" button is NOT shown
|
||||
- **AND** the status code is displayed for diagnostics
|
||||
|
||||
#### Scenario: Application is healthy
|
||||
- **GIVEN** a running instance with a tunnel URL
|
||||
- **WHEN** the health check receives HTTP 200-399
|
||||
- **THEN** the tunnel status is "healthy"
|
||||
- **AND** no error badge is shown
|
||||
|
||||
#### Scenario: Tunnel recreates successfully
|
||||
- **GIVEN** an instance with a broken tunnel (status "unreachable")
|
||||
- **WHEN** the user clicks "Recreate Tunnel"
|
||||
- **THEN** the old cloudflared process is stopped
|
||||
- **AND** a new cloudflared process is started
|
||||
- **AND** the instance URL is updated
|
||||
- **AND** the tunnel status becomes "healthy" (after verification)
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
None.
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,50 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Status Monitoring
|
||||
The system SHALL track tool status with startup and health states.
|
||||
|
||||
#### Scenario: Status check with health details
|
||||
- **GIVEN** a tool instance
|
||||
- **WHEN** status is queried
|
||||
- **THEN** the real-time container status is returned:
|
||||
- `pending`: Instance created, container not yet started
|
||||
- `starting`: Container is running, readiness probe in progress
|
||||
- `running`: Container is running and probe passed (or terminal tool)
|
||||
- `unhealthy`: Container is running but probe failed/timed out
|
||||
- `stopped`: Container was stopped by user
|
||||
- `error`: Container failed to start or crashed
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Health check endpoint enhancement
|
||||
The system SHALL provide detailed health information through the health check endpoint.
|
||||
|
||||
#### Scenario: Health check with container and tunnel status
|
||||
- **GIVEN** a running instance
|
||||
- **WHEN** `GET /instances/{id}/health` is called
|
||||
- **THEN** the response includes:
|
||||
- `healthy`: boolean - overall health
|
||||
- `container_status`: "running", "exited", "restarting", or "not_found"
|
||||
- `tunnel_status`: "healthy", "unreachable", "error_response", or "not_applicable"
|
||||
- `tunnel_status_code`: HTTP status code or null
|
||||
- `probe_status`: "passed", "failed", "pending", or "not_configured"
|
||||
- `last_probe_output`: string or null
|
||||
|
||||
### Requirement: Smart tunnel recreation
|
||||
The system SHALL only allow tunnel recreation when the tunnel itself is broken.
|
||||
|
||||
#### Scenario: Recreate tunnel for unreachable tunnel
|
||||
- **GIVEN** an instance with `tunnel_status: "unreachable"`
|
||||
- **WHEN** the recreate tunnel endpoint is called
|
||||
- **THEN** the tunnel is recreated
|
||||
- **AND** the new URL is returned
|
||||
|
||||
#### Scenario: Block recreation for application errors
|
||||
- **GIVEN** an instance with `tunnel_status: "error_response"` (e.g., HTTP 502)
|
||||
- **WHEN** the recreate tunnel endpoint is called
|
||||
- **THEN** the request is rejected with 400 Bad Request
|
||||
- **AND** the error message explains the tunnel is working but the application is returning errors
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
None.
|
||||
Reference in New Issue
Block a user