Fusion
d5f9df33b7
- Add new status badges: starting, probing, unhealthy - Show tunnel error only when tunnel_status is unreachable - Show app error badge with status code for error_response - Add collapsible probe output section for diagnostics - Update health polling to check all active instances - Only show Recreate Tunnel button for unreachable tunnels
80 lines
4.6 KiB
Markdown
80 lines
4.6 KiB
Markdown
## Context
|
|
|
|
The current instance management has critical gaps in health monitoring that lead to poor user experience:
|
|
|
|
1. **Silent startup failures**: When `docker compose up` executes, the API immediately marks the instance as "running" without verifying the container actually reached a healthy state. Containers that crash on startup or fail to bind to their port appear "running" in the UI but serve 502 errors.
|
|
|
|
2. **Tunnel-only health checks**: The existing health check at `GET /instances/{id}/health` only performs an HTTP HEAD request to the tunnel URL. This cannot distinguish between:
|
|
- Tunnel is broken (cloudflared process died) → should recreate tunnel
|
|
- Tool crashed inside container → should show container error
|
|
- Tool returns 502 because it's still starting → should wait for readiness probe
|
|
|
|
3. **Unused readiness probes**: The `readiness_probe.py` service was built during the tool-workshop change but is never called during instance startup. Tool types can configure readiness probes (e.g., `curl -f http://localhost:8080/health`) but these are ignored.
|
|
|
|
4. **Blind auto-recovery**: The frontend shows a "Recreate Tunnel" button when the health check fails, but this recreates the tunnel even when the application itself is returning 502 errors, wasting time and confusing users.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Verify containers actually start successfully before marking instances as "running"
|
|
- Distinguish container health from tunnel health in monitoring
|
|
- Integrate readiness probes into the instance startup flow
|
|
- Only recreate tunnels when the tunnel itself is broken, not when the tool returns errors
|
|
- Provide clear error messages when instances fail to start
|
|
|
|
**Non-Goals:**
|
|
- Persistent tunnels (keeping temporary cloudflared tunnels)
|
|
- Automatic restart of crashed containers (Docker already does this with restart policies)
|
|
- Health check WebSocket push (polling is sufficient)
|
|
- Changing the Docker compose architecture
|
|
|
|
## Decisions
|
|
|
|
**1. Startup verification via Docker API**
|
|
- After `docker compose up`, poll `docker ps` for 30 seconds to verify container state transitions to "running"
|
|
- If container exits or stays in "restarting" loop, mark instance as "error" with exit code
|
|
- Rationale: Direct Docker API check is more reliable than HTTP checks during startup when ports may not be bound yet
|
|
|
|
**2. Readiness probe as gate to "running" status**
|
|
- Instance status flow: `pending` → `starting` (container up) → `running` (probe passed)
|
|
- If probe fails after timeout, status becomes `unhealthy` (not `error` - container is still up)
|
|
- Rationale: Distinguishes "container won't start" from "container started but app isn't ready yet"
|
|
|
|
**3. Container + Tunnel dual health checks**
|
|
- Health endpoint returns both `container_status` (from Docker API) and `tunnel_status` (HTTP check)
|
|
- Frontend shows different badges: "container unhealthy" vs "tunnel error"
|
|
- Rationale: Users need to know if they should wait (app starting) or recreate tunnel
|
|
|
|
**4. Smart tunnel failure detection**
|
|
- Connection errors (ECONNREFUSED, ETIMEDOUT, DNS failure) → tunnel is broken → allow recreate
|
|
- HTTP 502/503/504 → application error → show "app error" badge, don't recreate
|
|
- HTTP 200-399 → healthy
|
|
- Rationale: 502 from the tool means the tunnel is working fine, the tool just isn't responding
|
|
|
|
**5. Readiness probe configuration from ToolType**
|
|
- Use existing `readiness_probe` JSON field on ToolType model
|
|
- Default probe for web tools: `curl -f http://localhost:{port}`
|
|
- Default probe for terminal tools: none (skip probe, mark running immediately)
|
|
- Rationale: Leverages existing infrastructure, provides sensible defaults
|
|
|
|
## Risks / Trade-offs
|
|
|
|
**[Risk] Startup polling adds latency** → Mitigation: Poll every 2 seconds with 30 second max timeout. Most containers start in <5 seconds.
|
|
|
|
**[Risk] Docker API calls from API container** → Mitigation: API container already has Docker CLI access for managing instances. Using `docker ps` is consistent with existing patterns.
|
|
|
|
**[Risk] False "unhealthy" from slow-starting tools** → Mitigation: 30 second default timeout with configurable override per tool type. Frontend shows "starting..." status during probe.
|
|
|
|
**[Risk] Probe commands may not exist in container** → Mitigation: Probe failures log stderr. If probe command missing, container still starts but marked as running without probe validation.
|
|
|
|
## Migration Plan
|
|
|
|
No database migration needed. This change:
|
|
1. Adds new status values ("starting", "unhealthy") to existing `status` enum
|
|
2. Uses existing `readiness_probe` column on `tool_types` table
|
|
3. Changes health check API response format (adds fields, doesn't remove)
|
|
|
|
## Open Questions
|
|
|
|
None.
|