## Context

The current instance management has critical gaps in health monitoring that lead to poor user experience:

1. **Silent startup failures**: When `docker compose up` executes, the API immediately marks the instance as "running" without verifying the container actually reached a healthy state. Containers that crash on startup or fail to bind to their port appear "running" in the UI but serve 502 errors.

2. **Tunnel-only health checks**: The existing health check at `GET /instances/{id}/health` only performs an HTTP HEAD request to the tunnel URL. This cannot distinguish between:
   - Tunnel is broken (cloudflared process died) → should recreate tunnel
   - Tool crashed inside container → should show container error
   - Tool returns 502 because it's still starting → should wait for readiness probe

3. **Unused readiness probes**: The `readiness_probe.py` service was built during the tool-workshop change but is never called during instance startup. Tool types can configure readiness probes (e.g., `curl -f http://localhost:8080/health`) but these are ignored.

4. **Blind auto-recovery**: The frontend shows a "Recreate Tunnel" button when the health check fails, but this recreates the tunnel even when the application itself is returning 502 errors, wasting time and confusing users.

## Goals / Non-Goals

**Goals:**
- Verify containers actually start successfully before marking instances as "running"
- Distinguish container health from tunnel health in monitoring
- Integrate readiness probes into the instance startup flow
- Only recreate tunnels when the tunnel itself is broken, not when the tool returns errors
- Provide clear error messages when instances fail to start

**Non-Goals:**
- Persistent tunnels (keeping temporary cloudflared tunnels)
- Automatic restart of crashed containers (Docker already does this with restart policies)
- Health check WebSocket push (polling is sufficient)
- Changing the Docker compose architecture

## Decisions

**1. Startup verification via Docker API**
- After `docker compose up`, poll `docker ps` for 30 seconds to verify container state transitions to "running"
- If container exits or stays in "restarting" loop, mark instance as "error" with exit code
- Rationale: Direct Docker API check is more reliable than HTTP checks during startup when ports may not be bound yet

**2. Readiness probe as gate to "running" status**
- Instance status flow: `pending` → `starting` (container up) → `running` (probe passed)
- If probe fails after timeout, status becomes `unhealthy` (not `error` - container is still up)
- Rationale: Distinguishes "container won't start" from "container started but app isn't ready yet"

**3. Container + Tunnel dual health checks**
- Health endpoint returns both `container_status` (from Docker API) and `tunnel_status` (HTTP check)
- Frontend shows different badges: "container unhealthy" vs "tunnel error"
- Rationale: Users need to know if they should wait (app starting) or recreate tunnel

**4. Smart tunnel failure detection**
- Connection errors (ECONNREFUSED, ETIMEDOUT, DNS failure) → tunnel is broken → allow recreate
- HTTP 502/503/504 → application error → show "app error" badge, don't recreate
- HTTP 200-399 → healthy
- Rationale: 502 from the tool means the tunnel is working fine, the tool just isn't responding

**5. Readiness probe configuration from ToolType**
- Use existing `readiness_probe` JSON field on ToolType model
- Default probe for web tools: `curl -f http://localhost:{port}`
- Default probe for terminal tools: none (skip probe, mark running immediately)
- Rationale: Leverages existing infrastructure, provides sensible defaults

## Risks / Trade-offs

**[Risk] Startup polling adds latency** → Mitigation: Poll every 2 seconds with 30 second max timeout. Most containers start in <5 seconds.

**[Risk] Docker API calls from API container** → Mitigation: API container already has Docker CLI access for managing instances. Using `docker ps` is consistent with existing patterns.

**[Risk] False "unhealthy" from slow-starting tools** → Mitigation: 30 second default timeout with configurable override per tool type. Frontend shows "starting..." status during probe.

**[Risk] Probe commands may not exist in container** → Mitigation: Probe failures log stderr. If probe command missing, container still starts but marked as running without probe validation.

## Migration Plan

No database migration needed. This change:
1. Adds new status values ("starting", "unhealthy") to existing `status` enum
2. Uses existing `readiness_probe` column on `tool_types` table
3. Changes health check API response format (adds fields, doesn't remove)

## Open Questions

None.