headquarter/openspec/changes/archive/2026-05-22-instance-health-monitoring/proposal.md at 38d116dcf746be1608246bed94de45cef8d1c4d5

Files

T

Fusion 063a839790 feat: implement repository clone mode with SSH key support

- Add clone_mode and branch fields to tool_instances
- Add ssh_key_id to git_repositories for per-repo SSH key assignment
- Implement host-side git cloning with branch selection (default: main)
- Mount SSH keys into containers for git operations in clone mode
- Add dirty state check on clone-mode instance deletion with confirmation
- Update SessionsPage with mount/clone selector, branch input, SSH key display
- Add SSH key selector to repository creation form
- Add dirty delete confirmation modal with changed files list
- Update API schemas and endpoints for new fields
- Sync delta specs to main specs (git-repo, tool-instances, repo-clone-mode)
- Archive completed OpenSpec change: repo-clone-mode-with-ssh
- Document git requirement for custom tool types

Quality gates: Frontend typecheck and build passed
OpenSpec: repo-clone-mode-with-ssh archived with all tasks complete

2026-05-22 22:56:35 +02:00

2.5 KiB

Raw Blame History

Why

The current instance management has significant gaps in health monitoring. When starting instances, there's no verification that containers actually boot successfully - failures only surface when users try to access broken tunnels. The existing health check only validates tunnel URLs, not container health, leading to false positives where a "healthy" tunnel serves 502 errors from a crashed tool. Additionally, readiness probes exist as unused infrastructure, and auto-recovery blindly recreates tunnels on any HTTP error including legitimate 502s from the application itself.

What Changes

Startup health checks: Verify containers reach a running state after docker compose up, with clear failure messages when containers crash or fail to start
Container health checks: Check container status via Docker API (docker ps, docker inspect) in addition to tunnel URL checks
Readiness probe integration: Wire the existing execute_probe() service into the instance startup flow, using tool type configured probes
Smart auto-recovery: Only recreate tunnels when the tunnel endpoint itself is unreachable (connection refused, timeout, DNS failure), NOT when the tool returns 502/503/504 errors
Instance status granularity: Distinguish between "starting" (container booting), "running" (healthy), "unhealthy" (container up but probe failing), and "error" (failed to start)

Capabilities

New Capabilities

instance-startup-health: Container startup verification and failure detection
instance-runtime-health: Continuous health monitoring combining container and tunnel checks
readiness-probe-integration: Tool-type configured readiness probes during instance startup
smart-tunnel-recovery: Context-aware tunnel recreation that distinguishes tunnel failures from application errors

Modified Capabilities

session-management-fixes: Update health check endpoint to include container status, modify tunnel health logic to be smarter about error codes

Impact

Backend: api/tool_instances.py (start_instance, health check, recreate tunnel), services/docker.py (container status checks), services/readiness_probe.py (integration into startup flow)
Frontend: pages/sessions.tsx (display new status states, show startup errors, smarter health badges)
Database: No schema changes - uses existing status field with new state values
API: New response fields in health check endpoint (container_status, probe_result, last_probe_at)

2.5 KiB Raw Blame History

Why