headquarter/openspec/changes/container-monitoring-notifications/explore.md

Explore: Container Monitoring & Notification System

1. Current Container Lifecycle Flow

Start → Run → Stop → Cleanup

1. Create (POST /projects/{pid}/repositories/{rid}/instances)

   • Generates instance_name, finds free port, builds image (Dockerfile) or renders compose template.
   • Writes docker-compose.yml, .env, config files to instance_dir.
   • DB record created with status = "pending".
   • File: apps/api/src/api/tool_instances.py (lines ~300–600)

2. Start (POST /.../instances/{id}/start)

   • status set to "building".
   • Applies config profile (env vars, git mounts, port override, start command).
   • Runs docker compose up -d via execute_compose_command().
   • Retrieves container_id and container_name via docker ps filters.
   • Connects container to "backend" network.
   • status set to "starting", then polls docker inspect every 2s for up to 30s (wait_for_container_running).
   • If container exits → status = "error", logs captured.
   • Executes readiness probe (configurable per ToolType, default curl for web tools).
   • Probe succeeds → status = "running"; fails → status = "unhealthy".
   • For web tools, starts cloudflared tunnel and stores tunnel_id + public_url.
   • File: apps/api/src/api/tool_instances.py (lines ~1100–1500)

3. Stop (POST /.../instances/{id}/stop)

   • Kills cloudflared tunnel by PID (stop_cloudflared_tunnel).
   • Runs docker compose stop.
   • status = "stopped", clears url/public_url/tunnel_id, sets last_stopped_at.
   • File: apps/api/src/api/tool_instances.py (lines ~1500–1550)

4. Restart (POST /.../instances/{id}/restart)

   • Stops old tunnel, re-applies config profile, runs docker compose restart, recreates tunnel.
   • File: apps/api/src/api/tool_instances.py (lines ~1550–1650)

5. Delete (DELETE /.../instances/{id})

   • Stops tunnel, runs docker compose down -v, deletes instance_dir (includes clone + SSH keys).
   • Removes DB row.
   • File: apps/api/src/api/tool_instances.py (lines ~1650–1720)

6. Health Check Endpoint (GET /.../instances/{id}/health)

   • Calls get_container_status() (docker inspect) + check_tunnel_health() (curl to public URL).
   • Returns composite healthy flag: container running AND tunnel healthy for web tools.
   • Includes probe_status and last_probe_output from DB probe_result JSON column.
   • File: apps/api/src/api/tool_instances.py (lines ~1800–1870)

Background Patterns

• TerminalManager runs an _idle_check_loop() every 60s to close idle terminal sessions.
  • File: apps/api/src/services/terminal_manager.py (lines ~40–80)
• No general container-level background monitor or reaper exists.

---

2. Health Check Gaps

What's Present	What's Missing
One-time readiness probe at startup (execute_probe)	No continuous health monitoring after startup
GET /health returns container + tunnel status on demand	No periodic background polling of container state
docker inspect reads State.Status, ExitCode, Health.Status	No liveness probe (only readiness at start)
Tunnel health checked via HTTP curl	No automatic recovery on container crash
probe_result JSON stored in DB	No health history / time-series
Frontend polls health every 30s for running instances	No server-side event push when health changes

Critical Gaps

1. Container crash goes unnoticed until a user manually refreshes or the frontend poll happens.
2. Tunnel failure is only detected on-demand (health endpoint or user action). No proactive retry or notification.
3. No OOM or exit-code tracking beyond the immediate startup phase.
4. No health state transitions (e.g., running → degraded → unhealthy → stopped).
5. Readiness probe is fire-and-forget; if it fails, status becomes "unhealthy" but no further action is taken.

---

3. Logging Gaps

Current Logging (apps/api/src/logging_config.py)

• Plain text format: %(asctime)s [%(levelname)s] %(name)s: %(message)s
• Request/response middleware logs timing and status codes.
• Exception middleware logs unhandled tracebacks.
• No structured logging (JSON), no correlation IDs, no container event stream persistence.

What's Logged (Container-Related)

• Docker compose up/down/start/stop return codes and stderr snippets.
• Container startup success/failure and wait time.
• Readiness probe attempts and results.
• Tunnel creation/failure.
• Permission fix warnings.

What's NOT Logged

• Container stdout/stderr is not persisted — only fetched on-demand via docker logs.
• No lifecycle event log (audit trail of who started/stopped/restarted what and when).
• No structured container events (create, start, die, oom, kill) from Docker daemon.
• No log aggregation — logs are ephemeral console output.
• No log levels per instance — all logs go through root logger.

---

4. Notification Gaps

Current State: No notification system exists.

Area	Finding
Backend events/pub-sub	None. No event bus, message queue, or broadcast mechanism.
WebSocket (non-terminal)	None. Only terminal uses WS (/ws/terminal/{instance_id}).
SSE	Not implemented.
Polling	Frontend polls instance list and health every 30s.
Frontend toast/alert	No toast, snackbar, or global notification component found.
Error display	Inline error-message divs and ErrorState component (data-states.tsx).

Frontend Evidence

• instance-list.tsx polls health every 30s for running instances and shows a "tunnel error" badge inline.
• session-card.tsx displays Tunnel Error / App Error badges but no push notification.
• api/client.ts has an axios interceptor for 401 redirect and retry logic, but no toast on errors.
• No toast, notification, snackbar, or alert components exist in apps/web/src/components/.

---

5. Database: Instance State Tracking

Table: tool_instances

File: apps/api/src/models/tool_instance.py

Column	Purpose
status	pending → building → starting → probing → running → unhealthy → stopped → error
container_id	Docker container ID (nullable)
container_name	Docker container name (nullable)
compose_path	Path to docker-compose.yml
port	Host port mapped to container
url / public_url	Cloudflare tunnel URL
tunnel_id	cloudflared PID
last_started_at / last_stopped_at	Timestamps
probe_result	JSON blob with last probe outcome
selected_config_profile_id	FK to config profile

Migrations

• 0006_tool_instances.py — base table with status, container_id, url, port.
• 0007_instance_container_name.py — adds container_name.
• 0011_tool_instance_tunnel_fields.py — adds public_url, tunnel_id.
• 0013_add_probe_result.py — adds probe_result (JSON).

Gaps

• No health_history table — can't track uptime, downtime, or flapping.
• No instance_events table — no audit log of state transitions.
• No notification_preferences or user_notifications table.

---

6. Key Files and Their Roles

File	Role
apps/api/src/services/docker.py	All Docker CLI interactions: compose up/down, container status/logs, tunnel management, port finding.
apps/api/src/api/tool_instances.py	CRUD + lifecycle endpoints for instances (create, start, stop, restart, delete, health, logs, proxy, tunnel recreate).
apps/api/src/services/terminal_manager.py	In-memory session registry + 60s idle cleanup loop. Pattern to emulate for container monitoring.
apps/api/src/services/readiness_probe.py	execute_probe() — runs a command inside a container with retry logic.
apps/api/src/logging_config.py	Plain-text logging setup, request/response middleware, exception middleware.
apps/api/src/models/tool_instance.py	SQLAlchemy model for tool_instances table.
apps/api/src/models/tool_type.py	SQLAlchemy model for tool_types, includes readiness_probe JSON config.
apps/api/src/api/health.py	System health endpoint (DB + disk), not per-instance health.
apps/web/src/components/instance-list.tsx	Displays instances with status dots, polls health every 30s, inline tunnel error badges.
apps/web/src/components/session-list.tsx	Grouped list of sessions (active vs recent), receives tunnelHealth prop.
apps/web/src/components/session-card.tsx	Card UI with status badges, stop/delete confirm, tunnel error display.
apps/web/src/api/sessions.ts	API client for instance CRUD and checkInstanceHealth().

---

7. Risks and Unknowns

1. Docker CLI dependency — All container operations shell out to docker / docker compose. No Docker SDK or lib used. This is slow and brittle under load.
2. Tunnel PID fragility — tunnel_id is a process ID string. If the API restarts, PIDs are lost and tunnels may leak.
3. No instance-level auto-restart — If a container exits (crash, OOM), it stays error or stopped until a user manually restarts it.
4. Probe result is a single JSON blob — Overwritten on every start. No history.
5. Polling load — Frontend polls every 30s per running instance. With many users + many instances, this generates significant health-check load.
6. No auth on WebSocket upgrade — Terminal WS endpoint may not validate session ownership on connection (not verified in this scout).
7. Cloudflared process leaks — If stop_cloudflared_tunnel fails or the API crashes, tunnel processes may become orphaned.
8. Log retention — docker logs is the only source; no rotation or persistence strategy.
9. Scaling limitation — In-memory TerminalManager and any future in-memory event bus won't work across multiple API replicas.

---

8. Recommended Architecture Approach

Short-Term (MVP): In-Memory Events + Server-Sent Events (SSE)

Rationale:

• The project already uses FastAPI. SSE is natively supported and simpler than WebSockets for one-way server→client push.
• No new infrastructure (message queue) needed.
• Matches the existing polling use case but eliminates 30s latency.

Components:

1. InstanceEventBus (in-memory singleton, similar to TerminalManager)
   • Publishes events: instance.created, instance.started, instance.stopped, instance.health_changed, instance.error.
2. Background Monitor Task (asyncio loop, like TerminalManager._idle_check_loop)
   • Every 10–30s, inspect running containers and tunnels.
   • On state change, update DB + publish event to bus.
3. SSE Endpoint (GET /events)
   • Stream JSON events to connected clients.
   • Frontend subscribes once, receives real-time updates.
4. Frontend Toast Layer
   • New lightweight toast component subscribed to SSE.
   • Shows notifications for errors, tunnel failures, successful starts.

Medium-Term: Persistent Event Log + Health History

1. instance_events table — append-only audit log of all lifecycle transitions.
2. health_checks table — periodic snapshots of container + tunnel health for trend analysis.
3. Structured logging — Switch to JSON format; include instance_id, event_type, correlation_id.

Long-Term: Message Queue (if multi-replica)

• If the API needs to scale horizontally, replace in-memory bus with Redis Pub/Sub or NATS.
• Background monitor becomes a separate worker process or scheduled task.

Decision Summary

Concern	Recommended Path
Real-time status updates	SSE from in-memory event bus
Health monitoring	Background asyncio task polling Docker + tunnels
User notifications	Lightweight toast component fed by SSE
Audit / history	New instance_events and health_checks tables
Log aggregation	JSON structured logs + optional log shipping
Multi-replica safety	Deferred to future; add Redis/NATS when needed