headquarter/openspec/changes/container-monitoring-notifications/explore.md

# Explore: Container Monitoring & Notification System

## 1. Current Container Lifecycle Flow

### Start → Run → Stop → Cleanup

1. **Create** (`POST /projects/{pid}/repositories/{rid}/instances`)
   - Generates `instance_name`, finds free port, builds image (Dockerfile) or renders compose template.
   - Writes `docker-compose.yml`, `.env`, config files to `instance_dir`.
   - DB record created with `status = "pending"`.
   - **File:** `apps/api/src/api/tool_instances.py` (lines ~300–600)

2. **Start** (`POST /.../instances/{id}/start`)
   - `status` set to `"building"`.
   - Applies config profile (env vars, git mounts, port override, start command).
   - Runs `docker compose up -d` via `execute_compose_command()`.
   - Retrieves `container_id` and `container_name` via `docker ps` filters.
   - Connects container to `"backend"` network.
   - `status` set to `"starting"`, then polls `docker inspect` every 2s for up to 30s (`wait_for_container_running`).
   - If container exits → `status = "error"`, logs captured.
   - Executes readiness probe (configurable per `ToolType`, default `curl` for web tools).
   - Probe succeeds → `status = "running"`; fails → `status = "unhealthy"`.
   - For web tools, starts `cloudflared` tunnel and stores `tunnel_id` + `public_url`.
   - **File:** `apps/api/src/api/tool_instances.py` (lines ~1100–1500)

3. **Stop** (`POST /.../instances/{id}/stop`)
   - Kills cloudflared tunnel by PID (`stop_cloudflared_tunnel`).
   - Runs `docker compose stop`.
   - `status = "stopped"`, clears `url`/`public_url`/`tunnel_id`, sets `last_stopped_at`.
   - **File:** `apps/api/src/api/tool_instances.py` (lines ~1500–1550)

4. **Restart** (`POST /.../instances/{id}/restart`)
   - Stops old tunnel, re-applies config profile, runs `docker compose restart`, recreates tunnel.
   - **File:** `apps/api/src/api/tool_instances.py` (lines ~1550–1650)

5. **Delete** (`DELETE /.../instances/{id}`)
   - Stops tunnel, runs `docker compose down -v`, deletes `instance_dir` (includes clone + SSH keys).
   - Removes DB row.
   - **File:** `apps/api/src/api/tool_instances.py` (lines ~1650–1720)

6. **Health Check Endpoint** (`GET /.../instances/{id}/health`)
   - Calls `get_container_status()` (docker inspect) + `check_tunnel_health()` (curl to public URL).
   - Returns composite `healthy` flag: container running AND tunnel healthy for web tools.
   - Includes `probe_status` and `last_probe_output` from DB `probe_result` JSON column.
   - **File:** `apps/api/src/api/tool_instances.py` (lines ~1800–1870)

### Background Patterns
- **TerminalManager** runs an `_idle_check_loop()` every 60s to close idle terminal sessions.
  - **File:** `apps/api/src/services/terminal_manager.py` (lines ~40–80)
- No general container-level background monitor or reaper exists.

---

## 2. Health Check Gaps

| What's Present | What's Missing |
|---|---|
| One-time readiness probe at startup (`execute_probe`) | **No continuous health monitoring** after startup |
| `GET /health` returns container + tunnel status on demand | **No periodic background polling** of container state |
| `docker inspect` reads `State.Status`, `ExitCode`, `Health.Status` | **No liveness probe** (only readiness at start) |
| Tunnel health checked via HTTP curl | **No automatic recovery** on container crash |
| `probe_result` JSON stored in DB | **No health history / time-series** |
| Frontend polls health every 30s for running instances | **No server-side event push** when health changes |

### Critical Gaps
1. **Container crash goes unnoticed** until a user manually refreshes or the frontend poll happens.
2. **Tunnel failure is only detected on-demand** (health endpoint or user action). No proactive retry or notification.
3. **No OOM or exit-code tracking** beyond the immediate startup phase.
4. **No health state transitions** (e.g., `running → degraded → unhealthy → stopped`).
5. **Readiness probe is fire-and-forget**; if it fails, status becomes `"unhealthy"` but no further action is taken.

---

## 3. Logging Gaps

### Current Logging (`apps/api/src/logging_config.py`)
- Plain text format: `%(asctime)s [%(levelname)s] %(name)s: %(message)s`
- Request/response middleware logs timing and status codes.
- Exception middleware logs unhandled tracebacks.
- **No structured logging** (JSON), **no correlation IDs**, **no container event stream persistence**.

### What's Logged (Container-Related)
- Docker compose up/down/start/stop return codes and stderr snippets.
- Container startup success/failure and wait time.
- Readiness probe attempts and results.
- Tunnel creation/failure.
- Permission fix warnings.

### What's NOT Logged
- **Container stdout/stderr is not persisted** — only fetched on-demand via `docker logs`.
- **No lifecycle event log** (audit trail of who started/stopped/restarted what and when).
- **No structured container events** (create, start, die, oom, kill) from Docker daemon.
- **No log aggregation** — logs are ephemeral console output.
- **No log levels per instance** — all logs go through root logger.

---

## 4. Notification Gaps

### Current State: **No notification system exists.**

| Area | Finding |
|---|---|
| **Backend events/pub-sub** | None. No event bus, message queue, or broadcast mechanism. |
| **WebSocket (non-terminal)** | None. Only terminal uses WS (`/ws/terminal/{instance_id}`). |
| **SSE** | Not implemented. |
| **Polling** | Frontend polls instance list and health every 30s. |
| **Frontend toast/alert** | No toast, snackbar, or global notification component found. |
| **Error display** | Inline `error-message` divs and `ErrorState` component (`data-states.tsx`). |

### Frontend Evidence
- `instance-list.tsx` polls health every 30s for running instances and shows a `"tunnel error"` badge inline.
- `session-card.tsx` displays `Tunnel Error` / `App Error` badges but no push notification.
- `api/client.ts` has an axios interceptor for 401 redirect and retry logic, but no toast on errors.
- No `toast`, `notification`, `snackbar`, or `alert` components exist in `apps/web/src/components/`.

---

## 5. Database: Instance State Tracking

### Table: `tool_instances`
**File:** `apps/api/src/models/tool_instance.py`

| Column | Purpose |
|---|---|
| `status` | `pending → building → starting → probing → running → unhealthy → stopped → error` |
| `container_id` | Docker container ID (nullable) |
| `container_name` | Docker container name (nullable) |
| `compose_path` | Path to `docker-compose.yml` |
| `port` | Host port mapped to container |
| `url` / `public_url` | Cloudflare tunnel URL |
| `tunnel_id` | cloudflared PID |
| `last_started_at` / `last_stopped_at` | Timestamps |
| `probe_result` | JSON blob with last probe outcome |
| `selected_config_profile_id` | FK to config profile |

### Migrations
- `0006_tool_instances.py` — base table with `status`, `container_id`, `url`, `port`.
- `0007_instance_container_name.py` — adds `container_name`.
- `0011_tool_instance_tunnel_fields.py` — adds `public_url`, `tunnel_id`.
- `0013_add_probe_result.py` — adds `probe_result` (JSON).

### Gaps
- **No `health_history` table** — can't track uptime, downtime, or flapping.
- **No `instance_events` table** — no audit log of state transitions.
- **No `notification_preferences` or `user_notifications` table**.

---

## 6. Key Files and Their Roles

| File | Role |
|---|---|
| `apps/api/src/services/docker.py` | All Docker CLI interactions: compose up/down, container status/logs, tunnel management, port finding. |
| `apps/api/src/api/tool_instances.py` | CRUD + lifecycle endpoints for instances (create, start, stop, restart, delete, health, logs, proxy, tunnel recreate). |
| `apps/api/src/services/terminal_manager.py` | In-memory session registry + 60s idle cleanup loop. Pattern to emulate for container monitoring. |
| `apps/api/src/services/readiness_probe.py` | `execute_probe()` — runs a command inside a container with retry logic. |
| `apps/api/src/logging_config.py` | Plain-text logging setup, request/response middleware, exception middleware. |
| `apps/api/src/models/tool_instance.py` | SQLAlchemy model for `tool_instances` table. |
| `apps/api/src/models/tool_type.py` | SQLAlchemy model for `tool_types`, includes `readiness_probe` JSON config. |
| `apps/api/src/api/health.py` | System health endpoint (DB + disk), **not** per-instance health. |
| `apps/web/src/components/instance-list.tsx` | Displays instances with status dots, polls health every 30s, inline tunnel error badges. |
| `apps/web/src/components/session-list.tsx` | Grouped list of sessions (active vs recent), receives `tunnelHealth` prop. |
| `apps/web/src/components/session-card.tsx` | Card UI with status badges, stop/delete confirm, tunnel error display. |
| `apps/web/src/api/sessions.ts` | API client for instance CRUD and `checkInstanceHealth()`. |

---

## 7. Risks and Unknowns

1. **Docker CLI dependency** — All container operations shell out to `docker` / `docker compose`. No Docker SDK or lib used. This is slow and brittle under load.
2. **Tunnel PID fragility** — `tunnel_id` is a process ID string. If the API restarts, PIDs are lost and tunnels may leak.
3. **No instance-level auto-restart** — If a container exits (crash, OOM), it stays `error` or `stopped` until a user manually restarts it.
4. **Probe result is a single JSON blob** — Overwritten on every start. No history.
5. **Polling load** — Frontend polls every 30s per running instance. With many users + many instances, this generates significant health-check load.
6. **No auth on WebSocket upgrade** — Terminal WS endpoint may not validate session ownership on connection (not verified in this scout).
7. **Cloudflared process leaks** — If `stop_cloudflared_tunnel` fails or the API crashes, tunnel processes may become orphaned.
8. **Log retention** — `docker logs` is the only source; no rotation or persistence strategy.
9. **Scaling limitation** — In-memory `TerminalManager` and any future in-memory event bus won't work across multiple API replicas.

---

## 8. Recommended Architecture Approach

### Short-Term (MVP): In-Memory Events + Server-Sent Events (SSE)

**Rationale:**
- The project already uses FastAPI. SSE is natively supported and simpler than WebSockets for one-way server→client push.
- No new infrastructure (message queue) needed.
- Matches the existing polling use case but eliminates 30s latency.

**Components:**
1. **InstanceEventBus** (in-memory singleton, similar to `TerminalManager`)
   - Publishes events: `instance.created`, `instance.started`, `instance.stopped`, `instance.health_changed`, `instance.error`.
2. **Background Monitor Task** (asyncio loop, like `TerminalManager._idle_check_loop`)
   - Every 10–30s, inspect running containers and tunnels.
   - On state change, update DB + publish event to bus.
3. **SSE Endpoint** (`GET /events`)
   - Stream JSON events to connected clients.
   - Frontend subscribes once, receives real-time updates.
4. **Frontend Toast Layer**
   - New lightweight toast component subscribed to SSE.
   - Shows notifications for errors, tunnel failures, successful starts.

### Medium-Term: Persistent Event Log + Health History

1. **`instance_events` table** — append-only audit log of all lifecycle transitions.
2. **`health_checks` table** — periodic snapshots of container + tunnel health for trend analysis.
3. **Structured logging** — Switch to JSON format; include `instance_id`, `event_type`, `correlation_id`.

### Long-Term: Message Queue (if multi-replica)

- If the API needs to scale horizontally, replace in-memory bus with **Redis Pub/Sub** or **NATS**.
- Background monitor becomes a separate worker process or scheduled task.

### Decision Summary

| Concern | Recommended Path |
|---|---|
| Real-time status updates | **SSE** from in-memory event bus |
| Health monitoring | Background asyncio task polling Docker + tunnels |
| User notifications | Lightweight toast component fed by SSE |
| Audit / history | New `instance_events` and `health_checks` tables |
| Log aggregation | JSON structured logs + optional log shipping |
| Multi-replica safety | Deferred to future; add Redis/NATS when needed |