# SDD Proposal: Container Monitoring & Notification System

## Status
**Phase:** proposal  
**Date:** 2026-05-28  
**Owner:** Gentle AI  
**Based on:** Exploration `container-monitoring-notifications`

---

## 1. Problem Statement

Users start containers via docker compose, but when something goes wrong — a build error, a missing container, a crashed process, a failed tunnel — there is **zero visibility**. Failures are buried in server logs. The only hint is a generic 4004 error in the terminal or a stale status badge that only updates when the frontend happens to poll (every 30 seconds).

Current pain points:
- **Silent failures**: A container exits or a tunnel dies and the user doesn't know until they manually refresh.
- **No push notifications**: The frontend polls every 30s; status changes have up to 30s latency.
- **No lifecycle audit trail**: There's no record of when a container started, stopped, or crashed.
- **No health history**: The `probe_result` JSON blob is overwritten on every restart — no trend data.
- **Ephemeral logs**: Container stdout/stderr is only available via `docker logs` on-demand; nothing is persisted.

This gap was surfaced by the terminal feature: when containers fail to build or start, the terminal shows a 4004 error with no explanation, leaving users stuck.

---

## 2. Goals

1. **Real-time status push**: Users see container lifecycle events (start, stop, error, health change) within seconds, not 30s.
2. **Proactive health monitoring**: Background task continuously monitors running containers and tunnels, not just at startup.
3. **User notifications**: Toast / alert notifications when containers fail, crash, or become unhealthy.
4. **Event audit trail**: Append-only log of all instance lifecycle transitions.
5. **Health history**: Time-series snapshots of container + tunnel health for debugging trends.
6. **Structured logging**: JSON logs with correlation IDs and instance IDs for traceability.

---

## 3. Non-Goals

- **Auto-restart of crashed containers** (out of scope for MVP; may be added later).
- **Multi-replica API support** (in-memory event bus is sufficient for now; Redis/NATS deferred).
- **Log aggregation / shipping to external systems** (e.g., Loki, ELK — structured JSON logs only).
- **Email / SMS / Slack notifications** (in-app toast only for MVP).
- **Container resource metrics** (CPU, memory, disk — Docker stats not in scope).
- **Replacing docker CLI with Docker SDK** (keep existing shell-out pattern).

---

## 4. User Stories

### US-MON-001: Container Start Notification
> As a user, when I start a container, I want to see a "Container starting..." toast so I know the system is working, followed by a "Container running" toast when it's ready.

### US-MON-002: Build Failure Alert
> As a user, when a container fails to build or start, I want an immediate toast with the error message and exit code so I don't have to dig through server logs.

### US-MON-003: Tunnel Failure Detection
> As a user, when a Cloudflare tunnel dies while my container is running, I want a real-time notification so I can restart it.

### US-MON-004: Health Status History
> As a user, when my container is flapping between healthy and unhealthy, I want to see a history of health checks to diagnose the issue.

### US-MON-005: Lifecycle Audit
> As a platform operator, I want an audit log of who started/stopped/restarted which container and when, for troubleshooting and accountability.

---

## 5. Proposed Solution

### Architecture Overview

```
┌─────────────────┐     SSE      ┌──────────────────┐
│  Frontend       │◄─────────────│  FastAPI         │
│  (toast +       │   events     │  SSE endpoint    │
│   status badges)│              │  /events/stream  │
└─────────────────┘              └────────┬─────────┘
                                          │
                              ┌───────────┴───────────┐
                              │   InstanceEventBus    │
                              │   (in-memory)         │
                              └───────────┬───────────┘
                                          │ publish
                    ┌─────────────────────┼─────────────────────┐
                    │                     │                     │
           ┌────────▼────────┐   ┌───────▼────────┐   ┌────────▼────────┐
           │ Lifecycle hooks │   │ Health Monitor │   │  Instance CRUD  │
           │ (start/stop/    │   │ (asyncio loop) │   │  (create/delete)│
           │  restart/delete)│   │                │   │                 │
           └─────────────────┘   └───────┬────────┘   └─────────────────┘
                                         │
                              ┌──────────▼──────────┐
                              │  Docker + Tunnel    │
                              │  (poll every 15s)   │
                              └─────────────────────┘
```

### Components

#### 5.1 InstanceEventBus (in-memory singleton)
- Pattern: Same singleton style as `TerminalManager`.
- Publishes typed events: `instance.created`, `instance.started`, `instance.stopped`, `instance.health_changed`, `instance.error`.
- Subscribers: SSE endpoint broadcasts to connected clients; health monitor subscribes for its own coordination.

#### 5.2 Background Health Monitor
- Pattern: `asyncio` loop, modeled after `TerminalManager._idle_check_loop` (every 60s → every 15s).
- For each running instance:
  1. Call `docker inspect` for container status + exit code.
  2. For web tools, curl the public URL for tunnel health.
  3. Compare with last known state.
  4. On change: update DB `status`, write `health_checks` row, publish event to bus.
- On container crash/OOM: publish `instance.error` with exit code and stderr snippet.

#### 5.3 SSE Endpoint
```
GET /events/stream
```
- FastAPI `StreamingResponse` with `text/event-stream`.
- Authenticated (same cookie/JWT as existing API).
- Sends JSON event payload per line.
- Frontend reconnects with exponential backoff on disconnect.

#### 5.4 Frontend Toast Layer
- New lightweight toast component (e.g., `sonner` or custom).
- Single SSE connection on app mount.
- Filters events by relevance (errors always shown; start/stop shown briefly).
- Also updates instance status badges in real-time (no more 30s polling lag).

#### 5.5 Database Additions

**New table: `instance_events`** — append-only audit log
```
id UUID PK
instance_id UUID FK → tool_instances.id ON DELETE CASCADE
event_type VARCHAR(50)  -- created, started, stopped, restarted, deleted, health_changed, error
status VARCHAR(50)      -- snapshot of instance status at time of event
message TEXT            -- human-readable description / error message
created_by UUID FK → users.id
metadata JSONB          -- exit_code, probe_output, tunnel_url, etc.
created_at TIMESTAMPTZ DEFAULT now()
```

**New table: `health_checks`** — periodic health snapshots
```
id UUID PK
instance_id UUID FK → tool_instances.id ON DELETE CASCADE
container_status VARCHAR(50)   -- running, exited, dead, etc.
container_healthy BOOLEAN
tunnel_healthy BOOLEAN
exit_code INT
probe_status VARCHAR(50)
probe_output TEXT
checked_at TIMESTAMPTZ DEFAULT now()
```

#### 5.6 Structured Logging
- Switch API container logs to JSON format.
- Fields: `timestamp`, `level`, `logger`, `message`, `instance_id`, `event_type`, `correlation_id`.
- Container stdout/stderr remains in Docker; we do not duplicate it.

---

## 6. Key Decisions

| Decision | Choice | Rationale |
|----------|--------|-----------|
| **Event transport** | **SSE** (not WebSocket) | One-way server→client push is all we need. SSE is simpler, uses HTTP, works through proxies, and FastAPI supports it natively. WebSocket is overkill and only used for terminal bidirectional streams. |
| **Event bus** | **In-memory** (not Redis/NATS) | No new infrastructure. Single API process assumption holds today. TerminalManager already uses in-memory state. Defer distributed bus to when horizontal scaling is needed. |
| **Health monitoring** | **Background asyncio poll** (not Docker events API) | Docker CLI events API requires a persistent stream and is tricky with shell-outs. A simple poll loop every 15s is predictable, testable, and matches our existing `docker inspect` usage. |
| **Frontend polling** | **Eliminate for status** (keep for list refresh) | Instance list may still poll occasionally, but status changes and errors push via SSE. Reduces server load and gives instant UX. |
| **Notification scope** | **In-app toast only** | No external integrations for MVP. Keeps scope tight. Toast library (e.g., `sonner`) is a small dependency. |
| **Log persistence** | **Structured JSON to stdout only** | We do not build a log storage system. Docker already retains container logs. Our structured API logs can be shipped later if needed. |
| **Auto-restart** | **Out of scope** | Detect and notify, but do not automatically restart crashed containers. User must explicitly restart to avoid surprise side effects. |

---

## 7. Risks

| Risk | Severity | Likelihood | Mitigation |
|------|----------|------------|------------|
| **Docker CLI brittleness under load** | Medium | Medium | Keep poll interval conservative (15s). Reuse existing `docker.py` service; do not add new CLI patterns. Monitor `execute_compose_command` latency. |
| **SSE connection leaks** | Medium | Low | Use FastAPI background task cleanup. Close stream on client disconnect. Limit max connections per user (e.g., 5). |
| **Memory growth from event bus** | Low | Low | Event bus holds only subscriber references, not event history. Health monitor does not retain old check results. |
| **Tunnel PID fragility** | High | High | Existing risk, not introduced by this change. Health monitor will at least *detect* leaked/orphaned tunnels and surface them. |
| **Frontend SSE reconnect storms** | Medium | Low | Exponential backoff on reconnect. Jitter to prevent thundering herd. |
| **Database write amplification** | Medium | Medium | Health checks every 15s × N running instances. Write only on state change, not every poll. `health_checks` table may grow; add retention policy (e.g., 30 days) in follow-up. |
| **Scope creep into full observability** | High | Medium | Explicitly exclude metrics dashboards, log storage, alerting rules, and PagerDuty-style on-call. Stay focused on lifecycle events + toast. |
| **Multi-replica incompatibility** | Low | Low | Document that in-memory bus won't work across replicas. Add Redis/NATS only when scaling need is proven. |

---

## 8. Acceptance Criteria

- [ ] **AC-1:** `POST /instances/{id}/start` publishes `instance.started` event; frontend shows "Container starting..." toast.
- [ ] **AC-2:** If container fails during start (exit code ≠ 0), `instance.error` event is published within 5s; frontend shows error toast with message + exit code.
- [ ] **AC-3:** Background health monitor runs every 15s and detects container crashes, OOMs, and tunnel failures.
- [ ] **AC-4:** On health state change (e.g., `running → unhealthy`), `instance.health_changed` event pushes via SSE and updates status badge without page refresh.
- [ ] **AC-5:** `instance_events` table records every lifecycle transition with `event_type`, `status`, `message`, and `created_by`.
- [ ] **AC-6:** `health_checks` table records a row on every state change (not every poll) with `container_status`, `tunnel_healthy`, `exit_code`, `checked_at`.
- [ ] **AC-7:** Frontend establishes one SSE connection on app load and receives events for all user's instances.
- [ ] **AC-8:** API logs are emitted in JSON format with `instance_id`, `event_type`, and `correlation_id` fields.
- [ ] **AC-9:** No regression in existing terminal WebSocket, instance CRUD, or tunnel functionality.
- [ ] **AC-10:** Backend tests cover event bus publish/subscribe, health monitor state transitions, and SSE endpoint auth.

---

## Effort Estimate

| Phase | Files | Lines (est) | Complexity |
|-------|-------|-------------|------------|
| DB migrations + models (`instance_events`, `health_checks`) | 3 | 150 | Low |
| InstanceEventBus backend | 2 | 200 | Low |
| Health monitor background task | 2 | 300 | Medium |
| SSE endpoint + auth | 2 | 200 | Medium |
| Lifecycle hook instrumentation | 3 | 200 | Low |
| Frontend toast component + SSE client | 4 | 400 | Medium |
| Real-time status badge updates | 3 | 150 | Low |
| Structured logging refactor | 2 | 100 | Low |
| Tests | 4 | 400 | Medium |
| **Total** | **25** | **~2100** | **Medium** |

**Review workload forecast:** ~2100 lines exceeds the 400-line budget. Recommend **chained PRs**:
1. **Backend core**: Event bus, health monitor, DB migrations, SSE endpoint (~1000 lines)
2. **Frontend**: Toast component, SSE client, real-time badge updates (~700 lines)
3. **Integration + logging**: Structured JSON logs, lifecycle hooks, tests (~400 lines)

---

## Next Recommended Phase

**Design** — Detail the `InstanceEventBus` interface, health monitor state machine, SSE payload schema, and toast UX behavior. Then proceed to `tasks.md` for implementation breakdown.
