headquarter/openspec/changes/container-monitoring-notifications/proposal.md

SDD Proposal: Container Monitoring & Notification System

Status

Phase: proposal
Date: 2026-05-28
Owner: Gentle AI
Based on: Exploration container-monitoring-notifications

---

1. Problem Statement

Users start containers via docker compose, but when something goes wrong — a build error, a missing container, a crashed process, a failed tunnel — there is zero visibility. Failures are buried in server logs. The only hint is a generic 4004 error in the terminal or a stale status badge that only updates when the frontend happens to poll (every 30 seconds).

Current pain points:

• Silent failures: A container exits or a tunnel dies and the user doesn't know until they manually refresh.
• No push notifications: The frontend polls every 30s; status changes have up to 30s latency.
• No lifecycle audit trail: There's no record of when a container started, stopped, or crashed.
• No health history: The probe_result JSON blob is overwritten on every restart — no trend data.
• Ephemeral logs: Container stdout/stderr is only available via docker logs on-demand; nothing is persisted.

This gap was surfaced by the terminal feature: when containers fail to build or start, the terminal shows a 4004 error with no explanation, leaving users stuck.

---

2. Goals

1. Real-time status push: Users see container lifecycle events (start, stop, error, health change) within seconds, not 30s.
2. Proactive health monitoring: Background task continuously monitors running containers and tunnels, not just at startup.
3. User notifications: Toast / alert notifications when containers fail, crash, or become unhealthy.
4. Event audit trail: Append-only log of all instance lifecycle transitions.
5. Health history: Time-series snapshots of container + tunnel health for debugging trends.
6. Structured logging: JSON logs with correlation IDs and instance IDs for traceability.

---

3. Non-Goals

• Auto-restart of crashed containers (out of scope for MVP; may be added later).
• Multi-replica API support (in-memory event bus is sufficient for now; Redis/NATS deferred).
• Log aggregation / shipping to external systems (e.g., Loki, ELK — structured JSON logs only).
• Email / SMS / Slack notifications (in-app toast only for MVP).
• Container resource metrics (CPU, memory, disk — Docker stats not in scope).
• Replacing docker CLI with Docker SDK (keep existing shell-out pattern).

---

4. User Stories

US-MON-001: Container Start Notification

As a user, when I start a container, I want to see a "Container starting..." toast so I know the system is working, followed by a "Container running" toast when it's ready.

US-MON-002: Build Failure Alert

As a user, when a container fails to build or start, I want an immediate toast with the error message and exit code so I don't have to dig through server logs.

US-MON-003: Tunnel Failure Detection

As a user, when a Cloudflare tunnel dies while my container is running, I want a real-time notification so I can restart it.

US-MON-004: Health Status History

As a user, when my container is flapping between healthy and unhealthy, I want to see a history of health checks to diagnose the issue.

US-MON-005: Lifecycle Audit

As a platform operator, I want an audit log of who started/stopped/restarted which container and when, for troubleshooting and accountability.

---

5. Proposed Solution

Architecture Overview

┌─────────────────┐     SSE      ┌──────────────────┐
│  Frontend       │◄─────────────│  FastAPI         │
│  (toast +       │   events     │  SSE endpoint    │
│   status badges)│              │  /events/stream  │
└─────────────────┘              └────────┬─────────┘
                                          │
                              ┌───────────┴───────────┐
                              │   InstanceEventBus    │
                              │   (in-memory)         │
                              └───────────┬───────────┘
                                          │ publish
                    ┌─────────────────────┼─────────────────────┐
                    │                     │                     │
           ┌────────▼────────┐   ┌───────▼────────┐   ┌────────▼────────┐
           │ Lifecycle hooks │   │ Health Monitor │   │  Instance CRUD  │
           │ (start/stop/    │   │ (asyncio loop) │   │  (create/delete)│
           │  restart/delete)│   │                │   │                 │
           └─────────────────┘   └───────┬────────┘   └─────────────────┘
                                         │
                              ┌──────────▼──────────┐
                              │  Docker + Tunnel    │
                              │  (poll every 15s)   │
                              └─────────────────────┘

Components

5.1 InstanceEventBus (in-memory singleton)

• Pattern: Same singleton style as TerminalManager.
• Publishes typed events: instance.created, instance.started, instance.stopped, instance.health_changed, instance.error.
• Subscribers: SSE endpoint broadcasts to connected clients; health monitor subscribes for its own coordination.

5.2 Background Health Monitor

• Pattern: asyncio loop, modeled after TerminalManager._idle_check_loop (every 60s → every 15s).
• For each running instance:
  1. Call docker inspect for container status + exit code.
  2. For web tools, curl the public URL for tunnel health.
  3. Compare with last known state.
  4. On change: update DB status, write health_checks row, publish event to bus.
• On container crash/OOM: publish instance.error with exit code and stderr snippet.

5.3 SSE Endpoint

GET /events/stream

• FastAPI StreamingResponse with text/event-stream.
• Authenticated (same cookie/JWT as existing API).
• Sends JSON event payload per line.
• Frontend reconnects with exponential backoff on disconnect.

5.4 Frontend Toast Layer

• New lightweight toast component (e.g., sonner or custom).
• Single SSE connection on app mount.
• Filters events by relevance (errors always shown; start/stop shown briefly).
• Also updates instance status badges in real-time (no more 30s polling lag).

5.5 Database Additions

New table: instance_events — append-only audit log

id UUID PK
instance_id UUID FK → tool_instances.id ON DELETE CASCADE
event_type VARCHAR(50)  -- created, started, stopped, restarted, deleted, health_changed, error
status VARCHAR(50)      -- snapshot of instance status at time of event
message TEXT            -- human-readable description / error message
created_by UUID FK → users.id
metadata JSONB          -- exit_code, probe_output, tunnel_url, etc.
created_at TIMESTAMPTZ DEFAULT now()

New table: health_checks — periodic health snapshots

id UUID PK
instance_id UUID FK → tool_instances.id ON DELETE CASCADE
container_status VARCHAR(50)   -- running, exited, dead, etc.
container_healthy BOOLEAN
tunnel_healthy BOOLEAN
exit_code INT
probe_status VARCHAR(50)
probe_output TEXT
checked_at TIMESTAMPTZ DEFAULT now()

5.6 Structured Logging

• Switch API container logs to JSON format.
• Fields: timestamp, level, logger, message, instance_id, event_type, correlation_id.
• Container stdout/stderr remains in Docker; we do not duplicate it.

---

6. Key Decisions

Decision	Choice	Rationale
Event transport	SSE (not WebSocket)	One-way server→client push is all we need. SSE is simpler, uses HTTP, works through proxies, and FastAPI supports it natively. WebSocket is overkill and only used for terminal bidirectional streams.
Event bus	In-memory (not Redis/NATS)	No new infrastructure. Single API process assumption holds today. TerminalManager already uses in-memory state. Defer distributed bus to when horizontal scaling is needed.
Health monitoring	Background asyncio poll (not Docker events API)	Docker CLI events API requires a persistent stream and is tricky with shell-outs. A simple poll loop every 15s is predictable, testable, and matches our existing docker inspect usage.
Frontend polling	Eliminate for status (keep for list refresh)	Instance list may still poll occasionally, but status changes and errors push via SSE. Reduces server load and gives instant UX.
Notification scope	In-app toast only	No external integrations for MVP. Keeps scope tight. Toast library (e.g., sonner) is a small dependency.
Log persistence	Structured JSON to stdout only	We do not build a log storage system. Docker already retains container logs. Our structured API logs can be shipped later if needed.
Auto-restart	Out of scope	Detect and notify, but do not automatically restart crashed containers. User must explicitly restart to avoid surprise side effects.

---

7. Risks

Risk	Severity	Likelihood	Mitigation
Docker CLI brittleness under load	Medium	Medium	Keep poll interval conservative (15s). Reuse existing docker.py service; do not add new CLI patterns. Monitor execute_compose_command latency.
SSE connection leaks	Medium	Low	Use FastAPI background task cleanup. Close stream on client disconnect. Limit max connections per user (e.g., 5).
Memory growth from event bus	Low	Low	Event bus holds only subscriber references, not event history. Health monitor does not retain old check results.
Tunnel PID fragility	High	High	Existing risk, not introduced by this change. Health monitor will at least detect leaked/orphaned tunnels and surface them.
Frontend SSE reconnect storms	Medium	Low	Exponential backoff on reconnect. Jitter to prevent thundering herd.
Database write amplification	Medium	Medium	Health checks every 15s × N running instances. Write only on state change, not every poll. health_checks table may grow; add retention policy (e.g., 30 days) in follow-up.
Scope creep into full observability	High	Medium	Explicitly exclude metrics dashboards, log storage, alerting rules, and PagerDuty-style on-call. Stay focused on lifecycle events + toast.
Multi-replica incompatibility	Low	Low	Document that in-memory bus won't work across replicas. Add Redis/NATS only when scaling need is proven.

---

8. Acceptance Criteria

• AC-1: POST /instances/{id}/start publishes instance.started event; frontend shows "Container starting..." toast.
• AC-2: If container fails during start (exit code ≠ 0), instance.error event is published within 5s; frontend shows error toast with message + exit code.
• AC-3: Background health monitor runs every 15s and detects container crashes, OOMs, and tunnel failures.
• AC-4: On health state change (e.g., running → unhealthy), instance.health_changed event pushes via SSE and updates status badge without page refresh.
• AC-5: instance_events table records every lifecycle transition with event_type, status, message, and created_by.
• AC-6: health_checks table records a row on every state change (not every poll) with container_status, tunnel_healthy, exit_code, checked_at.
• AC-7: Frontend establishes one SSE connection on app load and receives events for all user's instances.
• AC-8: API logs are emitted in JSON format with instance_id, event_type, and correlation_id fields.
• AC-9: No regression in existing terminal WebSocket, instance CRUD, or tunnel functionality.
• AC-10: Backend tests cover event bus publish/subscribe, health monitor state transitions, and SSE endpoint auth.

---

Effort Estimate

Phase	Files	Lines (est)	Complexity
DB migrations + models (instance_events, health_checks)	3	150	Low
InstanceEventBus backend	2	200	Low
Health monitor background task	2	300	Medium
SSE endpoint + auth	2	200	Medium
Lifecycle hook instrumentation	3	200	Low
Frontend toast component + SSE client	4	400	Medium
Real-time status badge updates	3	150	Low
Structured logging refactor	2	100	Low
Tests	4	400	Medium
Total	25	~2100	Medium

Review workload forecast: ~2100 lines exceeds the 400-line budget. Recommend chained PRs:

1. Backend core: Event bus, health monitor, DB migrations, SSE endpoint (~1000 lines)
2. Frontend: Toast component, SSE client, real-time badge updates (~700 lines)
3. Integration + logging: Structured JSON logs, lifecycle hooks, tests (~400 lines)

---

Next Recommended Phase

Design — Detail the InstanceEventBus interface, health monitor state machine, SSE payload schema, and toast UX behavior. Then proceed to tasks.md for implementation breakdown.