feat: container monitoring backend core (PR-1)

- Add instance_events and health_checks tables with Alembic migration - InstanceEventBus: typed pub/sub singleton with wildcard support - HealthMonitor: async background loop polling containers every 15s - SSE endpoint GET /events/stream with auth and connection limits - Lifecycle hooks in tool_instances.py (create/start/stop/restart/delete) - Structured JSON logging with correlation IDs - 15 new unit tests (EventBus, HealthMonitor, MonitoringModels) Quality gates: pytest 15 new passed, ruff clean
2026-05-28 23:09:27 +02:00
parent 0fdbef578f
commit 4a7f24348c
26 changed files with 4142 additions and 11 deletions
@@ -0,0 +1,790 @@
+# SDD Design: Container Monitoring & Notification System
+
+## Status
+**Phase:** design  
+**Date:** 2026-05-28  
+**Owner:** Gentle AI  
+**Scope:** Cross-cutting (backend + frontend)  
+**Est. Lines:** ~2,100 (recommend 3 chained PRs)  
+
+---
+
+## 1. Component Architecture
+
+### 1.1 InstanceEventBus — In-Memory Singleton Pub/Sub
+
+**Pattern:** Module-level singleton, modeled after `TerminalManager` (`apps/api/src/services/terminal_manager.py`).
+
+**Responsibilities:**
+- Maintain a registry of typed subscribers (`instance.created`, `instance.started`, `instance.stopped`, `instance.restarted`, `instance.deleted`, `instance.health_changed`, `instance.error`).
+- Deliver events to all subscribers in the same asyncio event loop iteration.
+- Catch subscriber exceptions, log them with `correlation_id`, and continue delivery.
+- Provide no persistence or queuing; offline subscribers miss events.
+
+**Class:**
+```python
+class InstanceEventBus:
+    _instance: "InstanceEventBus | None" = None
+    _lock: asyncio.Lock = asyncio.Lock()
+
+    def __new__(cls) -> "InstanceEventBus": ...
+
+    def subscribe(
+        self,
+        event_type: str,
+        callback: Callable[[InstanceEventPayload], Awaitable[None] | None],
+    ) -> Callable[[], None]: ...
+
+    def unsubscribe(self, event_type: str, callback_id: str) -> None: ...
+
+    async def publish(self, event_type: str, payload: InstanceEventPayload) -> None: ...
+```
+
+**Payload type:**
+```python
+class InstanceEventPayload(TypedDict):
+    event: str
+    instance_id: str
+    status: str | None
+    message: str | None
+    metadata: dict[str, Any]
+    timestamp: str          # ISO 8601 UTC
+    correlation_id: str     # UUID
+```
+
+**Location:** `apps/api/src/services/event_bus.py`
+
+---
+
+### 1.2 HealthMonitor — Asyncio Background Task
+
+**Pattern:** Singleton background task, modeled after `TerminalManager._idle_check_loop()`.
+
+**Responsibilities:**
+- Poll every 15 seconds for all instances whose `status` is NOT IN `("pending", "stopped", "error")`.
+- For each candidate:
+  1. Call `docker inspect` via `get_container_status()` in `docker.py`.
+  2. For web tools with `public_url`, perform HTTP HEAD/GET to check tunnel health.
+  3. Compare against last known in-memory state (`_last_known_state: dict[UUID, HealthSnapshot]`).
+- On state change:
+  1. Update `tool_instances.status` in DB.
+  2. Insert row into `health_checks`.
+  3. Publish appropriate event to `InstanceEventBus`.
+- Catch all exceptions per-instance, log structured error, and continue to next instance.
+
+**Class:**
+```python
+class HealthMonitor:
+    def __init__(self, event_bus: InstanceEventBus) -> None: ...
+
+    def start(self) -> None:
+        """Idempotent start of the background polling task."""
+
+    def stop(self) -> None:
+        """Cancel the background task and clear state."""
+
+    async def _poll_loop(self) -> None: ...
+    async def _check_instance(self, session: AsyncSession, instance: ToolInstance) -> None: ...
+    async def _publish_state_change(
+        self,
+        instance: ToolInstance,
+        previous: HealthSnapshot,
+        current: HealthSnapshot,
+    ) -> None: ...
+```
+
+**Location:** `apps/api/src/services/health_monitor.py`
+
+---
+
+### 1.3 SSEManager — FastAPI StreamingResponse
+
+**Pattern:** Stateless generator endpoint that bridges `InstanceEventBus` to HTTP `text/event-stream`.
+
+**Responsibilities:**
+- Authenticate via existing cookie/JWT (`get_current_user_id`).
+- Return `401` before starting stream if auth fails.
+- Subscribe a per-connection async callback to `InstanceEventBus`.
+- Yield SSE `data:` lines formatted as JSON.
+- Send SSE comment `:ping` every 30 seconds to keep proxies alive.
+- On disconnect (`asyncio.CancelledError` / client close), unsubscribe and release.
+- Enforce max 5 concurrent SSE connections per user.
+
+**Endpoint:**
+```python
+@router.get("/events/stream")
+async def events_stream(
+    request: Request,
+    user_id: uuid.UUID = Depends(get_current_user_id),
+) -> StreamingResponse:
+    ...
+```
+
+**Location:** `apps/api/src/api/events.py`
+
+---
+
+### 1.4 LifecycleHookService — Instrumentation Points
+
+**Responsibilities:**
+- Thin wrapper around existing lifecycle endpoints in `tool_instances.py`.
+- At each lifecycle action (create, start, stop, restart, delete), publish the corresponding typed event **after** the DB transaction commits.
+- Record an `instance_events` audit row for every transition.
+- Pass `created_by` (current user ID) for user-initiated actions; `NULL` for system-detected transitions.
+
+**Integration points (all in `apps/api/src/api/tool_instances.py`):**
+
+| Endpoint | Event Published | Status | Audit Row |
+|----------|----------------|--------|-----------|
+| `POST /instances` | `instance.created` | `"pending"` | Yes |
+| `POST /instances/{id}/start` | `instance.started` | `"starting"` | Yes |
+| Probe success | `instance.health_changed` | `"running"` | Yes |
+| Container exits during start | `instance.error` | `"error"` | Yes |
+| `POST /instances/{id}/stop` | `instance.stopped` | `"stopped"` | Yes |
+| `POST /instances/{id}/restart` | `instance.restarted` | `"starting"` | Yes |
+| `DELETE /instances/{id}` | `instance.deleted` | `"deleted"` | Yes |
+
+**Helper:** `LifecycleHookService` class or module-level async functions in `apps/api/src/services/lifecycle_hooks.py`.
+
+---
+
+### 1.5 ToastComponent — Frontend Event Consumer
+
+**Responsibilities:**
+- Single global `<Toaster />` component mounted in `AppShell`.
+- Subscribes to SSE via `useEvents()` hook.
+- Filters incoming events and maps to toast rules:
+  - `instance.error` → error toast, persistent (min 10s).
+  - `instance.started` → info toast, 3s.
+  - `instance.health_changed` → `running` = success 3s; `unhealthy` = warning 5s.
+- Deduplicates toasts for same `(instance_id, event_type)` within 1s.
+- Exposes a `toast.dismiss(id)` API.
+
+**Technology choice:** `sonner` (lightweight, headless-compatible) or a custom 150-line toast stack. **Decision:** Use `sonner` to minimize custom UI code.
+
+**Locations:**
+- `apps/web/src/components/toast-provider.tsx` — wraps `Toaster` + `useEvents`.
+- `apps/web/src/components/toast-rules.ts` — event-to-toast mapping logic.
+
+---
+
+## 2. File Structure
+
+### New Files
+
+| File | Purpose |
+|------|---------|
+| `apps/api/src/services/event_bus.py` | `InstanceEventBus` singleton + `InstanceEventPayload` type |
+| `apps/api/src/services/health_monitor.py` | `HealthMonitor` background task + `HealthSnapshot` dataclass |
+| `apps/api/src/services/lifecycle_hooks.py` | Helper functions to publish lifecycle events and write audit rows |
+| `apps/api/src/services/correlation.py` | Async context var `CORRELATION_ID` + middleware injection |
+| `apps/api/src/api/events.py` | SSE endpoint `/events/stream` + connection limiter |
+| `apps/api/src/models/instance_event.py` | SQLAlchemy `InstanceEvent` model |
+| `apps/api/src/models/health_check.py` | SQLAlchemy `HealthCheck` model |
+| `apps/api/alembic/versions/2026_05_28_add_monitoring_tables.py` | Alembic revision creating `instance_events` + `health_checks` + indexes |
+| `apps/web/src/hooks/use-events.ts` | `useEvents()` hook: SSE connect, reconnect backoff, event parsing |
+| `apps/web/src/components/toast-provider.tsx` | Global toast provider consuming SSE events |
+| `apps/web/src/components/toast-rules.ts` | Event-to-toast mapping and deduplication logic |
+| `apps/web/src/types/events.ts` | TypeScript `InstanceEventPayload` interface |
+| `tests/unit/test_event_bus.py` | EventBus pub/sub, exception isolation, unsubscribe |
+| `tests/unit/test_health_monitor.py` | State transition logic, DB write gating |
+| `tests/integration/test_sse_endpoint.py` | SSE auth, streaming, disconnect cleanup |
+
+### Modified Files
+
+| File | Purpose |
+|------|---------|
+| `apps/api/src/api/tool_instances.py` | Inject lifecycle hook calls at create/start/stop/restart/delete; pass `correlation_id` through async context |
+| `apps/api/src/main.py` | Import `events_router`; register at startup; start `HealthMonitor`; add `CorrelationIdMiddleware` |
+| `apps/api/src/logging_config.py` | Replace plain-text formatter with JSON formatter; include `correlation_id`, `instance_id`, `event_type` fields |
+| `apps/api/src/models/__init__.py` | Export `InstanceEvent`, `HealthCheck` for Alembic autogenerate |
+| `apps/web/src/components/instance-list.tsx` | Remove 30s health polling; consume `useEvents` for real-time badge updates; retain 60s list refresh |
+| `apps/web/src/components/session-card.tsx` | Update badge colors based on SSE `status` events |
+| `apps/web/src/components/app-shell.tsx` | Mount `<ToastProvider />` |
+| `apps/web/src/api/sessions.ts` | Remove `checkInstanceHealth` polling call (keep function for on-demand use) |
+| `apps/web/package.json` | Add `sonner` dependency |
+| `tests/conftest.py` (or api equivalent) | Add `event_bus` fixture and `health_monitor` fixture for tests |
+
+---
+
+## 3. Interface Design
+
+### 3.1 EventBus
+
+```python
+# apps/api/src/services/event_bus.py
+
+class InstanceEventBus:
+    """In-memory typed event bus. Singleton per process."""
+
+    def subscribe(
+        self,
+        event_type: str,
+        callback: Callable[[InstanceEventPayload], Awaitable[None] | None],
+    ) -> Callable[[], None]:
+        """Register a callback for an event type. Returns an unsubscribe function."""
+
+    async def publish(self, event_type: str, payload: InstanceEventPayload) -> None:
+        """Deliver payload to all subscribers of event_type."""
+
+    def unsubscribe_all(self, event_type: str) -> None:
+        """Remove all subscribers for an event type (used in tests)."""
+```
+
+**Usage in SSE endpoint:**
+```python
+async def event_generator(user_id: uuid.UUID):
+    queue: asyncio.Queue[InstanceEventPayload] = asyncio.Queue()
+
+    async def on_event(payload: InstanceEventPayload) -> None:
+        await queue.put(payload)
+
+    unsubscribe = event_bus.subscribe("*", on_event)  # or per-type
+    try:
+        while True:
+            payload = await asyncio.wait_for(queue.get(), timeout=30.0)
+            yield f"event: {payload['event']}\ndata: {json.dumps(payload)}\n\n"
+    finally:
+        unsubscribe()
+```
+
+### 3.2 HealthMonitor
+
+```python
+# apps/api/src/services/health_monitor.py
+
+class HealthMonitor:
+    POLL_INTERVAL_SECONDS: float = 15.0
+    MAX_STARTUP_WAIT_SECONDS: float = 30.0
+
+    def __init__(self, event_bus: InstanceEventBus) -> None: ...
+
+    def start(self) -> None:
+        """Idempotent. Creates `asyncio.Task` for `_poll_loop`."""
+
+    def stop(self) -> None:
+        """Cancel task and clear `_last_known_state`."""
+
+    async def force_check(self, instance_id: uuid.UUID) -> None:
+        """Immediate check for a single instance (used in tests)."""
+```
+
+### 3.3 SSEManager
+
+```python
+# apps/api/src/api/events.py
+
+@router.get("/events/stream")
+async def events_stream(
+    request: Request,
+    user_id: uuid.UUID = Depends(get_current_user_id),
+) -> StreamingResponse:
+    ...
+```
+
+**Headers returned:**
+- `Content-Type: text/event-stream`
+- `Cache-Control: no-cache`
+- `Connection: keep-alive`
+- `X-Accel-Buffering: no` (disable nginx buffering)
+
+**Rate limit:** Max 5 concurrent connections per `user_id`. Return `429` if exceeded.
+
+### 3.4 Frontend: useEvents() Hook
+
+```typescript
+// apps/web/src/hooks/use-events.ts
+
+export interface UseEventsReturn {
+  events: InstanceEventPayload[];
+  connected: boolean;
+  reconnectCount: number;
+  error: Error | null;
+}
+
+export function useEvents(): UseEventsReturn {
+  // Establishes SSE connection to `${BASE_URL}/events/stream`
+  // with exponential backoff reconnect.
+}
+```
+
+**Reconnect strategy (client-side):**
+- Initial delay: `1000ms`
+- Multiplier: `2×`
+- Cap: `30000ms`
+- Jitter: `±20%` (`delay * (0.8 + Math.random() * 0.4)`)
+- Max reconnect attempts: unlimited (persistent connection)
+
+### 3.5 Correlation ID Propagation
+
+```python
+# apps/api/src/services/correlation.py
+
+import contextvars
+
+CORRELATION_ID: contextvars.ContextVar[str] = contextvars.ContextVar("correlation_id")
+
+def get_correlation_id() -> str:
+    try:
+        return CORRELATION_ID.get()
+    except LookupError:
+        return str(uuid.uuid4())
+```
+
+**Middleware:** `CorrelationIdMiddleware` reads `X-Request-ID` header or generates new UUID, sets `CORRELATION_ID`, and includes it in all logs via a custom `logging.Filter`.
+
+---
+
+## 4. Data Flow Diagrams
+
+### 4.1 Container Start Flow
+
+```
+User clicks Start
+    │
+    ▼
+POST /instances/{id}/start
+    │
+    ├──► DB: tool_instances.status = "starting"
+    │
+    ├──► LifecycleHookService.publish("instance.started", {status: "starting", ...})
+    │         │
+    │         ▼
+    │    InstanceEventBus
+    │         │
+    │         ├──► SSEManager ──► Frontend toast: "Container starting..."
+    │         │
+    │         └──► InstanceEvent DB write (audit)
+    │
+    ├──► docker compose up -d
+    │
+    ├──► wait_for_container_running()
+    │         │
+    │         ├──► Success ──► DB.status = "running"
+    │         │                  LifecycleHookService.publish("instance.health_changed",
+    │         │                    {status: "running", previous_status: "starting"})
+    │         │                         │
+    │         │                         ▼
+    │         │                    Frontend toast: "Container running"
+    │         │
+    │         └──► Failure ──► DB.status = "error"
+    │                            LifecycleHookService.publish("instance.error",
+    │                              {status: "error", metadata: {exit_code, ...}})
+    │                                   │
+    │                                   ▼
+    │                              Frontend toast: Error (persistent)
+```
+
+### 4.2 Health Monitor Flow
+
+```
+HealthMonitor._poll_loop() (every 15s)
+    │
+    ├──► SELECT * FROM tool_instances WHERE status NOT IN ("pending","stopped","error")
+    │
+    ├──► For each instance:
+    │         │
+    │         ├──► get_container_status(container_id) ──► {State.Status, ExitCode, Health.Status}
+    │         │
+    │         ├──► if public_url: HTTP HEAD public_url ──► tunnel_healthy?
+    │         │
+    │         ├──► Compare with _last_known_state[instance_id]
+    │         │
+    │         ├──► If changed:
+    │         │         │
+    │         │         ├──► DB: UPDATE tool_instances SET status = ?
+    │         │         │
+    │         │         ├──► DB: INSERT INTO health_checks (...)
+    │         │         │
+    │         │         └──► EventBus.publish("instance.health_changed" OR "instance.error")
+    │         │                   │
+    │         │                   ▼
+    │         │              Frontend badge + toast update
+    │         │
+    │         └──► If unchanged: skip DB writes
+    │
+    └──► Catch exception per-instance ──► structured JSON log ──► continue next instance
+```
+
+### 4.3 SSE Flow
+
+```
+Frontend mount
+    │
+    ▼
+EventSource.open("GET /events/stream")
+    │
+    ├──► Server: auth cookie validation
+    │         │
+    │         ├──► Invalid ──► 401 (no stream)
+    │         │
+    │         └──► Valid ──► check connection count ≤ 5
+    │                    │
+    │                    ├──► Exceeded ──► 429
+    │                    │
+    │                    └──► OK ──► StreamingResponse
+    │                              │
+    │                              ├──► Subscribe callback to EventBus
+    │                              │
+    │                              ├──► yield "event: ...\ndata: {...}\n\n"
+    │                              │
+    │                              ├──► yield ":ping\n" (every 30s)
+    │                              │
+    │                              └──► Client disconnect
+    │                                    │
+    │                                    ├──► asyncio.CancelledError
+    │                                    └──► Unsubscribe callback
+    │
+    └──► Network interruption ──► Frontend closes EventSource
+              │
+              ├──► wait exponential backoff + jitter
+              │
+              └──► reopen EventSource (repeat from top)
+```
+
+---
+
+## 5. State Machine
+
+### 5.1 Instance Status Transitions
+
+```
+         +-----------+
+         |  pending  |
+         +-----+-----+
+               │ create()
+               v
+         +-----------+     build/compose failure     +-------+
+         | starting  +-------------------------------->│ error │
+         +-----+-----+                                 +---+---+
+               │ probe passes / monitor finds running       │ restart()
+               v                                          v
+         +-----------+     crash / OOM / exit ≠ 0    +-----------+
+    +--->|  running  +-------------------------------->│  error    |
+    |    +-----+-----+                                 +-----------+
+    |          │ tunnel/probe fail
+    |          v
+    |    +-----------+     recover (tunnel OK)        +-----------+
+    +----+ unhealthy +-------------------------------->│  running  |
+         +-----+-----+                                 +-----------+
+               │ stop()
+               v
+         +-----------+
+         |  stopped  |
+         +-----------+
+               │ delete()
+               v
+            [gone]
+```
+
+### 5.2 Transition Triggers
+
+| From | To | Trigger | DB Update | Event Published | Audit Row |
+|------|----|---------|-----------|-----------------|-----------|
+| `pending` | `starting` | User clicks Start | Yes | `instance.started` | Yes |
+| `starting` | `running` | Readiness probe passes | Yes | `instance.health_changed` | Yes |
+| `starting` | `error` | Container exits during start | Yes | `instance.error` | Yes |
+| `running` | `unhealthy` | Monitor: tunnel down or probe fail | Yes | `instance.health_changed` | Yes |
+| `running` | `error` | Monitor: container crashed / OOM | Yes | `instance.error` | Yes |
+| `unhealthy` | `running` | Monitor: recovery detected | Yes | `instance.health_changed` | Yes |
+| `running` | `stopped` | User clicks Stop | Yes | `instance.stopped` | Yes |
+| `unhealthy` | `stopped` | User clicks Stop | Yes | `instance.stopped` | Yes |
+| `error` | `starting` | User clicks Restart | Yes | `instance.restarted` | Yes |
+| any | `deleted` | User clicks Delete | Yes (then row removed) | `instance.deleted` | Yes |
+
+**Rule:** The monitor only evaluates instances with `status` in `{"starting", "running", "unhealthy"}`. It does NOT evaluate `pending`, `stopped`, or `error`.
+
+---
+
+## 6. Database Schema
+
+### 6.1 Table: `instance_events`
+
+```sql
+CREATE TABLE instance_events (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    instance_id UUID NOT NULL REFERENCES tool_instances(id) ON DELETE CASCADE,
+    event_type VARCHAR(50) NOT NULL,
+    status VARCHAR(50),
+    message TEXT,
+    created_by UUID REFERENCES users(id) ON DELETE SET NULL,
+    metadata JSONB NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX idx_instance_events_instance_id ON instance_events(instance_id);
+CREATE INDEX idx_instance_events_created_at ON instance_events(created_at DESC);
+CREATE INDEX idx_instance_events_event_type ON instance_events(event_type);
+```
+
+**SQLAlchemy model:**
+```python
+# apps/api/src/models/instance_event.py
+
+class InstanceEvent(UUIDPrimaryKeyMixin, Base):
+    __tablename__ = "instance_events"
+
+    instance_id: Mapped[uuid.UUID] = mapped_column(
+        Uuid(as_uuid=True), ForeignKey("tool_instances.id", ondelete="CASCADE"), nullable=False
+    )
+    event_type: Mapped[str] = mapped_column(String(50), nullable=False)
+    status: Mapped[str | None] = mapped_column(String(50), nullable=True)
+    message: Mapped[str | None] = mapped_column(Text, nullable=True)
+    created_by: Mapped[uuid.UUID | None] = mapped_column(
+        Uuid(as_uuid=True), ForeignKey("users.id", ondelete="SET NULL"), nullable=True
+    )
+    metadata: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False, default=dict)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), server_default=func.now(), nullable=False
+    )
+```
+
+### 6.2 Table: `health_checks`
+
+```sql
+CREATE TABLE health_checks (
+    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+    instance_id UUID NOT NULL REFERENCES tool_instances(id) ON DELETE CASCADE,
+    container_status VARCHAR(50),
+    container_healthy BOOLEAN,
+    tunnel_healthy BOOLEAN,
+    exit_code INT,
+    probe_status VARCHAR(50),
+    probe_output TEXT,
+    checked_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX idx_health_checks_instance_id ON health_checks(instance_id);
+CREATE INDEX idx_health_checks_checked_at ON health_checks(checked_at DESC);
+```
+
+**SQLAlchemy model:**
+```python
+# apps/api/src/models/health_check.py
+
+class HealthCheck(UUIDPrimaryKeyMixin, Base):
+    __tablename__ = "health_checks"
+
+    instance_id: Mapped[uuid.UUID] = mapped_column(
+        Uuid(as_uuid=True), ForeignKey("tool_instances.id", ondelete="CASCADE"), nullable=False
+    )
+    container_status: Mapped[str | None] = mapped_column(String(50), nullable=True)
+    container_healthy: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
+    tunnel_healthy: Mapped[bool | None] = mapped_column(Boolean, nullable=True)
+    exit_code: Mapped[int | None] = mapped_column(Integer, nullable=True)
+    probe_status: Mapped[str | None] = mapped_column(String(50), nullable=True)
+    probe_output: Mapped[str | None] = mapped_column(Text, nullable=True)
+    checked_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), server_default=func.now(), nullable=False
+    )
+```
+
+### 6.3 Migration
+
+**File:** `apps/api/alembic/versions/2026_05_28_add_monitoring_tables.py`
+
+**Dependency:** Depends on the latest existing revision (e.g., `2026_05_28_add_terminal_sessions_table.py` or whichever is `head` at apply time).
+
+**Operations:**
+1. `CREATE TABLE instance_events`
+2. `CREATE TABLE health_checks`
+3. Create all 5 indexes.
+4. No data backfill.
+
+**Rollback:** `op.drop_index(...)`, `op.drop_table("health_checks")`, `op.drop_table("instance_events")`.
+
+---
+
+## 7. Error Handling Strategy
+
+### 7.1 Docker CLI Timeout / Failure
+
+**Where:** `HealthMonitor._check_instance()` calling `get_container_status()` or HTTP tunnel probe.
+
+**Behavior:**
+- Wrap call in `try/except Exception`.
+- Log structured JSON error with `instance_id`, `correlation_id`, `error_type`, `message`.
+- **Do NOT** update `tool_instances.status`.
+- **Do NOT** insert `health_checks` row.
+- **Do NOT** publish event.
+- Continue to next instance in the poll loop.
+
+```python
+try:
+    status = await get_container_status(instance.container_id)
+except Exception as exc:
+    logger.error(
+        "Health check failed",
+        extra={
+            "instance_id": str(instance.id),
+            "correlation_id": get_correlation_id(),
+            "error": str(exc),
+        },
+    )
+    return
+```
+
+### 7.2 SSE Disconnect
+
+**Where:** `events_stream()` generator, proxy/network failure, client close.
+
+**Behavior:**
+- Detect disconnect via `asyncio.CancelledError` or `Starlette` disconnect sentinel.
+- Unsubscribe from `InstanceEventBus` in `finally` block.
+- **Do NOT** log error for normal disconnects (log at `INFO` level only).
+- Release connection slot in per-user counter.
+
+### 7.3 SSE Reconnect Storm
+
+**Where:** Frontend `useEvents()` hook.
+
+**Behavior:**
+- Exponential backoff with jitter (see §3.4).
+- If server returns `429`, add extra 5s penalty before retry.
+- If server returns `401`, stop reconnecting and redirect to login.
+
+### 7.4 Event Bus Subscriber Crash
+
+**Where:** `InstanceEventBus.publish()` iterating callbacks.
+
+**Behavior:**
+- Each callback wrapped in `try/except Exception`.
+- Log error with full payload and `correlation_id`.
+- Continue to next subscriber.
+- Publisher (`publish()` call) is never blocked by a slow/failing subscriber.
+
+```python
+for callback in self._subscribers[event_type]:
+    try:
+        if asyncio.iscoroutinefunction(callback):
+            await callback(payload)
+        else:
+            callback(payload)
+    except Exception:
+        logger.exception("Event subscriber failed", extra={"correlation_id": payload["correlation_id"]})
+```
+
+### 7.5 Auth Failure on SSE
+
+**Where:** `events_stream()` before `StreamingResponse`.
+
+**Behavior:**
+- `get_current_user_id` raises `HTTPException(401)`.
+- FastAPI returns `401 Unauthorized` **before** creating the stream.
+- No `InstanceEventBus` subscription is created.
+- No connection slot is consumed.
+
+---
+
+## 8. Testing Strategy
+
+### 8.1 Unit Tests
+
+| Test | File | What |
+|------|------|------|
+| EventBus publish delivers to all subscribers | `tests/unit/test_event_bus.py` | Register 3 callbacks; publish; assert all called with correct payload |
+| EventBus subscriber exception isolation | `tests/unit/test_event_bus.py` | Register callback that raises; publish; assert other callbacks still called |
+| EventBus unsubscribe removes callback | `tests/unit/test_event_bus.py` | Unsubscribe; publish; assert callback not called |
+| HealthMonitor detects crash | `tests/unit/test_health_monitor.py` | Mock `get_container_status` to return `"exited"`, `exit_code=137`; assert DB updated to `error`, event published |
+| HealthMonitor detects tunnel failure | `tests/unit/test_health_monitor.py` | Mock tunnel HEAD to 502; assert status → `unhealthy`, `health_checks` row inserted |
+| HealthMonitor skip on no change | `tests/unit/test_health_monitor.py` | Two identical polls; assert only one `health_checks` row |
+| HealthMonitor Docker exception resilience | `tests/unit/test_health_monitor.py` | Mock `get_container_status` to raise; assert no exception propagates, loop continues |
+
+**Fixtures needed:**
+- `event_bus`: fresh `InstanceEventBus()` instance (reset singleton state).
+- `health_monitor`: `HealthMonitor(event_bus)` with mocked `POLL_INTERVAL_SECONDS = 0.1`.
+- `db_session`: async SQLAlchemy session with rollback after each test.
+
+### 8.2 Integration Tests
+
+| Test | File | What |
+|------|------|------|
+| SSE endpoint requires auth | `tests/integration/test_sse_endpoint.py` | `GET /events/stream` without cookie → `401` |
+| SSE endpoint streams events | `tests/integration/test_sse_endpoint.py` | Authenticated client connects; backend publishes event; client receives SSE line within 1s |
+| SSE endpoint enforces connection limit | `tests/integration/test_sse_endpoint.py` | Open 6 connections; 6th returns `429` |
+| SSE disconnect unsubscribes | `tests/integration/test_sse_endpoint.py` | Connect; close client; publish event; assert no error, subscriber count = 0 |
+| Lifecycle hook publishes on start | `tests/integration/test_lifecycle_hooks.py` | Call start endpoint; assert `instance_events` row exists and event bus receives `instance.started` |
+
+### 8.3 E2E Tests
+
+| Test | File | What |
+|------|------|------|
+| Start container → toast appears | `tests/e2e/container_monitoring.spec.ts` (or Playwright) | Click Start; assert "Container starting..." toast; wait for probe; assert "Container running" toast |
+| Container crash → error toast | `tests/e2e/container_monitoring.spec.ts` | Start container; kill container externally; assert error toast within 5s |
+| Real-time badge update | `tests/e2e/container_monitoring.spec.ts` | Start container; badge green; kill container; badge turns red without refresh |
+
+### 8.4 Frontend Unit Tests
+
+| Test | File | What |
+|------|------|------|
+| useEvents reconnect backoff | `apps/web/src/hooks/use-events.test.ts` | Simulate `EventSource` error; assert reconnect delay doubles up to cap |
+| Toast deduplication | `apps/web/src/components/toast-rules.test.ts` | Two identical events within 1s; assert only one toast shown |
+| Event-to-toast mapping | `apps/web/src/components/toast-rules.test.ts` | Map each event type to correct toast type, message, duration |
+
+---
+
+## 9. Performance Considerations
+
+### 9.1 SSE Connection Pool
+
+- **Limit:** 5 concurrent SSE connections per user ID.
+- **Reasoning:** Prevents tab-spam from exhausting server memory. A typical user has 1–3 tabs open.
+- **Implementation:** In-memory `dict[uuid.UUID, int]` in `events.py`. In-memory is acceptable because single-process API is assumed.
+
+### 9.2 Health Monitor Batching
+
+- **Current approach:** `docker inspect` is called once per instance per poll cycle.
+- **Optimization (future):** Batch `docker ps --format json` to get all container statuses in a single CLI invocation, then match by `container_name`. **Not implemented in MVP** to keep changes minimal; document as follow-up.
+- **DB writes:** Only on state change. The monitor compares against `_last_known_state` in memory before touching the DB.
+
+### 9.3 Event Bus Memory Profile
+
+- **No event history:** The bus holds only subscriber callable references (lightweight).
+- **No queues:** SSE connections use per-connection `asyncio.Queue` capped at 100 items; if a client is slow, drop oldest events to prevent unbounded growth.
+
+```python
+queue: asyncio.Queue[InstanceEventPayload] = asyncio.Queue(maxsize=100)
+```
+
+### 9.4 Database Write Amplification
+
+- **Health checks:** Written only on state change, not every 15-second poll.
+- **Growth estimate:** 100 instances × 10 state changes/day × 365 days ≈ 365k rows/year. Acceptable for PostgreSQL.
+- **Retention (follow-up):** Add a scheduled cleanup job or pg_partman for `health_checks` older than 30 days.
+
+### 9.5 Frontend Polling Reduction
+
+- **Before:** Health poll every 30s per running instance = 2 req/min/instance.
+- **After:** One SSE connection per browser tab, zero polling for status. Fallback list refresh every 60s retained for resilience.
+- **Server load reduction:** For 50 running instances across all users, eliminates ~100 health-check HTTP requests per minute.
+
+### 9.6 JSON Logging Overhead
+
+- JSON formatter adds ~20% CPU overhead vs plain text for high-volume logs. Mitigate by:
+  - Keeping `uvicorn.access` at `WARNING`.
+  - Not logging every SSE ping.
+  - Using `orjson` for JSON serialization if available (fallback to stdlib `json`).
+
+---
+
+## 10. Rollout Plan
+
+| PR | Contents | Estimated Lines | Review Risk |
+|----|----------|-----------------|-------------|
+| **PR 1: Backend core** | DB migrations, models, `InstanceEventBus`, `HealthMonitor`, SSE endpoint, correlation ID middleware, JSON logging | ~1,000 | Medium |
+| **PR 2: Frontend** | `useEvents` hook, `ToastProvider`, `sonner` integration, badge real-time updates, remove 30s health polling | ~700 | Medium |
+| **PR 3: Integration + tests** | Lifecycle hook instrumentation in `tool_instances.py`, unit + integration tests, E2E tests | ~400 | Low |
+
+**Dependency order:** PR 1 → PR 2 → PR 3. PR 2 can be developed in parallel but must merge after PR 1.
+
+---
+
+## 11. Open Questions / Decisions
+
+| ID | Decision | Status |
+|----|----------|--------|
+| D1 | Use `sonner` for toasts (vs custom implementation) | **Decided:** `sonner` — reduces custom UI code by ~300 lines |
+| D2 | In-memory event bus (vs Redis/NATS) | **Decided:** In-memory — matches `TerminalManager` pattern; defer distributed bus |
+| D3 | SSE instead of WebSocket | **Decided:** SSE — one-way push, simpler auth, HTTP-compatible |
+| D4 | Batch `docker ps` for health monitor | **Deferred:** Keep per-instance `docker inspect` for MVP; document optimization |
+| D5 | `health_checks` retention policy | **Deferred:** 30-day retention to be added in follow-up |