# Container Monitoring & Notification System Specification

## Purpose

Provide real-time visibility into container lifecycle events, health state transitions, and failures through an in-memory event bus, a background health monitor, Server-Sent Events (SSE), and frontend toast notifications. Persist an append-only audit trail of lifecycle events and health state changes. Replace frontend polling with push-based updates and introduce structured JSON logging with correlation IDs.

> **Assumption:** This specification treats "Container Monitoring & Notification System" as a new cross-cutting domain. It introduces new tables, a new event bus, a new SSE endpoint, and new frontend components. Modifications to existing `tool_instances` lifecycle hooks and the frontend polling strategy are captured here as part of this feature domain.

---

## Non-Functional Requirements

| ID | Requirement |
|----|-------------|
| NFR-1 | **Performance:** The SSE endpoint MUST support at least 100 concurrent connections per API process without degrading event delivery latency below 1 second. |
| NFR-2 | **Latency:** Events MUST reach the frontend within 1 second of detection by the health monitor or a lifecycle hook. |
| NFR-3 | **Reliability:** The background health monitor MUST catch exceptions from Docker CLI commands, log the error, and continue the next polling cycle. It MUST NOT terminate the background task on transient errors. |
| NFR-4 | **Durability:** `instance_events` and `health_checks` rows MUST survive API restarts because they are stored in PostgreSQL. |

---

## Requirements

### Requirement: R1 — InstanceEventBus publishes typed lifecycle events

The system MUST provide an in-memory singleton event bus named `InstanceEventBus`.

- The bus MUST support publishing typed events to multiple subscribers.
- The bus MUST support subscribing and unsubscribing via callable callbacks.
- Events MUST be delivered to all subscribers in the same asyncio event loop iteration.
- If a subscriber raises an exception, the bus MUST catch it, log it, and continue delivering to remaining subscribers.

#### Event Payload Schema (JSON)

Every published event MUST conform to the following schema:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `event` | `string` | Yes | One of: `instance.created`, `instance.started`, `instance.stopped`, `instance.restarted`, `instance.deleted`, `instance.health_changed`, `instance.error` |
| `instance_id` | `string` (UUID) | Yes | The affected instance ID |
| `status` | `string` | No | Snapshot of the instance status at the time of the event |
| `message` | `string` | No | Human-readable description |
| `metadata` | `object` | No | Contextual data; see below |
| `timestamp` | `string` (ISO 8601) | Yes | Event timestamp in UTC |
| `correlation_id` | `string` (UUID) | Yes | Request correlation ID |

**`metadata` object fields:**

| Field | Type | Description |
|-------|------|-------------|
| `exit_code` | `integer` | Container exit code, if applicable |
| `tunnel_url` | `string` | Public tunnel URL at time of event |
| `probe_output` | `string` | Last probe stdout/stderr |
| `error_type` | `string` | One of: `"container"`, `"tunnel"`, `"probe"` |
| `previous_status` | `string` | Previous instance status on health changes |

#### Scenario: Event bus publish and subscribe

- **GIVEN** a subscriber callback is registered with `InstanceEventBus.subscribe(callback)`
- **WHEN** `InstanceEventBus.publish("instance.started", payload)` is called
- **THEN** the callback receives the payload within the same event loop iteration
- **AND** the payload contains `event: "instance.started"`, `instance_id`, `timestamp`, and `correlation_id`

#### Scenario: Subscriber exception isolation

- **GIVEN** two subscribers A and B are registered
- **WHEN** subscriber A raises an exception during event delivery
- **THEN** subscriber B still receives the event
- **AND** the exception from A is logged as an error with `correlation_id`

---

### Requirement: R2 — Background health monitor polls containers every 15 seconds

The system MUST run a background asyncio task that polls the health of all instances whose status is not `pending`, `stopped`, or `error`, every 15 seconds.

- For each candidate instance, the monitor MUST:
  1. Invoke `docker inspect` to read `State.Status`, `State.ExitCode`, and `State.Health.Status`.
  2. For web-enabled instances, perform an HTTP `GET` or `HEAD` to the `public_url` to determine tunnel health.
  3. Compare the result with the last known state stored in memory.
- On state change, the monitor MUST:
  1. Update `tool_instances.status` in the database.
  2. Insert a row into `health_checks`.
  3. Publish the appropriate event to `InstanceEventBus`.
- The monitor MUST NOT insert a `health_checks` row when the state has not changed.
- The monitor MUST catch all exceptions from Docker CLI or HTTP calls, log a structured error, and continue to the next instance.

#### Scenario: Monitor detects container crash

- **GIVEN** an instance with status `"running"`
- **WHEN** the monitor polls and `docker inspect` returns `State.Status = "exited"` and `State.ExitCode = 1`
- **THEN** `tool_instances.status` is updated to `"error"`
- **AND** a `health_checks` row is inserted with `container_status = "exited"` and `exit_code = 1`
- **AND** an `instance.error` event is published with `metadata.error_type = "container"`

#### Scenario: Monitor detects tunnel failure

- **GIVEN** an instance with status `"running"` and a previously healthy tunnel
- **WHEN** the monitor polls and the tunnel URL returns HTTP 502/503/504 or is unreachable
- **THEN** `tool_instances.status` is updated to `"unhealthy"`
- **AND** a `health_checks` row is inserted with `tunnel_healthy = false`
- **AND** an `instance.health_changed` event is published with `status = "unhealthy"` and `metadata.previous_status = "running"`

#### Scenario: Monitor detects recovery

- **GIVEN** an instance with status `"unhealthy"`
- **WHEN** the monitor polls and finds the container running and the tunnel returning HTTP 200
- **THEN** `tool_instances.status` is updated to `"running"`
- **AND** a `health_checks` row is inserted with `container_status = "running"` and `tunnel_healthy = true`
- **AND** an `instance.health_changed` event is published with `status = "running"` and `metadata.previous_status = "unhealthy"`

---

### Requirement: R3 — SSE endpoint streams events to authenticated clients

The system MUST expose `GET /events/stream`.

- The endpoint MUST require authentication using the same cookie/JWT session mechanism as the rest of the API.
- It MUST return `Content-Type: text/event-stream` with `Cache-Control: no-cache` and `Connection: keep-alive`.
- It MUST stream JSON event payloads formatted as SSE `data:` lines.
- On connection start, the server MUST subscribe to `InstanceEventBus`.
- On client disconnect, the server MUST unsubscribe and release resources.
- The endpoint MUST return `401 Unauthorized` if authentication is missing or invalid, and MUST NOT start a stream.

**SSE format per event:**

```
event: instance.started
data: {"event":"instance.started","instance_id":"...","status":"starting","message":"Container starting...","metadata":{},"timestamp":"2026-05-28T12:00:00Z","correlation_id":"..."}

```

#### Scenario: Authenticated client receives real-time events

- **GIVEN** an authenticated frontend session
- **WHEN** the client opens `GET /events/stream`
- **THEN** an SSE connection is established
- **AND** events published to `InstanceEventBus` are streamed within 1 second

#### Scenario: Unauthenticated client is rejected

- **GIVEN** a client with no valid session cookie or JWT
- **WHEN** the client opens `GET /events/stream`
- **THEN** the server responds with `401 Unauthorized`
- **AND** no SSE stream is started

#### Scenario: Client reconnects after network interruption

- **GIVEN** a connected SSE client that loses network connectivity
- **WHEN** the network recovers
- **THEN** the frontend reconnects with exponential backoff (1s, 2s, 4s, 8s, capped at 30s) with ±20% jitter
- **AND** a new SSE connection is established

---

### Requirement: R4 — Frontend displays toast notifications for errors

The system MUST display toast notifications in the frontend based on SSE events.

- **Error events** (`instance.error`) MUST display an error toast that persists until manually dismissed or for a minimum of 10 seconds.
- The error toast MUST show the event `message` and, if present, the `metadata.exit_code`.
- **Start events** (`instance.started`) SHOULD display an info toast with duration 3 seconds.
- **Running events** (`instance.health_changed` to `"running"`) SHOULD display a success toast with duration 3 seconds.
- **Unhealthy events** (`instance.health_changed` to `"unhealthy"`) SHOULD display a warning toast with duration 5 seconds.

#### Scenario: Build failure toast

- **GIVEN** the frontend is connected to the SSE stream
- **WHEN** an `instance.error` event is received with `metadata.exit_code = 137`
- **THEN** an error toast is displayed with the message and exit code `137`
- **AND** the toast remains visible for at least 10 seconds

#### Scenario: Successful start toast sequence

- **GIVEN** the frontend is connected to the SSE stream
- **WHEN** an `instance.started` event is received
- **THEN** an info toast "Container starting..." appears for 3 seconds
- **AND** when a subsequent `instance.health_changed` event with `status = "running"` is received
- **THEN** a success toast "Container running" appears for 3 seconds

---

### Requirement: R5 — Instance status badges update in real-time

The system MUST update instance status badges in the frontend within 1 second of receiving the corresponding SSE event.

- The frontend MUST stop polling for instance status every 30 seconds and instead rely on SSE events for status changes.
- The frontend MAY retain a lightweight fallback poll (e.g., every 60 seconds) for list refresh.
- Status badge colors MUST map to statuses as follows:
  - `running` → green
  - `starting`, `probing` → blue
  - `unhealthy` → yellow/amber
  - `error` → red
  - `stopped` → gray

#### Scenario: Badge updates on crash

- **GIVEN** an instance card showing a green `"running"` badge
- **WHEN** an `instance.error` event is received for that instance
- **THEN** the badge changes to red `"error"` without a page refresh
- **AND** the update occurs within 1 second

---

### Requirement: R6 — `instance_events` table records lifecycle transitions

The system MUST persist every lifecycle transition in an `instance_events` table.

**Table: `instance_events`**

| Column | Type | Constraints | Description |
|--------|------|-------------|-------------|
| `id` | `UUID` | PK | Unique event ID |
| `instance_id` | `UUID` | NOT NULL, FK → `tool_instances.id` ON DELETE CASCADE | Affected instance |
| `event_type` | `VARCHAR(50)` | NOT NULL | `created`, `started`, `stopped`, `restarted`, `deleted`, `health_changed`, `error` |
| `status` | `VARCHAR(50)` | | Instance status snapshot at time of event |
| `message` | `TEXT` | | Human-readable description |
| `created_by` | `UUID` | FK → `users.id` ON DELETE SET NULL | User who triggered the action (NULL for system events) |
| `metadata` | `JSONB` | DEFAULT `'{}'` | Contextual data (exit_code, tunnel_url, probe_output, etc.) |
| `created_at` | `TIMESTAMPTZ` | DEFAULT `now()` | Event timestamp |

**Indexes:**
- `idx_instance_events_instance_id` on (`instance_id`)
- `idx_instance_events_created_at` on (`created_at DESC`)
- `idx_instance_events_event_type` on (`event_type`)

#### Scenario: Start event recorded

- **GIVEN** an authenticated user starts an instance
- **WHEN** the start operation begins
- **THEN** an `instance_events` row is inserted with `event_type = "started"`, `status = "starting"`, and `created_by` set to the user's ID

#### Scenario: System error event recorded

- **GIVEN** the background monitor detects a container crash
- **WHEN** the state change is processed
- **THEN** an `instance_events` row is inserted with `event_type = "error"`, `status = "error"`, and `created_by = NULL`

---

### Requirement: R7 — `health_checks` table records state-change snapshots

The system MUST persist health state changes in a `health_checks` table.

**Table: `health_checks`**

| Column | Type | Constraints | Description |
|--------|------|-------------|-------------|
| `id` | `UUID` | PK | Unique check ID |
| `instance_id` | `UUID` | NOT NULL, FK → `tool_instances.id` ON DELETE CASCADE | Affected instance |
| `container_status` | `VARCHAR(50)` | | Docker container state (`running`, `exited`, `dead`, `not_found`) |
| `container_healthy` | `BOOLEAN` | | Result of Docker healthcheck, if configured |
| `tunnel_healthy` | `BOOLEAN` | | Result of HTTP probe to tunnel URL |
| `exit_code` | `INT` | | Container exit code, if applicable |
| `probe_status` | `VARCHAR(50)` | | `passed`, `failed`, `pending`, `not_configured` |
| `probe_output` | `TEXT` | | Last probe stdout/stderr |
| `checked_at` | `TIMESTAMPTZ` | DEFAULT `now()` | Timestamp of the check |

**Indexes:**
- `idx_health_checks_instance_id` on (`instance_id`)
- `idx_health_checks_checked_at` on (`checked_at DESC`)

#### Scenario: Health state change recorded

- **GIVEN** the monitor detects a transition from `"running"` to `"unhealthy"`
- **WHEN** the state change is processed
- **THEN** a `health_checks` row is inserted with `container_status`, `tunnel_healthy`, and `checked_at` set to the current timestamp
- **AND** no row is inserted on the next poll if the state remains `"unhealthy"`

---

### Requirement: R8 — Structured JSON logging with correlation IDs

The system MUST emit API logs in structured JSON format.

- Every log entry MUST include the fields: `timestamp`, `level`, `logger`, `message`, `correlation_id`.
- Log entries related to an instance MUST include `instance_id`.
- Log entries related to an event MUST include `event_type`.
- The system MUST generate a `correlation_id` for each incoming HTTP request and propagate it through the request lifecycle using an async context variable.
- The `correlation_id` MUST be included in all SSE event payloads published during that request.
- The `correlation_id` MUST be present in all logs emitted by the background health monitor for a given polling cycle (the monitor MAY generate a new `correlation_id` per cycle).

#### Scenario: Request logging with correlation ID

- **GIVEN** an incoming HTTP request with header `X-Request-ID: "abc-123"`
- **WHEN** the request triggers an instance start
- **THEN** all log entries for that request include `correlation_id: "abc-123"`
- **AND** the `instance.started` event published by that request includes `correlation_id: "abc-123"`

#### Scenario: Health monitor structured logging

- **GIVEN** the background health monitor is running
- **WHEN** a Docker CLI error occurs during a poll
- **THEN** the log entry is JSON formatted with `level: "ERROR"`, `instance_id`, `message`, and `correlation_id`

---

## API Contracts

### SSE Endpoint

```http
GET /events/stream
```

**Authentication:** Session cookie or JWT (same as existing API).

**Response Headers:**
- `Content-Type: text/event-stream`
- `Cache-Control: no-cache`
- `Connection: keep-alive`

**Success Response (200):** Stream of SSE events.

**Error Responses:**
- `401 Unauthorized` — Missing or invalid authentication.
- `429 Too Many Requests` — Client has exceeded the maximum of 5 concurrent SSE connections per user.

**Reconnection Strategy (Frontend):**
- Initial delay: 1 second.
- Multiplier: 2× per failed attempt.
- Maximum delay: 30 seconds.
- Jitter: ±20% randomization.

### Lifecycle Hook Event Mapping

| User Action / System Event | Published Event | Status | Metadata Notes |
|----------------------------|-----------------|--------|----------------|
| `POST /instances` (create) | `instance.created` | `"pending"` | — |
| `POST /instances/{id}/start` begins | `instance.started` | `"starting"` | — |
| Readiness probe passes | `instance.health_changed` | `"running"` | `previous_status: "starting"` |
| Container exits during start | `instance.error` | `"error"` | `error_type: "container"`, `exit_code` |
| `POST /instances/{id}/stop` | `instance.stopped` | `"stopped"` | — |
| `POST /instances/{id}/restart` | `instance.restarted` | `"starting"` | — |
| `DELETE /instances/{id}` | `instance.deleted` | `"deleted"` | — |
| Monitor detects crash | `instance.error` | `"error"` | `error_type: "container"`, `exit_code` |
| Monitor detects tunnel failure | `instance.health_changed` | `"unhealthy"` | `previous_status: "running"` |
| Monitor detects recovery | `instance.health_changed` | `"running"` | `previous_status: "unhealthy"` |

---

## Data Model

### New Tables

#### `instance_events`

```sql
CREATE TABLE instance_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    instance_id UUID NOT NULL REFERENCES tool_instances(id) ON DELETE CASCADE,
    event_type VARCHAR(50) NOT NULL,
    status VARCHAR(50),
    message TEXT,
    created_by UUID REFERENCES users(id) ON DELETE SET NULL,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_instance_events_instance_id ON instance_events(instance_id);
CREATE INDEX idx_instance_events_created_at ON instance_events(created_at DESC);
CREATE INDEX idx_instance_events_event_type ON instance_events(event_type);
```

#### `health_checks`

```sql
CREATE TABLE health_checks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    instance_id UUID NOT NULL REFERENCES tool_instances(id) ON DELETE CASCADE,
    container_status VARCHAR(50),
    container_healthy BOOLEAN,
    tunnel_healthy BOOLEAN,
    exit_code INT,
    probe_status VARCHAR(50),
    probe_output TEXT,
    checked_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_health_checks_instance_id ON health_checks(instance_id);
CREATE INDEX idx_health_checks_checked_at ON health_checks(checked_at DESC);
```

### Migration Strategy

- **Tool:** Alembic.
- **Revision:** Single revision creating both tables with indexes.
- **Data:** No backfill required; tables start empty.
- **Rollback:** Drop both tables and indexes.

---

## Behavior Specs

### Health Monitor State Machine

```
         +-----------+
         |  pending  |
         +-----+-----+
               | start()
               v
         +-----------+     probe fails / exit     +-------+
         | starting  +---------------------------> | error |
         +-----+-----+                             +---+---+
               | probe passes                           | restart()
               v                                        v
         +-----------+     crash / OOM          +-----------+
    +--->|  running  +----------------------->  |  error    |
    |    +-----+-----+                          +-----------+
    |          | tunnel/probe fail
    |          v
    |    +-----------+     recover              +-----------+
    +----+ unhealthy +----------------------->  |  running  |
         +-----+-----+                          +-----------+
               | stop
               v
         +-----------+
         |  stopped  |
         +-----------+
```

- Transitions are triggered by the background monitor or by user-initiated lifecycle actions.
- The monitor evaluates instances with status in `{starting, running, unhealthy}` every 15 seconds.
- Transitions to `error` or `stopped` from `running` or `unhealthy` are captured in `health_checks` and published to the event bus.

### Event Bus Publish/Subscribe Contract

- **Singleton:** `InstanceEventBus` is instantiated once per API process.
- **Subscribe:** `subscribe(callback: Callable[[dict], Awaitable[None] | None]) -> Callable[[], None]`
  - Returns an unsubscribe function.
- **Publish:** `publish(event_type: str, payload: dict) -> None`
  - Iterates over all subscribers.
  - If a callback is async, it is awaited; if sync, it is called directly.
  - Any exception is caught, logged with `correlation_id`, and delivery continues.
- **No persistence:** The bus does not queue events for offline subscribers.

### SSE Connection Lifecycle

1. **Connect:** Client sends `GET /events/stream` with valid auth.
2. **Validate:** Server verifies session; on failure returns `401`.
3. **Subscribe:** Server registers an `InstanceEventBus` subscriber callback.
4. **Stream:** Server yields SSE `data:` lines for each event received.
5. **Heartbeat:** Server sends an SSE comment (`:ping`) every 30 seconds to keep proxies alive.
6. **Disconnect:** Client closes connection; server catches `asyncio.CancelledError`, unsubscribes, and exits.
7. **Reconnect:** Client waits per backoff strategy and repeats step 1.

### Toast Display Rules

| SSE Event | Toast Type | Message | Duration |
|-----------|------------|---------|----------|
| `instance.started` | Info | `"Container starting..."` | 3s |
| `instance.health_changed` → `running` | Success | `"Container running"` | 3s |
| `instance.health_changed` → `unhealthy` | Warning | `"Container unhealthy"` | 5s |
| `instance.error` | Error | `message` + `exit_code` if present | 10s (or persistent) |

- The frontend MUST deduplicate toasts for the same `instance_id` and `event_type` received within 1 second.
- Only the `instance.error` toast MUST remain visible until manually dismissed; all others auto-dismiss after their duration.

---

## Scenarios (Acceptance Criteria)

### SC-1: User starts container → sees "starting..." toast → then "running" toast

- **GIVEN** the user clicks Start on an instance
- **WHEN** the start operation begins
- **THEN** an info toast `"Container starting..."` appears
- **AND** when the container passes the readiness probe
- **THEN** a success toast `"Container running"` appears

### SC-2: Container fails to build → sees error toast with exit code within 5 seconds

- **GIVEN** the user clicks Start on an instance
- **WHEN** the container exits during startup with `exit_code = 137`
- **THEN** an error toast appears with the message and exit code `137`
- **AND** the toast appears within 5 seconds of the container exiting

### SC-3: Container crashes while running → sees error toast + status changes to "error"

- **GIVEN** an instance with status `"running"`
- **WHEN** the background monitor detects the container has exited with a non-zero code
- **THEN** an error toast is displayed
- **AND** the instance status badge updates to `"error"`

### SC-4: Tunnel dies → sees tunnel error toast

- **GIVEN** an instance with status `"running"` and a healthy tunnel
- **WHEN** the background monitor detects the tunnel URL returns HTTP 502/503/504 or is unreachable
- **THEN** a warning toast `"Tunnel error"` is displayed (or error toast if mapped to `instance.error`)
- **AND** the instance status badge updates to `"unhealthy"`

### SC-5: Multiple instances running → each shows independent status updates

- **GIVEN** two instances with status `"running"`
- **WHEN** the first instance crashes and the second remains healthy
- **THEN** the first instance's status badge updates to `"error"`
- **AND** the second instance's status badge remains `"running"`
- **AND** only the first instance shows an error toast

### SC-6: Page reload → SSE reconnects, receives current state

- **GIVEN** the frontend is connected to SSE and an instance is running
- **WHEN** the user reloads the page
- **THEN** the frontend reconnects to `GET /events/stream`
- **AND** the SSE connection is established within 2 seconds
- **AND** subsequent state changes are received as events

---

## Error Handling

### Docker CLI failure during health check

- The monitor MUST catch `subprocess.CalledProcessError`, `TimeoutExpired`, and any other exception from the Docker CLI wrapper.
- It MUST log a structured JSON error with `instance_id`, `correlation_id`, and the exception details.
- It MUST skip the instance for the current cycle and retry on the next 15-second poll.
- It MUST NOT update `tool_instances.status` or publish an event for that instance during the failed cycle.

### SSE client disconnect

- The server MUST detect disconnect via `asyncio.CancelledError` or `Starlette` request disconnect signals.
- It MUST unsubscribe from `InstanceEventBus` and release the generator.
- It MUST NOT log an error for normal client disconnects.

### Auth failure on SSE

- If authentication is missing or invalid, the server MUST return `401 Unauthorized` before starting the `StreamingResponse`.
- It MUST NOT create an `InstanceEventBus` subscription.

### Event bus subscriber exception

- If a subscriber callback raises an exception, `InstanceEventBus` MUST catch it.
- It MUST log the exception with the event payload and `correlation_id`.
- It MUST continue calling the remaining subscribers.
- The publisher MUST NOT be blocked by a failing subscriber.

---

## Risks

1. **Legacy spec path:** This change uses the flat `openspec/changes/{change}/spec.md` path. Future archive steps should migrate to the nested `openspec/changes/{change}/specs/{domain}/spec.md` convention.
2. **Domain assumption:** The proposal did not contain an explicit "Capabilities" section. Domains were inferred from the proposed components (event bus, monitor, SSE, toasts, logging). If the parent orchestrator expects separate delta specs for `tool-instances`, `instance-runtime-health`, or `sessions-hub`, those should be extracted before the design phase.
3. **No canonical spec exists** for a "container-monitoring" or "notifications" domain, so this spec is written as a full new domain spec. Archive will need to create `openspec/specs/container-monitoring-notifications/spec.md` or similar.
