headquarter/openspec/changes/container-monitoring-notifications/spec.md

Container Monitoring & Notification System Specification

Purpose

Provide real-time visibility into container lifecycle events, health state transitions, and failures through an in-memory event bus, a background health monitor, Server-Sent Events (SSE), and frontend toast notifications. Persist an append-only audit trail of lifecycle events and health state changes. Replace frontend polling with push-based updates and introduce structured JSON logging with correlation IDs.

Assumption: This specification treats "Container Monitoring & Notification System" as a new cross-cutting domain. It introduces new tables, a new event bus, a new SSE endpoint, and new frontend components. Modifications to existing tool_instances lifecycle hooks and the frontend polling strategy are captured here as part of this feature domain.

---

Non-Functional Requirements

ID	Requirement
NFR-1	Performance: The SSE endpoint MUST support at least 100 concurrent connections per API process without degrading event delivery latency below 1 second.
NFR-2	Latency: Events MUST reach the frontend within 1 second of detection by the health monitor or a lifecycle hook.
NFR-3	Reliability: The background health monitor MUST catch exceptions from Docker CLI commands, log the error, and continue the next polling cycle. It MUST NOT terminate the background task on transient errors.
NFR-4	Durability: instance_events and health_checks rows MUST survive API restarts because they are stored in PostgreSQL.

---

Requirements

Requirement: R1 — InstanceEventBus publishes typed lifecycle events

The system MUST provide an in-memory singleton event bus named InstanceEventBus.

• The bus MUST support publishing typed events to multiple subscribers.
• The bus MUST support subscribing and unsubscribing via callable callbacks.
• Events MUST be delivered to all subscribers in the same asyncio event loop iteration.
• If a subscriber raises an exception, the bus MUST catch it, log it, and continue delivering to remaining subscribers.

Event Payload Schema (JSON)

Every published event MUST conform to the following schema:

Field	Type	Required	Description
event	string	Yes	One of: instance.created, instance.started, instance.stopped, instance.restarted, instance.deleted, instance.health_changed, instance.error
instance_id	string (UUID)	Yes	The affected instance ID
status	string	No	Snapshot of the instance status at the time of the event
message	string	No	Human-readable description
metadata	object	No	Contextual data; see below
timestamp	string (ISO 8601)	Yes	Event timestamp in UTC
correlation_id	string (UUID)	Yes	Request correlation ID

metadata object fields:

Field	Type	Description
exit_code	integer	Container exit code, if applicable
tunnel_url	string	Public tunnel URL at time of event
probe_output	string	Last probe stdout/stderr
error_type	string	One of: "container", "tunnel", "probe"
previous_status	string	Previous instance status on health changes

Scenario: Event bus publish and subscribe

• GIVEN a subscriber callback is registered with InstanceEventBus.subscribe(callback)
• WHEN InstanceEventBus.publish("instance.started", payload) is called
• THEN the callback receives the payload within the same event loop iteration
• AND the payload contains event: "instance.started", instance_id, timestamp, and correlation_id

Scenario: Subscriber exception isolation

• GIVEN two subscribers A and B are registered
• WHEN subscriber A raises an exception during event delivery
• THEN subscriber B still receives the event
• AND the exception from A is logged as an error with correlation_id

---

Requirement: R2 — Background health monitor polls containers every 15 seconds

The system MUST run a background asyncio task that polls the health of all instances whose status is not pending, stopped, or error, every 15 seconds.

• For each candidate instance, the monitor MUST:
  1. Invoke docker inspect to read State.Status, State.ExitCode, and State.Health.Status.
  2. For web-enabled instances, perform an HTTP GET or HEAD to the public_url to determine tunnel health.
  3. Compare the result with the last known state stored in memory.
• On state change, the monitor MUST:
  1. Update tool_instances.status in the database.
  2. Insert a row into health_checks.
  3. Publish the appropriate event to InstanceEventBus.
• The monitor MUST NOT insert a health_checks row when the state has not changed.
• The monitor MUST catch all exceptions from Docker CLI or HTTP calls, log a structured error, and continue to the next instance.

Scenario: Monitor detects container crash

• GIVEN an instance with status "running"
• WHEN the monitor polls and docker inspect returns State.Status = "exited" and State.ExitCode = 1
• THEN tool_instances.status is updated to "error"
• AND a health_checks row is inserted with container_status = "exited" and exit_code = 1
• AND an instance.error event is published with metadata.error_type = "container"

Scenario: Monitor detects tunnel failure

• GIVEN an instance with status "running" and a previously healthy tunnel
• WHEN the monitor polls and the tunnel URL returns HTTP 502/503/504 or is unreachable
• THEN tool_instances.status is updated to "unhealthy"
• AND a health_checks row is inserted with tunnel_healthy = false
• AND an instance.health_changed event is published with status = "unhealthy" and metadata.previous_status = "running"

Scenario: Monitor detects recovery

• GIVEN an instance with status "unhealthy"
• WHEN the monitor polls and finds the container running and the tunnel returning HTTP 200
• THEN tool_instances.status is updated to "running"
• AND a health_checks row is inserted with container_status = "running" and tunnel_healthy = true
• AND an instance.health_changed event is published with status = "running" and metadata.previous_status = "unhealthy"

---

Requirement: R3 — SSE endpoint streams events to authenticated clients

The system MUST expose GET /events/stream.

• The endpoint MUST require authentication using the same cookie/JWT session mechanism as the rest of the API.
• It MUST return Content-Type: text/event-stream with Cache-Control: no-cache and Connection: keep-alive.
• It MUST stream JSON event payloads formatted as SSE data: lines.
• On connection start, the server MUST subscribe to InstanceEventBus.
• On client disconnect, the server MUST unsubscribe and release resources.
• The endpoint MUST return 401 Unauthorized if authentication is missing or invalid, and MUST NOT start a stream.

SSE format per event:

event: instance.started
data: {"event":"instance.started","instance_id":"...","status":"starting","message":"Container starting...","metadata":{},"timestamp":"2026-05-28T12:00:00Z","correlation_id":"..."}

Scenario: Authenticated client receives real-time events

• GIVEN an authenticated frontend session
• WHEN the client opens GET /events/stream
• THEN an SSE connection is established
• AND events published to InstanceEventBus are streamed within 1 second

Scenario: Unauthenticated client is rejected

• GIVEN a client with no valid session cookie or JWT
• WHEN the client opens GET /events/stream
• THEN the server responds with 401 Unauthorized
• AND no SSE stream is started

Scenario: Client reconnects after network interruption

• GIVEN a connected SSE client that loses network connectivity
• WHEN the network recovers
• THEN the frontend reconnects with exponential backoff (1s, 2s, 4s, 8s, capped at 30s) with ±20% jitter
• AND a new SSE connection is established

---

Requirement: R4 — Frontend displays toast notifications for errors

The system MUST display toast notifications in the frontend based on SSE events.

• Error events (instance.error) MUST display an error toast that persists until manually dismissed or for a minimum of 10 seconds.
• The error toast MUST show the event message and, if present, the metadata.exit_code.
• Start events (instance.started) SHOULD display an info toast with duration 3 seconds.
• Running events (instance.health_changed to "running") SHOULD display a success toast with duration 3 seconds.
• Unhealthy events (instance.health_changed to "unhealthy") SHOULD display a warning toast with duration 5 seconds.

Scenario: Build failure toast

• GIVEN the frontend is connected to the SSE stream
• WHEN an instance.error event is received with metadata.exit_code = 137
• THEN an error toast is displayed with the message and exit code 137
• AND the toast remains visible for at least 10 seconds

Scenario: Successful start toast sequence

• GIVEN the frontend is connected to the SSE stream
• WHEN an instance.started event is received
• THEN an info toast "Container starting..." appears for 3 seconds
• AND when a subsequent instance.health_changed event with status = "running" is received
• THEN a success toast "Container running" appears for 3 seconds

---

Requirement: R5 — Instance status badges update in real-time

The system MUST update instance status badges in the frontend within 1 second of receiving the corresponding SSE event.

• The frontend MUST stop polling for instance status every 30 seconds and instead rely on SSE events for status changes.
• The frontend MAY retain a lightweight fallback poll (e.g., every 60 seconds) for list refresh.
• Status badge colors MUST map to statuses as follows:
  • running → green
  • starting, probing → blue
  • unhealthy → yellow/amber
  • error → red
  • stopped → gray

Scenario: Badge updates on crash

• GIVEN an instance card showing a green "running" badge
• WHEN an instance.error event is received for that instance
• THEN the badge changes to red "error" without a page refresh
• AND the update occurs within 1 second

---

Requirement: R6 — instance_events table records lifecycle transitions

The system MUST persist every lifecycle transition in an instance_events table.

Table: instance_events

Column	Type	Constraints	Description
id	UUID	PK	Unique event ID
instance_id	UUID	NOT NULL, FK → tool_instances.id ON DELETE CASCADE	Affected instance
event_type	VARCHAR(50)	NOT NULL	created, started, stopped, restarted, deleted, health_changed, error
status	VARCHAR(50)		Instance status snapshot at time of event
message	TEXT		Human-readable description
created_by	UUID	FK → users.id ON DELETE SET NULL	User who triggered the action (NULL for system events)
metadata	JSONB	DEFAULT '{}'	Contextual data (exit_code, tunnel_url, probe_output, etc.)
created_at	TIMESTAMPTZ	DEFAULT now()	Event timestamp

Indexes:

• idx_instance_events_instance_id on (instance_id)
• idx_instance_events_created_at on (created_at DESC)
• idx_instance_events_event_type on (event_type)

Scenario: Start event recorded

• GIVEN an authenticated user starts an instance
• WHEN the start operation begins
• THEN an instance_events row is inserted with event_type = "started", status = "starting", and created_by set to the user's ID

Scenario: System error event recorded

• GIVEN the background monitor detects a container crash
• WHEN the state change is processed
• THEN an instance_events row is inserted with event_type = "error", status = "error", and created_by = NULL

---

Requirement: R7 — health_checks table records state-change snapshots

The system MUST persist health state changes in a health_checks table.

Table: health_checks

Column	Type	Constraints	Description
id	UUID	PK	Unique check ID
instance_id	UUID	NOT NULL, FK → tool_instances.id ON DELETE CASCADE	Affected instance
container_status	VARCHAR(50)		Docker container state (running, exited, dead, not_found)
container_healthy	BOOLEAN		Result of Docker healthcheck, if configured
tunnel_healthy	BOOLEAN		Result of HTTP probe to tunnel URL
exit_code	INT		Container exit code, if applicable
probe_status	VARCHAR(50)		passed, failed, pending, not_configured
probe_output	TEXT		Last probe stdout/stderr
checked_at	TIMESTAMPTZ	DEFAULT now()	Timestamp of the check

Indexes:

• idx_health_checks_instance_id on (instance_id)
• idx_health_checks_checked_at on (checked_at DESC)

Scenario: Health state change recorded

• GIVEN the monitor detects a transition from "running" to "unhealthy"
• WHEN the state change is processed
• THEN a health_checks row is inserted with container_status, tunnel_healthy, and checked_at set to the current timestamp
• AND no row is inserted on the next poll if the state remains "unhealthy"

---

Requirement: R8 — Structured JSON logging with correlation IDs

The system MUST emit API logs in structured JSON format.

• Every log entry MUST include the fields: timestamp, level, logger, message, correlation_id.
• Log entries related to an instance MUST include instance_id.
• Log entries related to an event MUST include event_type.
• The system MUST generate a correlation_id for each incoming HTTP request and propagate it through the request lifecycle using an async context variable.
• The correlation_id MUST be included in all SSE event payloads published during that request.
• The correlation_id MUST be present in all logs emitted by the background health monitor for a given polling cycle (the monitor MAY generate a new correlation_id per cycle).

Scenario: Request logging with correlation ID

• GIVEN an incoming HTTP request with header X-Request-ID: "abc-123"
• WHEN the request triggers an instance start
• THEN all log entries for that request include correlation_id: "abc-123"
• AND the instance.started event published by that request includes correlation_id: "abc-123"

Scenario: Health monitor structured logging

• GIVEN the background health monitor is running
• WHEN a Docker CLI error occurs during a poll
• THEN the log entry is JSON formatted with level: "ERROR", instance_id, message, and correlation_id

---

API Contracts

SSE Endpoint

GET /events/stream

Authentication: Session cookie or JWT (same as existing API).

Response Headers:

• Content-Type: text/event-stream
• Cache-Control: no-cache
• Connection: keep-alive

Success Response (200): Stream of SSE events.

Error Responses:

• 401 Unauthorized — Missing or invalid authentication.
• 429 Too Many Requests — Client has exceeded the maximum of 5 concurrent SSE connections per user.

Reconnection Strategy (Frontend):

• Initial delay: 1 second.
• Multiplier: 2× per failed attempt.
• Maximum delay: 30 seconds.
• Jitter: ±20% randomization.

Lifecycle Hook Event Mapping

User Action / System Event	Published Event	Status	Metadata Notes
POST /instances (create)	instance.created	"pending"	—
POST /instances/{id}/start begins	instance.started	"starting"	—
Readiness probe passes	instance.health_changed	"running"	previous_status: "starting"
Container exits during start	instance.error	"error"	error_type: "container", exit_code
POST /instances/{id}/stop	instance.stopped	"stopped"	—
POST /instances/{id}/restart	instance.restarted	"starting"	—
DELETE /instances/{id}	instance.deleted	"deleted"	—
Monitor detects crash	instance.error	"error"	error_type: "container", exit_code
Monitor detects tunnel failure	instance.health_changed	"unhealthy"	previous_status: "running"
Monitor detects recovery	instance.health_changed	"running"	previous_status: "unhealthy"

---

Data Model

New Tables

instance_events

CREATE TABLE instance_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    instance_id UUID NOT NULL REFERENCES tool_instances(id) ON DELETE CASCADE,
    event_type VARCHAR(50) NOT NULL,
    status VARCHAR(50),
    message TEXT,
    created_by UUID REFERENCES users(id) ON DELETE SET NULL,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_instance_events_instance_id ON instance_events(instance_id);
CREATE INDEX idx_instance_events_created_at ON instance_events(created_at DESC);
CREATE INDEX idx_instance_events_event_type ON instance_events(event_type);

health_checks

CREATE TABLE health_checks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    instance_id UUID NOT NULL REFERENCES tool_instances(id) ON DELETE CASCADE,
    container_status VARCHAR(50),
    container_healthy BOOLEAN,
    tunnel_healthy BOOLEAN,
    exit_code INT,
    probe_status VARCHAR(50),
    probe_output TEXT,
    checked_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_health_checks_instance_id ON health_checks(instance_id);
CREATE INDEX idx_health_checks_checked_at ON health_checks(checked_at DESC);

Migration Strategy

• Tool: Alembic.
• Revision: Single revision creating both tables with indexes.
• Data: No backfill required; tables start empty.
• Rollback: Drop both tables and indexes.

---

Behavior Specs

Health Monitor State Machine

         +-----------+
         |  pending  |
         +-----+-----+
               | start()
               v
         +-----------+     probe fails / exit     +-------+
         | starting  +---------------------------> | error |
         +-----+-----+                             +---+---+
               | probe passes                           | restart()
               v                                        v
         +-----------+     crash / OOM          +-----------+
    +--->|  running  +----------------------->  |  error    |
    |    +-----+-----+                          +-----------+
    |          | tunnel/probe fail
    |          v
    |    +-----------+     recover              +-----------+
    +----+ unhealthy +----------------------->  |  running  |
         +-----+-----+                          +-----------+
               | stop
               v
         +-----------+
         |  stopped  |
         +-----------+

• Transitions are triggered by the background monitor or by user-initiated lifecycle actions.
• The monitor evaluates instances with status in {starting, running, unhealthy} every 15 seconds.
• Transitions to error or stopped from running or unhealthy are captured in health_checks and published to the event bus.

Event Bus Publish/Subscribe Contract

• Singleton: InstanceEventBus is instantiated once per API process.
• Subscribe: subscribe(callback: Callable[[dict], Awaitable[None] | None]) -> Callable[[], None]
  • Returns an unsubscribe function.
• Publish: publish(event_type: str, payload: dict) -> None
  • Iterates over all subscribers.
  • If a callback is async, it is awaited; if sync, it is called directly.
  • Any exception is caught, logged with correlation_id, and delivery continues.
• No persistence: The bus does not queue events for offline subscribers.

SSE Connection Lifecycle

1. Connect: Client sends GET /events/stream with valid auth.
2. Validate: Server verifies session; on failure returns 401.
3. Subscribe: Server registers an InstanceEventBus subscriber callback.
4. Stream: Server yields SSE data: lines for each event received.
5. Heartbeat: Server sends an SSE comment (:ping) every 30 seconds to keep proxies alive.
6. Disconnect: Client closes connection; server catches asyncio.CancelledError, unsubscribes, and exits.
7. Reconnect: Client waits per backoff strategy and repeats step 1.

Toast Display Rules

SSE Event	Toast Type	Message	Duration
instance.started	Info	"Container starting..."	3s
instance.health_changed → running	Success	"Container running"	3s
instance.health_changed → unhealthy	Warning	"Container unhealthy"	5s
instance.error	Error	message + exit_code if present	10s (or persistent)

• The frontend MUST deduplicate toasts for the same instance_id and event_type received within 1 second.
• Only the instance.error toast MUST remain visible until manually dismissed; all others auto-dismiss after their duration.

---

Scenarios (Acceptance Criteria)

SC-1: User starts container → sees "starting..." toast → then "running" toast

• GIVEN the user clicks Start on an instance
• WHEN the start operation begins
• THEN an info toast "Container starting..." appears
• AND when the container passes the readiness probe
• THEN a success toast "Container running" appears

SC-2: Container fails to build → sees error toast with exit code within 5 seconds

• GIVEN the user clicks Start on an instance
• WHEN the container exits during startup with exit_code = 137
• THEN an error toast appears with the message and exit code 137
• AND the toast appears within 5 seconds of the container exiting

SC-3: Container crashes while running → sees error toast + status changes to "error"

• GIVEN an instance with status "running"
• WHEN the background monitor detects the container has exited with a non-zero code
• THEN an error toast is displayed
• AND the instance status badge updates to "error"

SC-4: Tunnel dies → sees tunnel error toast

• GIVEN an instance with status "running" and a healthy tunnel
• WHEN the background monitor detects the tunnel URL returns HTTP 502/503/504 or is unreachable
• THEN a warning toast "Tunnel error" is displayed (or error toast if mapped to instance.error)
• AND the instance status badge updates to "unhealthy"

SC-5: Multiple instances running → each shows independent status updates

• GIVEN two instances with status "running"
• WHEN the first instance crashes and the second remains healthy
• THEN the first instance's status badge updates to "error"
• AND the second instance's status badge remains "running"
• AND only the first instance shows an error toast

SC-6: Page reload → SSE reconnects, receives current state

• GIVEN the frontend is connected to SSE and an instance is running
• WHEN the user reloads the page
• THEN the frontend reconnects to GET /events/stream
• AND the SSE connection is established within 2 seconds
• AND subsequent state changes are received as events

---

Error Handling

Docker CLI failure during health check

• The monitor MUST catch subprocess.CalledProcessError, TimeoutExpired, and any other exception from the Docker CLI wrapper.
• It MUST log a structured JSON error with instance_id, correlation_id, and the exception details.
• It MUST skip the instance for the current cycle and retry on the next 15-second poll.
• It MUST NOT update tool_instances.status or publish an event for that instance during the failed cycle.

SSE client disconnect

• The server MUST detect disconnect via asyncio.CancelledError or Starlette request disconnect signals.
• It MUST unsubscribe from InstanceEventBus and release the generator.
• It MUST NOT log an error for normal client disconnects.

Auth failure on SSE

• If authentication is missing or invalid, the server MUST return 401 Unauthorized before starting the StreamingResponse.
• It MUST NOT create an InstanceEventBus subscription.

Event bus subscriber exception

• If a subscriber callback raises an exception, InstanceEventBus MUST catch it.
• It MUST log the exception with the event payload and correlation_id.
• It MUST continue calling the remaining subscribers.
• The publisher MUST NOT be blocked by a failing subscriber.

---

Risks

1. Legacy spec path: This change uses the flat openspec/changes/{change}/spec.md path. Future archive steps should migrate to the nested openspec/changes/{change}/specs/{domain}/spec.md convention.
2. Domain assumption: The proposal did not contain an explicit "Capabilities" section. Domains were inferred from the proposed components (event bus, monitor, SSE, toasts, logging). If the parent orchestrator expects separate delta specs for tool-instances, instance-runtime-health, or sessions-hub, those should be extracted before the design phase.
3. No canonical spec exists for a "container-monitoring" or "notifications" domain, so this spec is written as a full new domain spec. Archive will need to create openspec/specs/container-monitoring-notifications/spec.md or similar.