headquarter/openspec/changes/container-monitoring-notifications/tasks.md

SDD Tasks: Container Monitoring & Notification System

Review Workload Forecast

Field	Value
Estimated changed lines	~2,100 total (PR-1 ~1,000; PR-2 ~700; PR-3 ~400)
400-line budget risk	High
Chained PRs recommended	Yes
Suggested split	PR 1 (Backend Core) → PR 2 (Frontend UI) → PR 3 (Integration + Polish)
Delivery strategy	auto-chain
Chain strategy	stacked-to-main

Decision needed before apply: No
Chained PRs recommended: Yes
Chain strategy: stacked-to-main
400-line budget risk: High

Note: PR-1 and PR-2 exceed the 400-line review budget. Within each PR, tasks are grouped into autonomous work units that can be reviewed independently. If review fanout is available, consider splitting PR-1 into (a) DB + EventBus + SSE and (b) HealthMonitor + Lifecycle Hooks + Logging. PR-2 can be split into (a) useEvents + ToastProvider and (b) Badge updates + Polling removal.

---

PR-1: Backend Core

Goal: Establish the backend infrastructure for real-time container monitoring: database schema, in-memory event bus, background health monitor, SSE endpoint, structured logging, and lifecycle instrumentation.

Estimated Lines: ~1,000 Review Risk: High

---

MON-PR1-001: Create Alembic migration for monitoring tables

Description: Write a single Alembic revision that creates instance_events and health_checks with all columns, constraints, and indexes defined in the spec.

Files to modify:

• apps/api/alembic/versions/2026_05_28_add_monitoring_tables.py (new)

Acceptance criteria:

• Migration creates instance_events table with columns: id, instance_id, event_type, status, message, created_by, metadata, created_at.
• Migration creates health_checks table with columns: id, instance_id, container_status, container_healthy, tunnel_healthy, exit_code, probe_status, probe_output, checked_at.
• All 5 indexes from the spec are created.
• upgrade() and downgrade() are both implemented and pass alembic upgrade head / alembic downgrade -1.
• Migration depends on current head revision.

Estimated effort: Small (2–3 hours) Dependencies: None

---

MON-PR1-002: Create SQLAlchemy models for InstanceEvent and HealthCheck

Description: Add SQLAlchemy models matching the migration schema, following the existing UUIDPrimaryKeyMixin + Base pattern (no TimestampMixin on InstanceEvent; created_at uses server_default).

Files to modify:

• apps/api/src/models/instance_event.py (new)
• apps/api/src/models/health_check.py (new)
• apps/api/src/models/__init__.py

Acceptance criteria:

• InstanceEvent model matches spec schema with correct FKs (ON DELETE CASCADE / SET NULL).
• HealthCheck model matches spec schema with correct FK (ON DELETE CASCADE).
• Both models exported in models/__init__.py.
• alembic revision --autogenerate produces no drift against the hand-written migration.

Estimated effort: Small (2–3 hours) Dependencies: MON-PR1-001

---

MON-PR1-003: Add correlation ID context variable and middleware

Description: Implement an async context variable CORRELATION_ID and a FastAPI middleware that reads X-Request-ID or generates a new UUID on every request. This must be available before structured logging and event publishing.

Files to modify:

• apps/api/src/services/correlation.py (new)
• apps/api/src/main.py

Acceptance criteria:

• CORRELATION_ID: contextvars.ContextVar[str] exists with get_correlation_id() helper.
• CorrelationIdMiddleware sets the context var from X-Request-ID header or uuid.uuid4().
• Middleware is registered in main.py before all routes.
• Calling get_correlation_id() inside a request handler returns the same ID for the full request lifecycle.

Estimated effort: Small (2–3 hours) Dependencies: None

---

MON-PR1-004: Refactor API logging to structured JSON format

Description: Replace the plain-text formatter in logging_config.py with a JSON formatter that includes timestamp, level, logger, message, correlation_id, instance_id, and event_type. Add a logging.Filter that reads from CORRELATION_ID.

Files to modify:

• apps/api/src/logging_config.py

Acceptance criteria:

• Log output is valid JSON lines with required fields.
• correlation_id is populated automatically from the context var.
• instance_id and event_type are included when passed as extra= to the logger.
• Request/response middleware logs remain functional but now emit JSON.
• Unhandled exception middleware logs tracebacks as JSON.
• uvicorn.access stays at WARNING to reduce noise.

Estimated effort: Small (2–3 hours) Dependencies: MON-PR1-003

---

MON-PR1-005: Implement InstanceEventBus singleton with typed pub/sub

Description: Create the in-memory event bus as a module-level singleton following the TerminalManager pattern. Support typed subscription, unsubscribe, and exception-isolated delivery.

Files to modify:

• apps/api/src/services/event_bus.py (new)

Acceptance criteria:

• InstanceEventBus is a singleton (__new__ + lock).
• subscribe(event_type, callback) returns an unsubscribe callable.
• publish(event_type, payload) delivers to all subscribers in the same event loop iteration.
• If a subscriber raises, the exception is logged with correlation_id and delivery continues to remaining subscribers.
• InstanceEventPayload TypedDict matches the spec schema exactly.
• unsubscribe_all(event_type) exists for test teardown.

Estimated effort: Small (3–4 hours) Dependencies: MON-PR1-003

---

MON-PR1-006: Implement HealthMonitor background polling task

Description: Build the background monitor that polls Docker + tunnel health every 15 seconds, compares against in-memory state, and only writes to DB / publishes events on actual state changes.

Files to modify:

• apps/api/src/services/health_monitor.py (new)
• apps/api/src/services/docker.py (read-only; reuse get_container_status)

Acceptance criteria:

• HealthMonitor accepts event_bus: InstanceEventBus and is a singleton-style service.
• start() is idempotent; creates an asyncio task for _poll_loop().
• stop() cancels the task and clears _last_known_state.
• Poll interval is 15.0 seconds (configurable for tests).
• Queries all instances where status NOT IN ("pending", "stopped", "error").
• Per instance: calls get_container_status(), then HTTP HEAD/GET to public_url if present.
• On state change: updates tool_instances.status, inserts health_checks row, publishes instance.health_changed or instance.error.
• On no change: skips all DB writes and event publication.
• Per-instance exceptions are caught, logged as structured JSON, and the loop continues.
• _last_known_state is a dict[UUID, HealthSnapshot] dataclass.

Estimated effort: Medium (5–7 hours) Dependencies: MON-PR1-002, MON-PR1-005

---

MON-PR1-007: Implement SSE streaming endpoint with auth and connection limits

Description: Create /events/stream using FastAPI StreamingResponse with text/event-stream. Enforce authentication and a max of 5 concurrent connections per user.

Files to modify:

• apps/api/src/api/events.py (new)
• apps/api/src/api/__init__.py

Acceptance criteria:

• GET /events/stream returns 401 before stream start if auth is missing/invalid.
• Returns 429 if user already has 5 open SSE connections.
• Sends SSE event: and data: lines formatted per spec.
• Sends :ping comment every 30 seconds.
• Per-connection asyncio.Queue(maxsize=100) drops oldest events if client is slow.
• On disconnect (asyncio.CancelledError or client close), unsubscribes from InstanceEventBus and releases the connection slot.
• Router is exported from api/__init__.py.

Estimated effort: Medium (4–6 hours) Dependencies: MON-PR1-005

---

MON-PR1-008: Instrument lifecycle hooks in tool_instances.py

Description: Add event publishing and audit-row writes at all lifecycle transition points in tool_instances.py. Create a thin lifecycle_hooks.py service to keep tool_instances.py readable.

Files to modify:

• apps/api/src/services/lifecycle_hooks.py (new)
• apps/api/src/api/tool_instances.py

Acceptance criteria:

• After DB commit on POST /instances → instance.created event + instance_events row.
• After DB commit on start begins → instance.started event + row.
• After probe success → instance.health_changed (running) event + row.
• After container exits during start → instance.error event + row.
• After DB commit on stop → instance.stopped event + row.
• After DB commit on restart → instance.restarted event + row.
• After DB commit on delete → instance.deleted event + row.
• created_by is set to current_user.id for user actions; NULL for system-detected transitions.
• correlation_id from the request context is propagated into the event payload.

Estimated effort: Medium (4–6 hours) Dependencies: MON-PR1-002, MON-PR1-005, MON-PR1-003

---

MON-PR1-009: Wire up HealthMonitor, EventBus, and events router in application startup

Description: Register the new events router and start/stop the HealthMonitor within FastAPI lifespan events.

Files to modify:

• apps/api/src/main.py

Acceptance criteria:

• events_router is included in the main FastAPI app with appropriate prefix.
• HealthMonitor is instantiated with the global InstanceEventBus and started during app startup.
• HealthMonitor.stop() is called during app shutdown.
• No import cycles introduced.
• App boots and passes a smoke test (GET /health still works).

Estimated effort: Small (1–2 hours) Dependencies: MON-PR1-006, MON-PR1-007

---

MON-PR1-010: Backend unit tests — EventBus

Description: Write pytest unit tests for InstanceEventBus covering pub/sub, exception isolation, and unsubscribe.

Files to modify:

• tests/unit/test_event_bus.py (new)

Acceptance criteria:

• test_publish_delivers_to_all_subscribers: 3 callbacks registered, all receive payload.
• test_subscriber_exception_isolation: callback A raises, B still receives event.
• test_unsubscribe_removes_callback: after unsubscribe, callback is not called.
• test_publish_to_empty_subscriber_list: no error raised.
• Tests use a fresh InstanceEventBus instance (reset singleton state in fixture).

Estimated effort: Small (2–3 hours) Dependencies: MON-PR1-005

---

MON-PR1-011: Backend unit tests — HealthMonitor

Description: Write pytest unit tests for HealthMonitor state-transition logic using mocked Docker and HTTP responses.

Files to modify:

• tests/unit/test_health_monitor.py (new)

Acceptance criteria:

• test_detects_container_crash: mock get_container_status → exited, exit_code=137; asserts DB status becomes error, event published, health_checks row inserted.
• test_detects_tunnel_failure: mock tunnel HEAD → 502; asserts status → unhealthy, tunnel_healthy=false in DB.
• test_detects_recovery: mock running + tunnel 200 after unhealthy; asserts status → running, health_checks row inserted.
• test_skips_writes_when_no_state_change: two identical polls; asserts only one health_checks row.
• test_docker_exception_resilience: mock raises CalledProcessError; asserts no exception propagates, loop continues.
• Uses db_session and event_bus fixtures; mocks poll interval to 0.1s.

Estimated effort: Medium (4–5 hours) Dependencies: MON-PR1-006, MON-PR1-010

---

MON-PR1-012: Backend integration tests — SSE endpoint

Description: Write integration tests for the SSE endpoint covering auth, streaming, connection limits, and disconnect cleanup.

Files to modify:

• tests/integration/test_sse_endpoint.py (new)

Acceptance criteria:

• test_sse_requires_auth: GET /events/stream without cookie → 401.
• test_sse_streams_event: authenticated client connects; backend publishes event; client receives valid SSE line within 1s.
• test_sse_enforces_connection_limit: open 6 connections; 6th returns 429.
• test_sse_disconnect_unsubscribes: connect, close client, publish event; assert subscriber count is 0 and no error logged.
• Uses authenticated_client fixture.

Estimated effort: Medium (4–5 hours) Dependencies: MON-PR1-007

---

PR-2: Frontend UI

Goal: Build the frontend event consumption layer: SSE client hook, toast notification system, and real-time status badge updates.

Estimated Lines: ~700 Review Risk: High

---

MON-PR2-001: Install sonner and create event TypeScript types

Description: Add sonner to the frontend dependencies and create the InstanceEventPayload TypeScript interface that mirrors the backend spec.

Files to modify:

• apps/web/package.json
• apps/web/src/types/events.ts (new)

Acceptance criteria:

• sonner is added to dependencies (not devDependencies).
• InstanceEventPayload interface includes all required fields: event, instance_id, status, message, metadata, timestamp, correlation_id.
• metadata sub-type includes optional fields: exit_code, tunnel_url, probe_output, error_type, previous_status.
• pnpm install (or equivalent) succeeds and lockfile updated.

Estimated effort: Small (1–2 hours) Dependencies: PR-1 merged (backend SSE endpoint must exist)

---

MON-PR2-002: Implement useEvents() SSE hook with reconnect backoff

Description: Create a React hook that opens an EventSource to /events/stream, handles reconnections with exponential backoff + jitter, and exposes parsed events.

Files to modify:

• apps/web/src/hooks/use-events.ts (new)

Acceptance criteria:

• Hook connects to ${API_BASE_URL}/events/stream with credentials included.
• Parsed events are returned in a reactive list/array.
• connected boolean reflects EventSource ready state.
• On error/disconnect: waits delay = min(30000, 1000 * 2^attempts) * (0.8 + Math.random() * 0.4) before reconnect.
• On 401 response: stops reconnecting and redirects to login.
• On 429 response: adds extra 5s penalty before next retry.
• Hook cleans up EventSource on unmount.
• reconnectCount is exposed for debugging.

Estimated effort: Medium (4–5 hours) Dependencies: MON-PR2-001

---

MON-PR2-003: Implement toast rules and deduplication logic

Description: Create a pure module that maps SSE event types to toast configurations and deduplicates rapid duplicate events.

Files to modify:

• apps/web/src/components/toast-rules.ts (new)

Acceptance criteria:

• instance.started → info toast, message "Container starting...", duration 3s.
• instance.health_changed to running → success toast, message "Container running", duration 3s.
• instance.health_changed to unhealthy → warning toast, message "Container unhealthy", duration 5s.
• instance.error → error toast, uses event message + metadata.exit_code if present, duration 10s (or persistent if sonner supports it).
• Deduplication: same (instance_id, event_type) within 1s produces only one toast.
• Function is pure and testable without React rendering.

Estimated effort: Small (2–3 hours) Dependencies: MON-PR2-001

---

MON-PR2-004: Implement ToastProvider component

Description: Build a global toast provider that wraps sonner's <Toaster />, consumes useEvents(), and renders toasts via the rules module.

Files to modify:

• apps/web/src/components/toast-provider.tsx (new)
• apps/web/src/components/app-shell.tsx

Acceptance criteria:

• <ToastProvider /> mounts <Toaster /> and calls useEvents().
• Incoming events are passed through toast-rules.ts mapping.
• Mounted inside AppShell so it is active on every authenticated page.
• Deduplication state is managed internally (e.g., Map<string, number> of last toast timestamp).
• Does not cause re-renders of the entire app on every SSE event (uses narrow subscription or memoization).

Estimated effort: Small (3–4 hours) Dependencies: MON-PR2-002, MON-PR2-003

---

MON-PR2-005: Replace health polling with real-time SSE updates in instance list

Description: Remove the 30-second health polling loop from instance-list.tsx and session-card.tsx. Consume useEvents() to update status badges in real time. Retain a 60-second lightweight list refresh.

Files to modify:

• apps/web/src/components/instance-list.tsx
• apps/web/src/components/session-card.tsx
• apps/web/src/api/sessions.ts

Acceptance criteria:

• setInterval health polling (every 30s) is removed from instance-list.tsx.
• session-card.tsx badge colors map to statuses: running → green, starting/probing → blue, unhealthy → amber, error → red, stopped → gray.
• Badge text and color update within 1s of receiving the matching SSE event.
• api/sessions.ts still exports checkInstanceHealth for on-demand use (do not delete the function).
• A 60s list refresh poll remains for resilience (full list re-fetch, not per-instance health).
• Multiple instances update independently (no global refresh on single-instance event).

Estimated effort: Medium (4–5 hours) Dependencies: MON-PR2-002

---

MON-PR2-006: Frontend unit tests — useEvents hook

Description: Write tests for the useEvents hook using mocked EventSource to verify reconnect logic and event parsing.

Files to modify:

• apps/web/src/hooks/use-events.test.ts (new)

Acceptance criteria:

• test_reconnects_with_backoff: simulate EventSource error; assert reconnect delay follows exponential pattern up to 30s cap.
• test_parses_sse_event: simulate incoming message event with JSON payload; assert hook state contains parsed event.
• test_stops_on_401: simulate 401; assert EventSource is closed and reconnect stops.
• test_cleans_up_on_unmount: unmount component; assert EventSource.close() called.

Estimated effort: Small (3–4 hours) Dependencies: MON-PR2-002

---

MON-PR2-007: Frontend unit tests — toast rules

Description: Write tests for toast-rules.ts covering mapping correctness and deduplication.

Files to modify:

• apps/web/src/components/toast-rules.test.ts (new)

Acceptance criteria:

• test_maps_error_event_to_error_toast: asserts type, message includes exit code, duration.
• test_maps_running_health_change_to_success_toast: asserts type, message, duration.
• test_deduplicates_within_one_second: two identical events at t=0 and t=0.5 → one toast call.
• test_allows_duplicate_after_one_second: two identical events at t=0 and t=1.1 → two toast calls.

Estimated effort: Small (2–3 hours) Dependencies: MON-PR2-003

---

PR-3: Integration + Polish

Goal: Validate the end-to-end event flow, add cross-stack integration tests, tune performance, update documentation, and ensure zero regression.

Estimated Lines: ~400 Review Risk: Medium

---

MON-PR3-001: Integration tests — lifecycle event flow

Description: Write backend integration tests that exercise real lifecycle endpoints and assert both DB audit rows and event bus publications.

Files to modify:

• tests/integration/test_lifecycle_hooks.py (new)

Acceptance criteria:

• test_start_publishes_started_event: call start endpoint; assert instance_events row with event_type="started" and event bus subscriber receives instance.started.
• test_stop_publishes_stopped_event: call stop endpoint; assert event_type="stopped" row and subscriber receives instance.stopped.
• test_restart_publishes_restarted_event: call restart endpoint; assert event_type="restarted".
• test_delete_publishes_deleted_event: call delete endpoint; assert event_type="deleted".
• test_created_by_set_to_user_id: user-initiated actions have created_by populated.
• Uses authenticated_client, db_session, and a test subscriber on InstanceEventBus.

Estimated effort: Medium (4–5 hours) Dependencies: PR-1 merged, PR-2 merged

---

MON-PR3-002: End-to-end tests — container start to toast

Description: Write an E2E test (Playwright or Cypress) that starts a container and verifies the toast sequence in the browser.

Files to modify:

• tests/e2e/container_monitoring.spec.ts (new)

Acceptance criteria:

• User clicks Start on an instance.
• Toast "Container starting..." appears within 3s.
• After readiness probe passes, toast "Container running" appears within 10s.
• No manual page refresh is performed between steps.
• Test is tagged @monitoring for selective CI runs.

Estimated effort: Medium (4–6 hours) Dependencies: PR-1 merged, PR-2 merged

---

MON-PR3-003: End-to-end tests — container crash detection

Description: Write an E2E test that kills a running container externally and verifies the error toast + badge update.

Files to modify:

• tests/e2e/container_monitoring.spec.ts

Acceptance criteria:

• Start a container and wait for "running" state.
• Kill the container via Docker CLI (or API call) from the test setup.
• Error toast appears within 5s.
• Status badge changes from green "running" to red "error" without page refresh.
• instance_events table contains event_type="error" with exit_code.

Estimated effort: Medium (4–6 hours) Dependencies: MON-PR3-002

---

MON-PR3-004: Performance tuning — connection limits and queue bounds

Description: Verify and harden performance constraints: SSE queue cap, heartbeat ping, and connection-per-user limit.

Files to modify:

• apps/api/src/api/events.py
• apps/web/src/hooks/use-events.ts

Acceptance criteria:

• Per-connection asyncio.Queue is capped at 100 events; oldest dropped on overflow.
• SSE ping (:ping) is sent every 30s and confirmed with a test.
• Max 5 connections per user is enforced and load-tested (even 10 rapid tab opens).
• Frontend reconnect jitter prevents thundering herd (simulate 50 clients disconnect/reconnect).
• Document any latency findings; no regressions in existing terminal WS.

Estimated effort: Small (2–3 hours) Dependencies: PR-1 merged, PR-2 merged

---

MON-PR3-005: Documentation updates

Description: Add user-facing and developer-facing documentation for the monitoring system.

Files to modify:

• docs/features/container-monitoring.md (new)
• docs/api/events.md (new)
• docs/architecture/event-bus.md (new)

Acceptance criteria:

• docs/features/container-monitoring.md explains real-time status, toasts, and health history to users.
• docs/api/events.md documents GET /events/stream auth, headers, reconnection strategy, and event payload schema.
• docs/architecture/event-bus.md documents the in-memory bus design, health monitor loop, and state machine.
• README or nav index updated with links to new docs.

Estimated effort: Small (2–3 hours) Dependencies: PR-1 merged, PR-2 merged

---

MON-PR3-006: Final cleanup and regression validation

Description: Run the full test suite, fix any flakes, remove debug logging, and verify no existing functionality is broken.

Files to modify:

• Any files with temporary debug code or TODOs introduced in PR-1/PR-2.

Acceptance criteria:

• pytest passes (unit + integration) with no failures.
• Frontend build passes with no TypeScript errors.
• Existing terminal WebSocket functionality verified manually or via existing E2E tests.
• Existing instance CRUD (create, start, stop, restart, delete) works end-to-end.
• Tunnel creation and recreation still function.
• No console.log or debug logger.debug left from development.
• All TODO comments resolved or converted to tracked issues.
• CHANGELOG or release notes entry added if project maintains one.

Estimated effort: Small (2–3 hours) Dependencies: MON-PR3-001, MON-PR3-002, MON-PR3-003, MON-PR3-004

---

Dependency Graph (PR Level)

PR-1: Backend Core
│
├─► MON-PR1-001 ──► MON-PR1-002
│
├─► MON-PR1-003 ──► MON-PR1-004
│                    └─► MON-PR1-008
│
├─► MON-PR1-005 ──► MON-PR1-006 ──► MON-PR1-009
│                    │
│                    └─► MON-PR1-007 ──► MON-PR1-012
│
├─► MON-PR1-010
│
└─► MON-PR1-011

PR-2: Frontend UI (depends on PR-1 merged)
│
├─► MON-PR2-001 ──► MON-PR2-002 ──► MON-PR2-004
│                    │
│                    └─► MON-PR2-005
│
├─► MON-PR2-003 ──► MON-PR2-004
│
├─► MON-PR2-006
│
└─► MON-PR2-007

PR-3: Integration + Polish (depends on PR-1 + PR-2 merged)
│
├─► MON-PR3-001
│
├─► MON-PR3-002 ──► MON-PR3-003
│
├─► MON-PR3-004
│
├─► MON-PR3-005
│
└─► MON-PR3-006

---

Task Summary

PR	Task ID	Description	Effort
1	MON-PR1-001	Alembic migration for monitoring tables	S
1	MON-PR1-002	SQLAlchemy models for InstanceEvent and HealthCheck	S
1	MON-PR1-003	Correlation ID context variable and middleware	S
1	MON-PR1-004	Structured JSON logging refactor	S
1	MON-PR1-005	InstanceEventBus singleton	S
1	MON-PR1-006	HealthMonitor background polling task	M
1	MON-PR1-007	SSE streaming endpoint	M
1	MON-PR1-008	Lifecycle hook instrumentation	M
1	MON-PR1-009	Wire up startup/shutdown and router registration	S
1	MON-PR1-010	Unit tests — EventBus	S
1	MON-PR1-011	Unit tests — HealthMonitor	M
1	MON-PR1-012	Integration tests — SSE endpoint	M
2	MON-PR2-001	Install sonner + TypeScript event types	S
2	MON-PR2-002	useEvents() SSE hook	M
2	MON-PR2-003	Toast rules and deduplication	S
2	MON-PR2-004	ToastProvider component	S
2	MON-PR2-005	Real-time badge updates + polling removal	M
2	MON-PR2-006	Unit tests — useEvents hook	S
2	MON-PR2-007	Unit tests — toast rules	S
3	MON-PR3-001	Integration tests — lifecycle event flow	M
3	MON-PR3-002	E2E tests — container start to toast	M
3	MON-PR3-003	E2E tests — container crash detection	M
3	MON-PR3-004	Performance tuning (limits, queue, jitter)	S
3	MON-PR3-005	Documentation updates	S
3	MON-PR3-006	Final cleanup and regression validation	S

Total tasks: 25
Total estimated effort: ~95 hours (backend ~55h, frontend ~25h, integration ~15h)