headquarter/openspec/changes/container-monitoring-notifications/apply-pr1.md

PR-1 Apply Report: Backend Core for Container Monitoring & Notifications

Status: COMPLETE

All 11 assigned tasks (MON-PR1-001 through MON-PR1-011) have been implemented and validated.

---

Changed Files

New Files (11)

File	Purpose
apps/api/alembic/versions/2026_05_28_add_monitoring_tables.py	Alembic migration creating instance_events + health_checks + 5 indexes
apps/api/src/models/instance_event.py	SQLAlchemy InstanceEvent model
apps/api/src/models/health_check.py	SQLAlchemy HealthCheck model
apps/api/src/services/event_bus.py	InstanceEventBus singleton with typed pub/sub
apps/api/src/services/health_monitor.py	HealthMonitor background polling task
apps/api/src/services/correlation.py	Async CORRELATION_ID context var + CorrelationIdMiddleware
apps/api/src/services/lifecycle_hooks.py	publish_lifecycle_event helper
apps/api/src/api/events.py	SSE endpoint GET /events/stream
apps/api/tests/unit/test_event_bus.py	Unit tests for EventBus
apps/api/tests/unit/test_health_monitor.py	Unit tests for HealthMonitor
apps/api/tests/unit/test_monitoring_models.py	Unit tests for new models

Modified Files (6)

File	Change
apps/api/src/models/__init__.py	Export InstanceEvent, HealthCheck
apps/api/src/api/__init__.py	Export events_router
apps/api/src/api/tool_instances.py	Lifecycle hooks at create/start/stop/restart/delete
apps/api/src/logging_config.py	JSON formatter + CorrelationIdFilter
apps/api/src/main.py	Register events router, middleware, HealthMonitor lifespan

---

Implementation Summary

MON-PR1-001/002: Database Migration

• Single Alembic revision 2026_05_28_add_monitoring_tables depends on current head.
• Creates instance_events (7 columns, 3 indexes) and health_checks (8 columns, 2 indexes).
• Proper FK constraints: ON DELETE CASCADE for instance_id, ON DELETE SET NULL for created_by.
• upgrade() and downgrade() both implemented.

MON-PR1-003/004: SQLAlchemy Models

• InstanceEvent: UUIDPrimaryKeyMixin, no TimestampMixin, created_at uses server_default.
• HealthCheck: UUIDPrimaryKeyMixin, checked_at uses server_default.
• Both exported in models/__init__.py for Alembic autogenerate.

MON-PR1-005: InstanceEventBus

• Singleton via __new__ + module-level _instance.
• subscribe(event_type, callback) returns unsubscribe callable.
• publish(event_type, payload) delivers in same event loop iteration.
• Exception isolation: subscriber failures are logged and delivery continues.
• Added wildcard "*" subscription support for SSE endpoint.

MON-PR1-006: HealthMonitor

• Accepts event_bus in constructor; poll interval 15.0s (overridable in tests).
• start() is idempotent; stop() cancels task and clears _last_known_state.
• Queries instances with status NOT IN ("pending", "stopped", "error").
• Per instance: get_container_status() + check_tunnel_health() if public_url present.
• State-change gating via HealthSnapshot dataclass; writes to DB + publishes events only on change.
• Per-instance exceptions caught and logged as structured JSON; loop continues.

MON-PR1-007: SSE Endpoint

• GET /events/stream authenticated via existing get_current_user_id cookie/JWT.
• Returns 401 before stream start if auth missing; 429 if >5 concurrent connections per user.
• Per-connection asyncio.Queue(maxsize=100) drops oldest on overflow.
• :ping comment every 30 seconds.
• On disconnect: unsubscribes from EventBus and releases connection slot.

MON-PR1-008: Lifecycle Hooks

• lifecycle_hooks.py provides publish_lifecycle_event() which writes instance_events row + publishes to EventBus.
• Instrumented in tool_instances.py:
  • create_instance → instance.created
  • start_instance → instance.started (at "starting"), instance.error (on crash), instance.health_changed (probe success/failure)
  • stop_instance → instance.stopped
  • restart_instance → instance.restarted
  • delete_instance → instance.deleted (before row deletion)

MON-PR1-009: Structured JSON Logging

• logging_config.py replaced plain-text formatter with JSONFormatter.
• Fields: timestamp, level, logger, message, correlation_id, plus optional instance_id/event_type from extra=.
• CorrelationIdMiddleware reads X-Request-ID or generates UUID; sets async context var.
• uvicorn.access remains at WARNING.

MON-PR1-010/011: Unit Tests

• EventBus: 6 tests covering pub/sub, exception isolation, unsubscribe, empty list, async subscriber, unsubscribe_all.
• HealthMonitor: 6 tests covering crash detection, tunnel failure, recovery, skip on no change, Docker exception resilience, start/stop lifecycle.
• All tests use fresh EventBus instances (_reset_for_testing) and mocked Docker/HTTP responses.

---

Test Commands & Exit Codes

# Focused new tests
cd apps/api && python -m pytest tests/unit/test_event_bus.py tests/unit/test_health_monitor.py tests/unit/test_monitoring_models.py -v
# Exit code: 0  (15 passed)

# Full unit suite — no regressions from this PR
cd apps/api && python -m pytest tests/unit/ -v
# Exit code: 1  (172 passed, 4 failed — all pre-existing failures in test_config.py and test_git_repository_clone_preflight.py)

# Ruff linting
cd apps/api && python -m ruff check src/services/event_bus.py src/services/health_monitor.py src/services/correlation.py src/services/lifecycle_hooks.py src/api/events.py src/models/instance_event.py src/models/health_check.py src/models/__init__.py src/logging_config.py src/main.py src/api/__init__.py alembic/versions/2026_05_28_add_monitoring_tables.py
# Exit code: 0  (All checks passed)

---

Surprises & Decisions

1. metadata column collision: SQLAlchemy DeclarativeBase reserves metadata as a class-level MetaData attribute. Workaround: Python attribute named event_metadata with mapped_column("metadata", ...) to preserve the DB column name.
2. SQLite JSONB incompatibility: Used generic JSON type in SQLAlchemy models so SQLite-based unit tests work. Migration still uses sa.JSON() which is portable.
3. Delete audit row survivability: ON DELETE CASCADE on instance_events.instance_id means the instance.deleted audit row cannot survive the instance deletion. Inserted before deletion so it exists briefly; event bus publication is the durable signal.
4. Integration tests require asyncpg: Existing integration tests fail locally because asyncpg is not installed in the host Python environment. These are pre-existing infrastructure limitations, not regressions.
5. EventBus wildcard: Added "*" support to publish() so the SSE endpoint can subscribe once and receive all event types without maintaining a list of subscriptions.

---

PR Boundary

This PR includes the complete backend core for container monitoring. The next PR (PR-2) should cover:

• Frontend useEvents() SSE hook
• ToastProvider + toast-rules.ts
• Real-time badge updates and polling removal

The final PR (PR-3) should cover:

• Integration tests for SSE and lifecycle hooks
• E2E tests
• Documentation