# PR-1 Apply Report: Backend Core for Container Monitoring & Notifications

## Status: COMPLETE

All 11 assigned tasks (MON-PR1-001 through MON-PR1-011) have been implemented and validated.

---

## Changed Files

### New Files (11)
| File | Purpose |
|------|---------|
| `apps/api/alembic/versions/2026_05_28_add_monitoring_tables.py` | Alembic migration creating `instance_events` + `health_checks` + 5 indexes |
| `apps/api/src/models/instance_event.py` | SQLAlchemy `InstanceEvent` model |
| `apps/api/src/models/health_check.py` | SQLAlchemy `HealthCheck` model |
| `apps/api/src/services/event_bus.py` | `InstanceEventBus` singleton with typed pub/sub |
| `apps/api/src/services/health_monitor.py` | `HealthMonitor` background polling task |
| `apps/api/src/services/correlation.py` | Async `CORRELATION_ID` context var + `CorrelationIdMiddleware` |
| `apps/api/src/services/lifecycle_hooks.py` | `publish_lifecycle_event` helper |
| `apps/api/src/api/events.py` | SSE endpoint `GET /events/stream` |
| `apps/api/tests/unit/test_event_bus.py` | Unit tests for EventBus |
| `apps/api/tests/unit/test_health_monitor.py` | Unit tests for HealthMonitor |
| `apps/api/tests/unit/test_monitoring_models.py` | Unit tests for new models |

### Modified Files (6)
| File | Change |
|------|--------|
| `apps/api/src/models/__init__.py` | Export `InstanceEvent`, `HealthCheck` |
| `apps/api/src/api/__init__.py` | Export `events_router` |
| `apps/api/src/api/tool_instances.py` | Lifecycle hooks at create/start/stop/restart/delete |
| `apps/api/src/logging_config.py` | JSON formatter + `CorrelationIdFilter` |
| `apps/api/src/main.py` | Register events router, middleware, HealthMonitor lifespan |

---

## Implementation Summary

### MON-PR1-001/002: Database Migration
- Single Alembic revision `2026_05_28_add_monitoring_tables` depends on current head.
- Creates `instance_events` (7 columns, 3 indexes) and `health_checks` (8 columns, 2 indexes).
- Proper FK constraints: `ON DELETE CASCADE` for `instance_id`, `ON DELETE SET NULL` for `created_by`.
- `upgrade()` and `downgrade()` both implemented.

### MON-PR1-003/004: SQLAlchemy Models
- `InstanceEvent`: `UUIDPrimaryKeyMixin`, no `TimestampMixin`, `created_at` uses `server_default`.
- `HealthCheck`: `UUIDPrimaryKeyMixin`, `checked_at` uses `server_default`.
- Both exported in `models/__init__.py` for Alembic autogenerate.

### MON-PR1-005: InstanceEventBus
- Singleton via `__new__` + module-level `_instance`.
- `subscribe(event_type, callback)` returns unsubscribe callable.
- `publish(event_type, payload)` delivers in same event loop iteration.
- Exception isolation: subscriber failures are logged and delivery continues.
- Added wildcard `"*"` subscription support for SSE endpoint.

### MON-PR1-006: HealthMonitor
- Accepts `event_bus` in constructor; poll interval `15.0s` (overridable in tests).
- `start()` is idempotent; `stop()` cancels task and clears `_last_known_state`.
- Queries instances with `status NOT IN ("pending", "stopped", "error")`.
- Per instance: `get_container_status()` + `check_tunnel_health()` if `public_url` present.
- State-change gating via `HealthSnapshot` dataclass; writes to DB + publishes events only on change.
- Per-instance exceptions caught and logged as structured JSON; loop continues.

### MON-PR1-007: SSE Endpoint
- `GET /events/stream` authenticated via existing `get_current_user_id` cookie/JWT.
- Returns `401` before stream start if auth missing; `429` if >5 concurrent connections per user.
- Per-connection `asyncio.Queue(maxsize=100)` drops oldest on overflow.
- `:ping` comment every 30 seconds.
- On disconnect: unsubscribes from EventBus and releases connection slot.

### MON-PR1-008: Lifecycle Hooks
- `lifecycle_hooks.py` provides `publish_lifecycle_event()` which writes `instance_events` row + publishes to EventBus.
- Instrumented in `tool_instances.py`:
  - `create_instance` → `instance.created`
  - `start_instance` → `instance.started` (at "starting"), `instance.error` (on crash), `instance.health_changed` (probe success/failure)
  - `stop_instance` → `instance.stopped`
  - `restart_instance` → `instance.restarted`
  - `delete_instance` → `instance.deleted` (before row deletion)

### MON-PR1-009: Structured JSON Logging
- `logging_config.py` replaced plain-text formatter with `JSONFormatter`.
- Fields: `timestamp`, `level`, `logger`, `message`, `correlation_id`, plus optional `instance_id`/`event_type` from `extra=`.
- `CorrelationIdMiddleware` reads `X-Request-ID` or generates UUID; sets async context var.
- `uvicorn.access` remains at `WARNING`.

### MON-PR1-010/011: Unit Tests
- EventBus: 6 tests covering pub/sub, exception isolation, unsubscribe, empty list, async subscriber, unsubscribe_all.
- HealthMonitor: 6 tests covering crash detection, tunnel failure, recovery, skip on no change, Docker exception resilience, start/stop lifecycle.
- All tests use fresh EventBus instances (`_reset_for_testing`) and mocked Docker/HTTP responses.

---

## Test Commands & Exit Codes

```bash
# Focused new tests
cd apps/api && python -m pytest tests/unit/test_event_bus.py tests/unit/test_health_monitor.py tests/unit/test_monitoring_models.py -v
# Exit code: 0  (15 passed)

# Full unit suite — no regressions from this PR
cd apps/api && python -m pytest tests/unit/ -v
# Exit code: 1  (172 passed, 4 failed — all pre-existing failures in test_config.py and test_git_repository_clone_preflight.py)

# Ruff linting
cd apps/api && python -m ruff check src/services/event_bus.py src/services/health_monitor.py src/services/correlation.py src/services/lifecycle_hooks.py src/api/events.py src/models/instance_event.py src/models/health_check.py src/models/__init__.py src/logging_config.py src/main.py src/api/__init__.py alembic/versions/2026_05_28_add_monitoring_tables.py
# Exit code: 0  (All checks passed)
```

---

## Surprises & Decisions

1. **`metadata` column collision**: SQLAlchemy `DeclarativeBase` reserves `metadata` as a class-level `MetaData` attribute. Workaround: Python attribute named `event_metadata` with `mapped_column("metadata", ...)` to preserve the DB column name.
2. **SQLite `JSONB` incompatibility**: Used generic `JSON` type in SQLAlchemy models so SQLite-based unit tests work. Migration still uses `sa.JSON()` which is portable.
3. **Delete audit row survivability**: `ON DELETE CASCADE` on `instance_events.instance_id` means the `instance.deleted` audit row cannot survive the instance deletion. Inserted before deletion so it exists briefly; event bus publication is the durable signal.
4. **Integration tests require `asyncpg`**: Existing integration tests fail locally because `asyncpg` is not installed in the host Python environment. These are pre-existing infrastructure limitations, not regressions.
5. **EventBus wildcard**: Added `"*"` support to `publish()` so the SSE endpoint can subscribe once and receive all event types without maintaining a list of subscriptions.

---

## PR Boundary

This PR includes the complete backend core for container monitoring. The next PR (PR-2) should cover:
- Frontend `useEvents()` SSE hook
- `ToastProvider` + `toast-rules.ts`
- Real-time badge updates and polling removal

The final PR (PR-3) should cover:
- Integration tests for SSE and lifecycle hooks
- E2E tests
- Documentation