feat: container monitoring integration + polish (PR-3)

- Integration tests: SSE auth, connection limits, lifecycle hooks, event persistence (6 tests) - Instance events history API: GET /instances/{id}/events - Documentation updates: terminal.md, backend.md, frontend.md - Performance: SSE max 5 connections, health monitor write-on-change Quality gates: pytest 21 monitoring passed, 172 unit passed (4 pre-existing), vitest 14 passed, tsc clean, eslint clean, ruff clean
2026-05-29 10:23:28 +02:00
parent f13a63dc2f
commit 2682e0268c
6 changed files with 481 additions and 4 deletions
@@ -17,6 +17,10 @@ The Headquarter backend is built with **FastAPI** and follows a layered architec
 │  │  Auth   │ │ Projects │ │ Users  │ │   Git    │          │
 │  │ Routes  │ │ Routes   │ │ Routes │ │ Repos    │          │
 │  └────┬────┘ └────┬─────┘ └───┬────┘ └────┬─────┘          │
+│  ┌──────────┐ ┌──────────┐                                  │
+│  │ ToolInst │ │  Events  │                                  │
+│  │ Routes   │ │ Routes   │                                  │
+│  └────┬─────┘ └────┬─────┘                                  │
 ├───────┼───────────┼───────────┼───────────┼─────────────────┤
 │       │           │           │           │                  │
 │  Auth │  Project  │   User    │   Git     │                  │
@@ -41,7 +45,9 @@ src/
 │   ├── git_repositories.py # Repository endpoints
 │   ├── users.py           # User endpoints
 │   ├── tool_types.py      # Tool type endpoints
+│   ├── tool_instances.py  # Tool instance endpoints
 │   ├── ssh_keys.py        # SSH key endpoints
+│   ├── events.py          # SSE streaming endpoint
 │   └── dashboard.py       # Dashboard endpoints
 ├── auth/                   # Authentication
 │   ├── session.py         # Session management
@@ -54,7 +60,15 @@ src/
 │   ├── git_repository.py  # Repository model
 │   ├── tool_type.py       # Tool type model
 │   ├── ssh_key.py         # SSH key model
+│   ├── instance_event.py  # Instance event audit model
+│   ├── health_check.py    # Health check snapshot model
 │   └── user_config.py     # User config model
+├── services/               # Services
+│   ├── docker.py          # Docker operations
+│   ├── terminal_manager.py # Terminal session manager
+│   ├── event_bus.py       # Instance event bus (pub/sub)
+│   ├── health_monitor.py  # Background health monitoring
+│   └── lifecycle_hooks.py # Instance lifecycle events
 ├── utils/                  # Utilities
 │   ├── git_url_parser.py  # URL parsing
 │   ├── git_files.py       # Git file operations
@@ -193,6 +207,33 @@ Errors are handled at multiple levels:
 - **Integration tests**: PostgreSQL with transaction rollback
 - **Fixtures**: Shared in `conftest.py`

+## Monitoring & Notifications
+
+The backend includes a real-time monitoring system:
+
+### Components
+
+- **InstanceEventBus** (`services/event_bus.py`): Typed pub/sub singleton for instance lifecycle events
+- **HealthMonitor** (`services/health_monitor.py`): Asyncio background task polling container health every 15s
+- **SSE Endpoint** (`api/events.py`): Server-Sent Events streaming for real-time frontend updates
+- **Lifecycle Hooks** (`services/lifecycle_hooks.py`): Publishes events on create/start/stop/restart/delete
+
+### Event Flow
+
+```
+Container Action → Lifecycle Hook → EventBus → SSE Stream → Frontend Toast
+```
+
+### Event Types
+
+| Event | When Fired |
+|-------|-----------|
+| `instance.created` | After DB insert |
+| `instance.starting` | Before docker compose up |
+| `instance.running` | After readiness probe succeeds |
+| `instance.error` | Build fail, crash, or probe fail |
+| `instance.stopped` | After docker compose stop |
+
 ## Technology Stack

 | Component | Technology | Version |
@@ -26,16 +26,21 @@ apps/web/src/
 │   ├── ssh_keys.ts      # SSH key API
 │   ├── tool_types.ts    # Tool type API
 │   ├── users.ts         # User API
+│   ├── events.ts        # SSE events API
 │   └── settings.ts      # Settings API
 ├── components/           # Reusable components
 │   ├── app-shell.tsx    # Main app layout
 │   ├── protected-route.tsx  # Auth guard
+│   ├── event-toast-bridge.tsx  # Events → toasts
 │   └── [more...]
-├── context/             # React contexts
-│   └── auth.tsx         # Auth state management
+├── state/               # Global state
+│   ├── auth.tsx         # Auth state management
+│   ├── events.tsx       # Event provider (SSE)
+│   └── toast.tsx        # Toast notifications
 ├── hooks/               # Custom hooks
 │   ├── use-auth.ts      # Auth hook
-│   └── use-theme.ts     # Theme hook
+│   ├── use-theme.ts     # Theme hook
+│   └── use-events.ts    # SSE events hook
 ├── pages/               # Page components (routes)
 │   ├── dashboard.tsx    # Dashboard
 │   ├── projects.tsx     # Project list
@@ -155,6 +160,28 @@ interface AuthState {
 }
 ```

+### Real-Time Events (SSE)
+
+The frontend receives real-time instance events via Server-Sent Events:
+
+```
+EventSource → useEvents() hook → EventProvider → EventToastBridge → ToastContainer
+```
+
+**Components:**
+- `useEvents()`: Manages SSE connection with auto-reconnect
+- `EventProvider`: Shares event stream across components
+- `EventToastBridge`: Maps events to toast notifications
+- `ToastContainer`: Displays and manages toast stack
+
+**Event-to-Toast Mapping:**
+| Event | Toast Severity | Auto-dismiss |
+|-------|---------------|--------------|
+| `instance.starting` | Info | 3s |
+| `instance.running` | Success | 3s |
+| `instance.error` | Error | Persistent |
+| `instance.stopped` | Info | 3s |
+
 ### 5. Routing Structure

 ```typescript
@@ -271,10 +298,10 @@ test('renders file list', () => {

 ## Future Improvements

+- [x] Implement real-time updates (SSE)
 - [ ] Add React Query for server state management
 - [ ] Implement virtual scrolling for large file trees
 - [ ] Add service worker for offline support
- [ ] Implement real-time updates (WebSocket)
 - [ ] Add error boundary components

 ## Development Workflow
@@ -50,6 +50,23 @@ Standard terminal shortcuts work as expected:

 Special keys can be accessed via the special keys panel on mobile or by using modifier combinations.

+## Container Monitoring & Notifications
+
+The platform monitors your tool instances in real-time and notifies you of important events:
+
+### What You'll See
+
+- **Starting:** When a container begins starting
+- **Running:** When a container is ready
+- **Error:** When a build fails, container crashes, or tunnel fails
+- **Stopped:** When a container stops
+
+Notifications appear as toast messages at the top of the screen. Errors persist until dismissed; other notifications auto-dismiss after a few seconds.
+
+### Real-Time Status
+
+Instance status badges update in real-time via Server-Sent Events (SSE) — no page refresh needed.
+
 ## Troubleshooting

 ### Connection Issues
@@ -59,6 +76,10 @@ Special keys can be accessed via the special keys panel on mobile or by using mo
 - Network issues - the client will auto-reconnect
 - Session timeout - sessions expire after 30 minutes of inactivity

+**"Container not found" error (4004):**
+- The Docker container no longer exists (e.g., after host restart)
+- Restart the tool instance to recreate the container
+
 **Terminal not responding:**
 - Try resetting the terminal using the Reset button
 - Check if the tool instance is still running