From 089d802f1d96a0a49ee4da708c93b067664ac676 Mon Sep 17 00:00:00 2001 From: Developer Date: Sun, 14 Jun 2026 21:52:02 +0000 Subject: [PATCH] fix: prevent failed containers from showing as running on dashboard - Add final get_container_status check in start_tool_instance before writing status=running; mark as error and return logs if container stopped - Treat restarting as error in HealthMonitor when DB status was already running, so crash loops are surfaced instead of preserved - Disable auto-restart (restart: unless-stopped -> restart: no) for tool instances in manifest compiler, legacy dockerfile path, and built-in seeds Quality gates: - pytest tests/unit: 210 passed - ruff: clean on changed files - mypy: clean on changed files --- .pi-map.index.md | 3 +- .pi-map.md | 7 +- apps/.pi-map.index.md | 2 +- apps/.pi-map.md | 4 +- apps/api/.pi-map.md | 2 +- apps/api/src/.pi-map.index.md | 2 +- apps/api/src/.pi-map.md | 4 +- apps/api/src/seeds/.pi-map.index.md | 2 +- apps/api/src/seeds/.pi-map.md | 6 +- apps/api/src/seeds/builtin_tool_types.py | 6 +- apps/api/src/services/.pi-map.index.md | 2 +- apps/api/src/services/.pi-map.md | 4 +- .../src/services/build/manifest_compiler.py | 2 +- .../src/services/instance/.pi-map.index.md | 2 +- apps/api/src/services/instance/.pi-map.md | 6 +- .../src/services/instance/health_monitor.py | 5 ++ apps/api/src/services/tool/.pi-map.index.md | 2 +- apps/api/src/services/tool/.pi-map.md | 6 +- .../api/src/services/tool/instance_service.py | 79 +++++++++++++++---- openspec/.pi-map.index.md | 2 +- openspec/changes/.pi-map.index.md | 5 +- .../.pi-map.index.md | 20 +++++ .../.pi-map.md | 20 +++++ .../change.md | 32 ++++++++ .../tasks.md | 8 ++ 25 files changed, 185 insertions(+), 48 deletions(-) create mode 100644 openspec/changes/fix-container-status-false-positive/.pi-map.index.md create mode 100644 openspec/changes/fix-container-status-false-positive/.pi-map.md create mode 100644 openspec/changes/fix-container-status-false-positive/change.md create mode 100644 openspec/changes/fix-container-status-false-positive/tasks.md diff --git a/.pi-map.index.md b/.pi-map.index.md index ba97793..058fb6f 100644 --- a/.pi-map.index.md +++ b/.pi-map.index.md @@ -16,7 +16,7 @@ dir: . Trust boundary: index routes, map orients, source decides. ## role -Infrastructure and deployment configuration for a self-hosted project management platform with containerized services, SSO integration, and AI agent development workflows. +A self-hosted web-based project management and git repository platform with OAuth2 SSO authentication, containerized for Docker deployment. ## parent - ## children @@ -69,6 +69,7 @@ Infrastructure and deployment configuration for a self-hosted project management - CHANGELOG.md - Makefile - README.md +- context.md - docker-compose.traefik.yml - docker-compose.yml - progress.md diff --git a/.pi-map.md b/.pi-map.md index 825f002..6d65a92 100644 --- a/.pi-map.md +++ b/.pi-map.md @@ -18,7 +18,7 @@ index: ./.pi-map.index.md Trust boundary: index routes, map orients, source decides. ## role -Infrastructure and deployment configuration for a self-hosted project management platform with containerized services, SSO integration, and AI agent development workflows. +A self-hosted web-based project management and git repository platform with OAuth2 SSO authentication, containerized for Docker deployment. ## files - .env.example | Provides a template of environment variables for configuring a Headquarter application with PostgreSQL, Redis, Authentik SSO, and Docker/Traefik deployment - .gitignore | Specifies files and directories for Git to ignore across a multi-language project with Python, Node, and custom tooling | dep: Git @@ -26,14 +26,15 @@ Infrastructure and deployment configuration for a self-hosted project management - CHANGELOG.md | Documents version history and notable changes for a Git-based project management web application - Makefile | Provides standard development commands for containerized web application lifecycle management via Docker Compose | dep: docker compose, alembic, pytest, ruff, mypy, playwright, npm, postgres, redis - README.md | A self-hosted platform for managing projects, git repositories, and development tools with OAuth2 authentication. | dep: FastAPI, SQLAlchemy, Pydantic, Alembic, python-jose, React, TypeScript, Vite, React Router, Docker, PostgreSQL, Traefik, Authentik, Git +- context.md | Documents a bug investigation where crashed Docker containers incorrectly display "running" status on the dashboard due to optimistic status writes, auto-restart policies, and health monitor logic that preserves running status for restarting containers. | dep: docker, docker-compose, FastAPI/SQLAlchemy, SSE event bus, React/TypeScript frontend - docker-compose.traefik.yml | Deploys a multi-service web application (frontend, API, PostgreSQL, Redis) behind an existing Traefik reverse proxy with TLS termination and environment-configurable domains. | dep: docker, traefik, postgres, redis, authentik, docker-compose - docker-compose.yml | Defines a multi-service Docker Compose stack with PostgreSQL, Redis, API backend, and web frontend services for a "headquarter" application | dep: Docker, PostgreSQL, Redis, Vite, asyncpg, nginx - progress.md | Tracks completed and remaining tasks for a backend-frontend code refactoring project organized in 7 phases - swap-pane | Empty file with no functionality ## arch -Docker Compose-based microservices architecture with PostgreSQL/Redis data layer, Traefik reverse proxy, OAuth2 SSO via Authentik, and OpenSpec-driven AI agent orchestration. +Full-stack containerized architecture with separate frontend/API services, PostgreSQL/Redis data layer, Traefik reverse proxy with TLS, environment-driven configuration, and health monitoring with auto-restart policies. ## tags -docker, redis, git, application, postgresql, compose, traefik, project +docker, redis, git, compose, application, postgresql, traefik, project ## symbols - ## workflows diff --git a/apps/.pi-map.index.md b/apps/.pi-map.index.md index 25f1413..3f1ff04 100644 --- a/apps/.pi-map.index.md +++ b/apps/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps ## role -Contains the deployable application entry points and main executables for the project. +Contains the deployable application entry points and executable configurations for the project. ## parent index: ./.pi-map.index.md map: ./.pi-map.md diff --git a/apps/.pi-map.md b/apps/.pi-map.md index 1b3844a..07cdb24 100644 --- a/apps/.pi-map.md +++ b/apps/.pi-map.md @@ -4,10 +4,10 @@ dir: apps index: apps/.pi-map.index.md ## role -Contains the deployable application entry points and main executables for the project. +Contains the deployable application entry points and executable configurations for the project. ## files ## arch -Typically follows a multi-app workspace pattern where each subdirectory represents a standalone application with its own main.ts, configuration, and dependency injection setup, often sharing common libraries from other packages. +Modular application structure where each subdirectory represents a separate deployable unit or service, following a monorepo or multi-app pattern. ## tags - ## symbols diff --git a/apps/api/.pi-map.md b/apps/api/.pi-map.md index a863945..1b68790 100644 --- a/apps/api/.pi-map.md +++ b/apps/api/.pi-map.md @@ -14,7 +14,7 @@ FastAPI backend API that manages projects, git repositories, and development too - uv.lock | Lock file for the uv Python package manager that pins exact dependency versions and their artifact hashes for reproducible installations | dep: uv, Python 3.11+, aiosqlite, alembic, annotated-doc, annotated-types, anyio, ast-serialize, asyncpg, and many other PyPI packages - wait-for-db.sh | Wait for a PostgreSQL database to become available before executing a command, with configurable retry logic. | dep: nc (netcat), sh (POSIX shell), sleep ## arch -Async Python/FastAPI service using multi-stage Docker builds, Alembic migrations, uv package management, and container orchestration with external tunneling and database dependency waiting patterns. +Async Python/FastAPI service with multi-stage Docker containerization, Alembic database migrations, uv package management, and Docker socket access for container orchestration. ## tags docker, alembic, python, database, fastapi, postgresql, asyncpg, uvicorn ## symbols diff --git a/apps/api/src/.pi-map.index.md b/apps/api/src/.pi-map.index.md index 1c13f68..f6a693d 100644 --- a/apps/api/src/.pi-map.index.md +++ b/apps/api/src/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src ## role -Core API application package for the "Headquarter API" providing configuration, database infrastructure, logging, and FastAPI application initialization. +Core API server package for the "Headquarter" backend, handling configuration, database connectivity, logging, and FastAPI application initialization. ## parent index: apps/api/.pi-map.index.md map: apps/api/.pi-map.md diff --git a/apps/api/src/.pi-map.md b/apps/api/src/.pi-map.md index 5f6cbbc..c2d8975 100644 --- a/apps/api/src/.pi-map.md +++ b/apps/api/src/.pi-map.md @@ -4,7 +4,7 @@ dir: apps/api/src index: apps/api/src/.pi-map.index.md ## role -Core API application package for the "Headquarter API" providing configuration, database infrastructure, logging, and FastAPI application initialization. +Core API server package for the "Headquarter" backend, handling configuration, database connectivity, logging, and FastAPI application initialization. ## files - __init__.py | Marks the directory as a Python package for the Headquarter API. - config.py | Defines application configuration settings with environment-based overrides using Pydantic, including database URLs, service domains, OAuth/Authentik integration, JWT/session settings, and computed properties for environment-specific behavior. | exp: class:Settings, func:build_database_url(user: str, password: str, host: str, port: int, database: str) → str | dep: pydantic, pydantic_settings @@ -12,7 +12,7 @@ Core API application package for the "Headquarter API" providing configuration, - logging_config.py | Configures structured JSON logging with correlation ID injection, custom formatters, and HTTP request/exception middleware for a FastAPI application. | exp: class:CorrelationIdFilter, method:filter(self, record: logging.LogRecord) → bool, call:get_correlation_id, class:JSONFormatter, method:format(self, record: logging.LogRecord) → str, call:self.formatTime, call:record.getMessage, call:getattr, call:self.formatException, call:json.dumps, method:formatTime(self, record: logging.LogRecord, datefmt) → str, call:time.strftime, call:time.gmtime, class:RequestLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:time.time, call:logger.info, call:call_next, call:int, call:logger.error, call:type, call:traceback.format_exc, class:ExceptionLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:call_next, call:logger.critical, call:traceback.format_exc, func:configure_logging(level) → None, call:JSONFormatter, call:logging.StreamHandler, call:console_handler.setFormatter, call:console_handler.addFilter, call:CorrelationIdFilter, call:root_logger.setLevel, call:logging.getLogger("uvicorn").setLevel, call:logging.getLogger("uvicorn.access").setLevel, call:logging.getLogger("sqlalchemy.engine").setLevel, call:logger.info, call:logging.getLevelName | dep: json, logging, sys, time, traceback, collections.abc, fastapi, starlette.middleware.base, src.services.shared.correlation - main.py | Initializes and configures a FastAPI application for the "Headquarter API" with database setup, middleware, routing, and background services. | exp: func:_sanitize_validation_errors(errors), call:error.get, call:str, call:ctx.items, call:isinstance, call:type, call:sanitized.append, func:validation_exception_handler(request: Request, exc: RequestValidationError), call:exc.errors, call:logger.warning, call:_sanitize_validation_errors, call:JSONResponse, func:on_startup(), call:logger.info, call:init_database, call:logger.error, call:sys.exit, call:_health_monitor.start, call:seed_builtin_tool_types, func:on_shutdown(), call:logger.info, call:_health_monitor.stop | dep: logging, os, fastapi, fastapi.exceptions, fastapi.middleware.cors, fastapi.responses, fastapi.staticfiles, src.api.config, src.api.project, src.api.system, src.api.tool, src.api.user, src.api.workspace, src.config, src.models, src.database, src.logging_config, src.seeds.builtin_tool_types, src.services.instance, src.services.shared, sys, src.api.* ## arch -Layered architecture with Pydantic-based config management, async SQLAlchemy with Alembic migrations, structured JSON logging with correlation IDs, and FastAPI middleware/routing pattern. +Layered async architecture using FastAPI with Pydantic settings management, SQLAlchemy async ORM with Alembic migrations, structured JSON logging with correlation ID tracking, and environment-driven configuration with OAuth/Authentik integration. ## tags src, database, logging, call:logger.info, api, middleware, fastapi, filter ## symbols diff --git a/apps/api/src/seeds/.pi-map.index.md b/apps/api/src/seeds/.pi-map.index.md index ebf2352..504a1a7 100644 --- a/apps/api/src/seeds/.pi-map.index.md +++ b/apps/api/src/seeds/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/seeds ## role -Provides database seeding utilities for initializing built-in tool type configurations in the API application. +Provides database seeding utilities for initializing and synchronizing built-in data records in the API application. ## parent index: apps/api/src/.pi-map.index.md map: apps/api/src/.pi-map.md diff --git a/apps/api/src/seeds/.pi-map.md b/apps/api/src/seeds/.pi-map.md index b053518..fe374a1 100644 --- a/apps/api/src/seeds/.pi-map.md +++ b/apps/api/src/seeds/.pi-map.md @@ -4,12 +4,12 @@ dir: apps/api/src/seeds index: apps/api/src/seeds/.pi-map.index.md ## role -Provides database seeding utilities for initializing built-in tool type configurations in the API application. +Provides database seeding utilities for initializing and synchronizing built-in data records in the API application. ## files - __init__.py | Marks the directory as a Python package for database seeding utilities. -- builtin_tool_types.py | Seeds built-in tool types (code-server, jupyter-notebook, opencode) into a database with upsert logic, creating or updating Docker Compose-based development environment templates. | exp: func:_table_exists(session, table_name: str) → bool, call:session.execute, call:text, call:result.scalar, func:seed_builtin_tool_types(), call:SessionLocal, call:_table_exists, call:logger.warning, call:session.scalar, call:select(ToolType).where, call:ToolType, call:tool_data.get, call:session.add, call:logger.info, call:session.commit | dep: logging, sqlalchemy, src.database, src.models +- builtin_tool_types.py | Seeds predefined built-in tool types (code-server, jupyter-notebook, opencode) into a database with upsert logic, creating them if missing or updating existing ones to match code changes. | exp: func:_table_exists(session, table_name: str) → bool, call:session.execute, call:text, call:result.scalar, func:seed_builtin_tool_types(), call:SessionLocal, call:_table_exists, call:logger.warning, call:session.scalar, call:select(ToolType).where, call:ToolType, call:tool_data.get, call:session.add, call:logger.info, call:session.commit | dep: logging, sqlalchemy, src.database, src.models ## arch -Simple procedural seeding script using SQLAlchemy upsert operations to populate reference data for containerized development environment templates. +Simple imperative seeding scripts with upsert pattern for idempotent data initialization, using direct database operations without abstraction layers. ## tags tool, types, table, exists, builtin, call:tool, database, init ## symbols diff --git a/apps/api/src/seeds/builtin_tool_types.py b/apps/api/src/seeds/builtin_tool_types.py index c1f0184..2afe1ef 100644 --- a/apps/api/src/seeds/builtin_tool_types.py +++ b/apps/api/src/seeds/builtin_tool_types.py @@ -65,7 +65,7 @@ services: - {{REPO_PATH}}:/config/workspace ports: - "8443:8443" - restart: unless-stopped""", + restart: 'no'""", "default_port": 8443, "required_variables": ["REPO_PATH", "TOOL_NAME"], }, @@ -87,7 +87,7 @@ services: - {{REPO_PATH}}:/home/jovyan/work ports: - "8888:8888" - restart: unless-stopped""", + restart: 'no'""", "required_variables": ["REPO_PATH", "TOOL_NAME"], }, { @@ -122,7 +122,7 @@ services: exec tail -f /dev/null" stdin_open: true tty: true - restart: unless-stopped""", + restart: 'no'""", "required_variables": ["REPO_PATH", "TOOL_NAME"], }, ] diff --git a/apps/api/src/services/.pi-map.index.md b/apps/api/src/services/.pi-map.index.md index 802a994..6ac8421 100644 --- a/apps/api/src/services/.pi-map.index.md +++ b/apps/api/src/services/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/services ## role -Provides business logic and service layer abstractions for the API application. +Marker package for the services layer in the API application ## parent index: apps/api/src/.pi-map.index.md map: apps/api/src/.pi-map.md diff --git a/apps/api/src/services/.pi-map.md b/apps/api/src/services/.pi-map.md index e855e0b..2e73231 100644 --- a/apps/api/src/services/.pi-map.md +++ b/apps/api/src/services/.pi-map.md @@ -4,11 +4,11 @@ dir: apps/api/src/services index: apps/api/src/services/.pi-map.index.md ## role -Provides business logic and service layer abstractions for the API application. +Marker package for the services layer in the API application ## files - __init__.py | Empty file with no functionality ## arch -Minimal or placeholder package structure with no implemented services yet, following standard Python package conventions. +Standard Python package structure using __init__.py for namespace declaration ## tags init, empty, functionality ## symbols diff --git a/apps/api/src/services/build/manifest_compiler.py b/apps/api/src/services/build/manifest_compiler.py index 47473f7..518c99f 100644 --- a/apps/api/src/services/build/manifest_compiler.py +++ b/apps/api/src/services/build/manifest_compiler.py @@ -411,7 +411,7 @@ def compile_compose(manifest: dict, variables: dict[str, Any]) -> str: service: dict[str, Any] = { "image": variables["IMAGE_TAG"], "container_name": variables["INSTANCE_NAME"], - "restart": "unless-stopped", + "restart": "no", } # Terminal-specific fields diff --git a/apps/api/src/services/instance/.pi-map.index.md b/apps/api/src/services/instance/.pi-map.index.md index 94a6e22..a9ec2aa 100644 --- a/apps/api/src/services/instance/.pi-map.index.md +++ b/apps/api/src/services/instance/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/services/instance ## role -Provides infrastructure for managing tool instance lifecycle events, health monitoring, and asynchronous communication within the API service. +Coordinates tool instance lifecycle events, health monitoring, and notifications across the API service. ## parent index: apps/api/src/services/.pi-map.index.md map: apps/api/src/services/.pi-map.md diff --git a/apps/api/src/services/instance/.pi-map.md b/apps/api/src/services/instance/.pi-map.md index c089b9f..ccc977e 100644 --- a/apps/api/src/services/instance/.pi-map.md +++ b/apps/api/src/services/instance/.pi-map.md @@ -4,14 +4,14 @@ dir: apps/api/src/services/instance index: apps/api/src/services/instance/.pi-map.index.md ## role -Provides infrastructure for managing tool instance lifecycle events, health monitoring, and asynchronous communication within the API service. +Coordinates tool instance lifecycle events, health monitoring, and notifications across the API service. ## files - __init__.py | Exports public API for instance lifecycle services module | dep: src.services.instance.event_bus, src.services.instance.health_monitor, src.services.instance.lifecycle_hooks - event_bus.py | Implements a singleton in-memory typed event bus with publish/subscribe pattern for instance lifecycle and health events, supporting both sync and async callbacks with exception isolation. | exp: class:InstanceEventBus, method:__init__(self) → None, method:__new__(cls) → "InstanceEventBus", call:super().__new__, method:_reset_for_testing(self) → None, call:self._subscribers.clear, method:subscribe(self, event_type: str, callback: EventCallback) → Callable[[], None], call:str, call:uuid.uuid4, call:self._subscribers[event_type].append, call:self.unsubscribe, method:unsubscribe(self, event_type: str, callback_id: str) → None, method:unsubscribe_all(self, event_type: str) → None, call:self._subscribers.pop, method:publish(self, event_type: str, payload: InstanceEventPayload) → None, call:callbacks.extend, call:self._subscribers.get, call:inspect.iscoroutinefunction, call:callback, call:payload.get, call:logger.exception | dep: asyncio, inspect, logging, uuid, collections.abc, typing -- health_monitor.py | Background health monitor that periodically polls Docker container and tunnel health for tool instances, publishing state change events and notifications. | exp: class:HealthSnapshot, class:HealthMonitor, method:__init__(self, event_bus: InstanceEventBus) → None, method:start(self) → None, call:self._task.done, call:asyncio.get_running_loop, call:loop.create_task, call:self._poll_loop, method:stop(self) → None, call:self._task.done, call:self._task.cancel, call:self._last_known_state.clear, method:_poll_loop(self) → None, call:asyncio.sleep, call:self._run_check_cycle, call:logger.exception, method:_run_check_cycle(self) → None, call:SessionLocal, call:session.execute, call:select(ToolInstance).where, call:ToolInstance.status.in_, call:result.scalars().all, call:self._check_instance, method:_check_instance(self, session: AsyncSession, instance: ToolInstance) → None, call:logger.debug, call:get_container_status, call:logger.exception, call:str, call:get_correlation_id, call:check_tunnel_health, call:tunnel_result.get, call:HealthSnapshot, call:self._last_known_state.get, call:self._derive_status, call:self._snapshots_equal, call:self._handle_state_change, method:_derive_status(self, snapshot: HealthSnapshot, previous: HealthSnapshot | None, current_status: str | None) → str, method:_snapshots_equal(self, a: HealthSnapshot, b: HealthSnapshot) → bool, method:_handle_state_change(self, session: AsyncSession, instance: ToolInstance, previous: HealthSnapshot | None, snapshot: HealthSnapshot, new_status: str) → None, call:HealthCheck, call:session.add, call:session.commit, call:get_correlation_id, call:str, call:datetime.now(timezone.utc).isoformat, call:self._event_bus.publish, call:notification_service.create_notification, call:logger.exception | dep: asyncio, logging, uuid, dataclasses, datetime, sqlalchemy, sqlalchemy.ext.asyncio, src.database, src.models, src.services.shared.correlation, src.services.docker, src.services.shared.tunnel, src.services.instance.event_bus, src.services.shared.notification_service +- health_monitor.py | Background health monitor that polls Docker container and tunnel health for tool instances, publishes state change events, and creates notifications for errors/unhealthy states. | exp: class:HealthSnapshot, class:HealthMonitor, method:__init__(self, event_bus: InstanceEventBus) → None, method:start(self) → None, call:self._task.done, call:asyncio.get_running_loop, call:loop.create_task, call:self._poll_loop, method:stop(self) → None, call:self._task.done, call:self._task.cancel, call:self._last_known_state.clear, method:_poll_loop(self) → None, call:asyncio.sleep, call:self._run_check_cycle, call:logger.exception, method:_run_check_cycle(self) → None, call:SessionLocal, call:session.execute, call:select(ToolInstance).where, call:ToolInstance.status.in_, call:result.scalars().all, call:self._check_instance, method:_check_instance(self, session: AsyncSession, instance: ToolInstance) → None, call:logger.debug, call:get_container_status, call:logger.exception, call:str, call:get_correlation_id, call:check_tunnel_health, call:tunnel_result.get, call:HealthSnapshot, call:self._last_known_state.get, call:self._derive_status, call:self._snapshots_equal, call:self._handle_state_change, method:_derive_status(self, snapshot: HealthSnapshot, previous: HealthSnapshot | None, current_status: str | None) → str, method:_snapshots_equal(self, a: HealthSnapshot, b: HealthSnapshot) → bool, method:_handle_state_change(self, session: AsyncSession, instance: ToolInstance, previous: HealthSnapshot | None, snapshot: HealthSnapshot, new_status: str) → None, call:HealthCheck, call:session.add, call:session.commit, call:get_correlation_id, call:str, call:datetime.now(timezone.utc).isoformat, call:self._event_bus.publish, call:notification_service.create_notification, call:logger.exception | dep: asyncio, logging, uuid, dataclasses, datetime, sqlalchemy, sqlalchemy.ext.asyncio, src.database, src.models, src.services.shared.correlation, src.services.docker, src.services.shared.tunnel, src.services.instance.event_bus, src.services.shared.notification_service - lifecycle_hooks.py | Provides helpers to publish tool instance lifecycle events, persist audit records, and conditionally send user notifications. | exp: func:_derive_title(event_type: str) → str, call:mapping.get, call:event_type.replace("instance.", "").replace("_", " ").title, func:_should_notify(event_type: str, status: str | None) → bool, func:_build_payload(event_type: str, instance: ToolInstance, status, message, metadata) → InstanceEventPayload, call:str, call:datetime.now(timezone.utc).isoformat, call:get_correlation_id, func:_write_audit_row(session: AsyncSession, instance: ToolInstance, event_type: str, created_by, status, message, metadata) → InstanceEvent, call:InstanceEvent, call:event_type.replace, call:session.add, call:session.commit, func:publish_lifecycle_event(event_bus: InstanceEventBus, session: AsyncSession, instance: ToolInstance, event_type: str, created_by, status, message, metadata) → None, call:_build_payload, call:_write_audit_row, call:event_bus.publish, call:_should_notify, call:_derive_title, call:notification_service.create_notification, call:logger.exception, call:payload.get | dep: logging, uuid, datetime, sqlalchemy.ext.asyncio, src.models, src.services.shared.correlation, src.services.instance.event_bus, src.services.shared.notification_service ## arch -Event-driven architecture using a singleton in-memory pub/sub event bus with typed messages, background polling workers, and lifecycle hooks that bridge domain events to persistence and notifications with exception isolation between sync/async handlers. +Observer pattern via typed singleton event bus with async/sync subscribers, background polling loops, and side-effect hooks for persistence and notifications. ## tags call:self., instance, src, services, event, health, call:logger.exception, check ## symbols diff --git a/apps/api/src/services/instance/health_monitor.py b/apps/api/src/services/instance/health_monitor.py index 95b39ec..fd9290a 100644 --- a/apps/api/src/services/instance/health_monitor.py +++ b/apps/api/src/services/instance/health_monitor.py @@ -203,6 +203,11 @@ class HealthMonitor: # Transient states (created, restarting) — preserve current status # instead of treating them as an error. The next poll will resolve. if snapshot.container_status in ("created", "restarting"): + # A container that was already running and is now restarting has + # crashed (e.g. entrypoint failure / restart loop). Mark it failed + # so the dashboard does not keep showing it as running. + if current_status == "running": + return "error" return current_status or "starting" # Unknown/unexpected state (paused, etc.) diff --git a/apps/api/src/services/tool/.pi-map.index.md b/apps/api/src/services/tool/.pi-map.index.md index c7bdade..f2fa958 100644 --- a/apps/api/src/services/tool/.pi-map.index.md +++ b/apps/api/src/services/tool/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/services/tool ## role -Provides containerized execution environment for tools by managing Docker instances, git repositories, and compose orchestration. +Provides Docker container lifecycle management for tool instances with git repository mounting, configuration resolution, and SSH tunnel connectivity. ## parent index: apps/api/src/services/.pi-map.index.md map: apps/api/src/services/.pi-map.md diff --git a/apps/api/src/services/tool/.pi-map.md b/apps/api/src/services/tool/.pi-map.md index 0da1ac3..fa3cca0 100644 --- a/apps/api/src/services/tool/.pi-map.md +++ b/apps/api/src/services/tool/.pi-map.md @@ -4,11 +4,11 @@ dir: apps/api/src/services/tool index: apps/api/src/services/tool/.pi-map.index.md ## role -Provides containerized execution environment for tools by managing Docker instances, git repositories, and compose orchestration. +Provides Docker container lifecycle management for tool instances with git repository mounting, configuration resolution, and SSH tunnel connectivity. ## files -- instance_service.py | Manages Docker-based tool instances including git repository mounting, config profile resolution, container lifecycle operations, and compose file orchestration. | exp: func:_chown_path(path: str, uid: int, gid: int) → None, call:os.path.isdir, call:os.walk, call:os.path.join, call:contextlib.suppress, call:os.chown, call:logger.warning, func:_chown_staged_mounts(extra_volumes: list[dict], instance_dir: str, uid: int, gid: int) → None, call:vol.get, call:source.startswith, call:_chown_path, func:resolve_git_mounts(session: AsyncSession, resolved: ResolvedProfile, instance_dir, working_directory, home_dir) → list[dict], call:tasks.append, call:resolve_single_git_mount, call:asyncio.gather, call:isinstance, call:logger.warning, call:volume_mounts.extend, func:normalize_git_mount(entry: dict) → dict, call:dict, call:entry.get, call:entry.pop, func:clone_git_repo(remote_url: str, branch: str | None, clone_parent: str) → str, call:hashlib.md5(remote_url.encode()).hexdigest, call:remote_url.encode, call:remote_url.split("/")[-1].replace, call:os.path.join, call:os.path.exists, call:os.makedirs, call:clone_repository, call:logger.debug, call:logger.warning, call:pull_repository_updates, call:checkout_branch, func:resolve_git_mount_mappings(repo_path: str, mappings: list[dict], working_directory: str | None, home_dir) → list[dict], call:mapping.get, call:logger.warning, call:expand_container_path, call:target_path.startswith, call:os.path.join, call:expand_glob_source, call:os.path.exists, call:len, call:os.path.relpath, call:volume_mounts.append, call:logger.debug, func:resolve_single_git_mount(session: AsyncSession, git_mount: dict, instance_dir, working_directory, home_dir) → list[dict], call:normalize_git_mount, call:git_mount.get, call:logger.warning, call:asyncio.to_thread, call:resolve_git_mount_mappings, func:checkout_branch(repo_path: str, branch: str) → bool, call:subprocess.run, call:logger.warning, call:result.stderr.strip, func:pull_repository_updates(repo_path: str, remote_url: str) → None, call:subprocess.run, raise:RuntimeError, func:expand_glob_source(source_path: str, repo_path: str) → list[str], call:any, call:os.path.exists, call:glob_module.glob, call:len, call:os.path.abspath, call:abs_path.startswith, call:results.append, call:logger.warning, func:validate_config_profile(session: AsyncSession, profile_id: str | None, user_id: uuid.UUID, project_id: uuid.UUID, tool_type_id: uuid.UUID) → uuid.UUID | None, call:uuid.UUID, call:session.get, raise:HTTPException, func:sanitize_compose_file(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:isinstance, call:port_mapping.split, call:len, call:valid_ports.append, call:compose_file.write_text, call:yaml.dump, func:modify_compose_file(compose_path: str, port_override, start_command, working_directory, extra_volumes, home_dir) → None, call:Path, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:enumerate, call:isinstance, call:port_mapping.split, call:expand_container_path, call:vol.get, call:service_config["volumes"].append, call:service_config.get, call:sort_volumes_by_specificity, call:compose_file.write_text, call:yaml.dump, func:ensure_container_name_in_compose(compose_path: str, container_name: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:container_name.lower, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:ensure_web_bind_address(compose_path: str, tool_type_name: str, default_port: int) → None, call:KNOWN_BIND_FIXES.get, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].values, call:service_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.warning, call:logger.info, func:ensure_backend_network_in_compose(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:get_backend_network_name, call:compose_data["services"].values, call:svc_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:prepare_manifest_instance(session: AsyncSession, instance: ToolInstance, instance_dir: str, repo_path: str, env_vars: dict, extra_volumes: list, working_directory: str | None) → tuple[str, str, dict, str], call:session.get, call:dict, call:resolve_base, call:deep_merge, call:logger.warning, call:merge_with_config, call:manifest.pop, call:env_vars.update, call:extra_volumes.extend, call:compute_image_tag, call:subprocess.run, call:check.stdout.strip, call:compile_dockerfile, call:compile_entrypoint, call:logger.debug, call:len, call:asyncio.to_thread, call:logger.info, call:resolve_profile, call:gm.get, call:os.path.basename, call:os.path.normpath, call:instance.name.lower, call:compile_compose, call:manifest.get, call:datetime.now, call:get_manifest_home_dir, raise:RuntimeError, func:create_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, data: "CreateInstanceRequest") → ToolInstance, call:session.get, call:uuid.UUID, call:validate_config_profile, call:uuid.uuid4, call:select(ToolInstance) .where(ToolInstance.workspace_id == workspace_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:select(ToolInstance) .where(ToolInstance.repository_id == repo_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:session.execute, call:len, call:result.scalars().all, call:ensure_instance_directory, call:os.path.join, call:find_free_port, call:f"headquarter/{instance_name}:latest".lower, call:asyncio.to_thread, call:logger.error, call:logger.info, call:os.path.basename, call:os.path.normpath, call:instance_name.lower, call:write_compose_file, call:dict, call:resolve_base, call:deep_merge, call:compute_image_tag, call:compile_compose, call:str, call:render_compose_template, call:ToolInstance, call:session.add, call:session.commit, call:session.refresh, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:start_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, data: "StartInstanceRequest | None") → dict, call:session.get, call:validate_config_profile, call:session.commit, call:os.path.exists, call:logger.info, call:dict, call:resolve_base, call:deep_merge, call:get_manifest_home_dir, call:manifest.get, call:user_cfg.get, call:logger.debug, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:env_vars.update, call:config_files.update, call:extra_volumes.extend, call:resolve_git_mounts, call:profile_hints.get, call:len, call:logger.error, call:write_env_file, call:write_config_files, call:uuid.UUID, call:ssh_keys_to_mount.append, call:logger.warning, call:os.path.join, call:os.makedirs, call:_sanitize_filename, call:key_filenames.append, call:prepare_ssh_key_files, call:write_ssh_config, call:extra_volumes.append, call:_chown_staged_mounts, call:prepare_manifest_instance, call:write_compose_file, call:logger.exception, call:modify_compose_file, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:instance.name.lower, call:get_container_id, call:publish_lifecycle_event, call:wait_for_container_running, call:get_container_logs, call:resolved_manifest.get, call:apply_mount_permissions, call:home_dir.startswith, call:apply_ssh_permissions, call:probe_config.get, call:execute_probe, call:datetime.now().isoformat, call:"\n".join, call:start_tunnel, call:str, call:traceback.format_exc, raise:ValueError, raise:RuntimeError, func:restart_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.debug, call:logger.warning, call:os.path.exists, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:write_env_file, call:logger.error, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:datetime.now, call:session.commit, call:start_tunnel, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:delete_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, force) → None, call:session.get, call:os.path.dirname, call:os.path.join, call:os.path.exists, call:check_dirty_state, call:stop_tunnel, call:logger.debug, call:logger.warning, call:execute_compose_command, call:shutil.rmtree, call:publish_lifecycle_event, call:session.delete, call:session.commit, raise:ValueError, raise:RuntimeError, func:recreate_instance_tunnel(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:instance.name.lower, call:logger.info, call:get_container_id, call:logger.error, call:get_backend_network_name, call:is_container_on_network, call:connect_container_to_network, call:get_container_ip_on_network, call:logger.warning, call:recreate_tunnel, call:check_tunnel_health, call:health.get, call:subprocess.run, call:probe.stdout.strip, call:session.commit, call:logger.exception, raise:ValueError, raise:RuntimeError, func:stop_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.warning, call:os.path.exists, call:execute_compose_command, call:datetime.now, call:session.commit, call:publish_lifecycle_event, raise:ValueError, func:rename_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, display_name: str) → ToolInstance, call:session.get, call:display_name.strip, call:session.commit, call:session.refresh, raise:ValueError | dep: asyncio, contextlib, logging, os, subprocess, uuid, datetime, fastapi, sqlalchemy, sqlalchemy.ext.asyncio, src.models, src.schemas.tool, src.services.git.clone, src.services.config.config_profile_resolver, src.services.docker, src.services.shared.tunnel, src.services.build.docker_build, src.services.build.manifest_compiler, src.services.shared.permission_fixer, src.services.shared.readiness_probe, src.services.shared.ssh_keys, src.services.instance.event_bus, src.services.instance.lifecycle_hooks, hashlib, yaml, pathlib, traceback, shutil +- instance_service.py | Manages Docker-based tool instance lifecycle including git repository mounting, config profile resolution, container orchestration, and SSH tunnel management. | exp: func:_chown_path(path: str, uid: int, gid: int) → None, call:os.path.isdir, call:os.walk, call:os.path.join, call:contextlib.suppress, call:os.chown, call:logger.warning, func:_chown_staged_mounts(extra_volumes: list[dict], instance_dir: str, uid: int, gid: int) → None, call:vol.get, call:source.startswith, call:_chown_path, func:resolve_git_mounts(session: AsyncSession, resolved: ResolvedProfile, instance_dir, working_directory, home_dir) → list[dict], call:tasks.append, call:resolve_single_git_mount, call:asyncio.gather, call:isinstance, call:logger.warning, call:volume_mounts.extend, func:normalize_git_mount(entry: dict) → dict, call:dict, call:entry.get, call:entry.pop, func:clone_git_repo(remote_url: str, branch: str | None, clone_parent: str) → str, call:hashlib.md5(remote_url.encode()).hexdigest, call:remote_url.encode, call:remote_url.split("/")[-1].replace, call:os.path.join, call:os.path.exists, call:os.makedirs, call:clone_repository, call:logger.debug, call:logger.warning, call:pull_repository_updates, call:checkout_branch, func:resolve_git_mount_mappings(repo_path: str, mappings: list[dict], working_directory: str | None, home_dir) → list[dict], call:mapping.get, call:logger.warning, call:expand_container_path, call:target_path.startswith, call:os.path.join, call:expand_glob_source, call:os.path.exists, call:len, call:os.path.relpath, call:volume_mounts.append, call:logger.debug, func:resolve_single_git_mount(session: AsyncSession, git_mount: dict, instance_dir, working_directory, home_dir) → list[dict], call:normalize_git_mount, call:git_mount.get, call:logger.warning, call:asyncio.to_thread, call:resolve_git_mount_mappings, func:checkout_branch(repo_path: str, branch: str) → bool, call:subprocess.run, call:logger.warning, call:result.stderr.strip, func:pull_repository_updates(repo_path: str, remote_url: str) → None, call:subprocess.run, raise:RuntimeError, func:expand_glob_source(source_path: str, repo_path: str) → list[str], call:any, call:os.path.exists, call:glob_module.glob, call:len, call:os.path.abspath, call:abs_path.startswith, call:results.append, call:logger.warning, func:validate_config_profile(session: AsyncSession, profile_id: str | None, user_id: uuid.UUID, project_id: uuid.UUID, tool_type_id: uuid.UUID) → uuid.UUID | None, call:uuid.UUID, call:session.get, raise:HTTPException, func:sanitize_compose_file(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:isinstance, call:port_mapping.split, call:len, call:valid_ports.append, call:compose_file.write_text, call:yaml.dump, func:modify_compose_file(compose_path: str, port_override, start_command, working_directory, extra_volumes, home_dir) → None, call:Path, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:enumerate, call:isinstance, call:port_mapping.split, call:expand_container_path, call:vol.get, call:service_config["volumes"].append, call:service_config.get, call:sort_volumes_by_specificity, call:compose_file.write_text, call:yaml.dump, func:ensure_container_name_in_compose(compose_path: str, container_name: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:container_name.lower, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:ensure_web_bind_address(compose_path: str, tool_type_name: str, default_port: int) → None, call:KNOWN_BIND_FIXES.get, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].values, call:service_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.warning, call:logger.info, func:ensure_backend_network_in_compose(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:get_backend_network_name, call:compose_data["services"].values, call:svc_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:prepare_manifest_instance(session: AsyncSession, instance: ToolInstance, instance_dir: str, repo_path: str, env_vars: dict, extra_volumes: list, working_directory: str | None) → tuple[str, str, dict, str], call:session.get, call:dict, call:resolve_base, call:deep_merge, call:logger.warning, call:merge_with_config, call:manifest.pop, call:env_vars.update, call:extra_volumes.extend, call:compute_image_tag, call:subprocess.run, call:check.stdout.strip, call:compile_dockerfile, call:compile_entrypoint, call:logger.debug, call:len, call:asyncio.to_thread, call:logger.info, call:resolve_profile, call:gm.get, call:os.path.basename, call:os.path.normpath, call:instance.name.lower, call:compile_compose, call:manifest.get, call:datetime.now, call:get_manifest_home_dir, raise:RuntimeError, func:create_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, data: "CreateInstanceRequest") → ToolInstance, call:session.get, call:uuid.UUID, call:validate_config_profile, call:uuid.uuid4, call:select(ToolInstance) .where(ToolInstance.workspace_id == workspace_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:select(ToolInstance) .where(ToolInstance.repository_id == repo_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:session.execute, call:len, call:result.scalars().all, call:ensure_instance_directory, call:os.path.join, call:find_free_port, call:f"headquarter/{instance_name}:latest".lower, call:asyncio.to_thread, call:logger.error, call:logger.info, call:os.path.basename, call:os.path.normpath, call:instance_name.lower, call:write_compose_file, call:dict, call:resolve_base, call:deep_merge, call:compute_image_tag, call:compile_compose, call:str, call:render_compose_template, call:ToolInstance, call:session.add, call:session.commit, call:session.refresh, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:start_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, data: "StartInstanceRequest | None") → dict, call:session.get, call:validate_config_profile, call:session.commit, call:os.path.exists, call:logger.info, call:dict, call:resolve_base, call:deep_merge, call:get_manifest_home_dir, call:manifest.get, call:user_cfg.get, call:logger.debug, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:env_vars.update, call:config_files.update, call:extra_volumes.extend, call:resolve_git_mounts, call:profile_hints.get, call:len, call:logger.error, call:write_env_file, call:write_config_files, call:uuid.UUID, call:ssh_keys_to_mount.append, call:logger.warning, call:os.path.join, call:os.makedirs, call:_sanitize_filename, call:key_filenames.append, call:prepare_ssh_key_files, call:write_ssh_config, call:extra_volumes.append, call:_chown_staged_mounts, call:prepare_manifest_instance, call:write_compose_file, call:logger.exception, call:modify_compose_file, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:instance.name.lower, call:get_container_id, call:publish_lifecycle_event, call:wait_for_container_running, call:get_container_logs, call:resolved_manifest.get, call:apply_mount_permissions, call:home_dir.startswith, call:apply_ssh_permissions, call:probe_config.get, call:execute_probe, call:datetime.now().isoformat, call:"\n".join, call:get_container_status, call:start_tunnel, call:str, call:traceback.format_exc, raise:ValueError, raise:RuntimeError, func:restart_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.debug, call:logger.warning, call:os.path.exists, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:write_env_file, call:logger.error, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:datetime.now, call:session.commit, call:start_tunnel, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:delete_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, force) → None, call:session.get, call:os.path.dirname, call:os.path.join, call:os.path.exists, call:check_dirty_state, call:stop_tunnel, call:logger.debug, call:logger.warning, call:execute_compose_command, call:shutil.rmtree, call:publish_lifecycle_event, call:session.delete, call:session.commit, raise:ValueError, raise:RuntimeError, func:recreate_instance_tunnel(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:instance.name.lower, call:logger.info, call:get_container_id, call:logger.error, call:get_backend_network_name, call:is_container_on_network, call:connect_container_to_network, call:get_container_ip_on_network, call:logger.warning, call:recreate_tunnel, call:check_tunnel_health, call:health.get, call:subprocess.run, call:probe.stdout.strip, call:session.commit, call:logger.exception, raise:ValueError, raise:RuntimeError, func:stop_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.warning, call:os.path.exists, call:execute_compose_command, call:datetime.now, call:session.commit, call:publish_lifecycle_event, raise:ValueError, func:rename_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, display_name: str) → ToolInstance, call:session.get, call:display_name.strip, call:session.commit, call:session.refresh, raise:ValueError | dep: asyncio, contextlib, logging, os, subprocess, uuid, datetime, fastapi, sqlalchemy, sqlalchemy.ext.asyncio, src.models, src.schemas.tool, src.services.git.clone, src.services.config.config_profile_resolver, src.services.docker, src.services.shared.tunnel, src.services.build.docker_build, src.services.build.manifest_compiler, src.services.shared.permission_fixer, src.services.shared.readiness_probe, src.services.shared.ssh_keys, src.services.instance.event_bus, src.services.instance.lifecycle_hooks, hashlib, yaml, pathlib, traceback, shutil, glob ## arch -Service-oriented architecture with Docker container lifecycle management, git repository mounting, config profile resolution, and compose file orchestration patterns. +Service-layer orchestration pattern combining Docker SDK container management, git repository volume mounting, hierarchical config profile resolution, and SSH tunnel proxying for external tool access. ## tags compose, call:compose, call:logger.warning, error, instance, container, git, text ## symbols diff --git a/apps/api/src/services/tool/instance_service.py b/apps/api/src/services/tool/instance_service.py index ce0138f..a3d1172 100644 --- a/apps/api/src/services/tool/instance_service.py +++ b/apps/api/src/services/tool/instance_service.py @@ -13,7 +13,14 @@ from fastapi import HTTPException, status from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from src.models import ConfigProfile, GitRepository, Project, SSHKey, ToolInstance, ToolType +from src.models import ( + ConfigProfile, + GitRepository, + Project, + SSHKey, + ToolInstance, + ToolType, +) from src.schemas.tool import CreateInstanceRequest, StartInstanceRequest from src.services.git.clone import check_dirty_state, clone_repository from src.services.config.config_profile_resolver import ( @@ -32,6 +39,7 @@ from src.services.docker import ( get_container_id, get_container_ip_on_network, get_container_logs, + get_container_status, is_container_on_network, render_compose_template, sort_volumes_by_specificity, @@ -774,7 +782,6 @@ def ensure_backend_network_in_compose(compose_path: str) -> None: logger.info("Injected backend network '%s' into compose file", network_name) - async def prepare_manifest_instance( session: AsyncSession, instance: ToolInstance, @@ -918,8 +925,6 @@ async def prepare_manifest_instance( return image_tag, compose_content, manifest, home_dir - - async def create_tool_instance( session: AsyncSession, user_id: uuid.UUID, @@ -1047,15 +1052,25 @@ async def create_tool_instance( repo_name = os.path.basename(os.path.normpath(repo_path)) workspace_target = f"{home_dir}/{repo_name}" - compose_content = f"""version: "3.8"\nservices:\n app:\n image: {image_tag}\n container_name: {instance_name.lower()}\n stdin_open: true\n tty: true\n{ports_section} environment:\n - HOME={home_dir}\n volumes:\n - {repo_path}:{workspace_target}\n working_dir: {workspace_target}\n restart: unless-stopped\n""" + compose_content = f"""version: "3.8"\nservices: + app: + image: {image_tag} + container_name: {instance_name.lower()} + stdin_open: true + tty: true +{ports_section} environment: + - HOME={home_dir} + volumes: + - {repo_path}:{workspace_target} + working_dir: {workspace_target} + restart: "no" +""" write_compose_file(instance_dir, compose_content) elif tool_type.definition_type == "manifest": from src.models import ToolDefinitionManifest - manifest_def = await session.get( - ToolDefinitionManifest, tool_type.manifest_id - ) + manifest_def = await session.get(ToolDefinitionManifest, tool_type.manifest_id) if not manifest_def: raise RuntimeError("Manifest definition not found for this tool type") @@ -1065,9 +1080,7 @@ async def create_tool_instance( ToolDefinitionManifest, manifest_def.base_definition_id ) if base_def: - manifest = resolve_base( - deep_merge(dict(base_def.manifest), manifest) - ) + manifest = resolve_base(deep_merge(dict(base_def.manifest), manifest)) image_tag = compute_image_tag(tool_type.name, manifest) @@ -1105,9 +1118,7 @@ async def create_tool_instance( "WORKSPACE_NAME": os.path.basename(os.path.normpath(repo_path)), "HOME_DIRECTORY": tool_type.home_directory or "/home/user", } - compose_content = render_compose_template( - tool_type.compose_template, variables - ) + compose_content = render_compose_template(tool_type.compose_template, variables) write_compose_file(instance_dir, compose_content) @@ -1674,6 +1685,41 @@ async def start_tool_instance( logger.info("Readiness probe succeeded for instance %s", instance.id) + # Final stability check: the container must still be running after all + # post-start setup. If it has already exited/restarted, mark it failed now + # instead of optimistically reporting "running". + if instance.container_id: + final_check = get_container_status(instance.container_id) + if final_check["status"] != "running": + error_msg = ( + f"Container stopped during startup: status={final_check['status']}" + ) + if final_check["exit_code"] is not None: + error_msg += f", exit_code={final_check['exit_code']}" + logs = get_container_logs(instance.container_id, tail=50) + instance.status = "error" + await session.commit() + await publish_lifecycle_event( + event_bus=_event_bus, + session=session, + instance=instance, + event_type="instance.error", + created_by=user_id, + status="error", + message=error_msg, + metadata={ + "exit_code": final_check["exit_code"], + "error_type": "container", + }, + ) + logger.error( + "Instance %s container stopped during startup: %s\nLogs:\n%s", + instance.id, + error_msg, + logs, + ) + return {"status": "error", "error": error_msg, "logs": logs} + instance.status = "running" await session.commit() await publish_lifecycle_event( @@ -2102,7 +2148,9 @@ async def stop_tool_instance( try: stop_tunnel(instance.name) except Exception as exc: - logger.warning("Failed to stop tunnel for instance %s: %s", instance.id, exc) + logger.warning( + "Failed to stop tunnel for instance %s: %s", instance.id, exc + ) if instance.compose_path and os.path.exists(instance.compose_path): execute_compose_command(instance.compose_path, "stop") @@ -2144,4 +2192,3 @@ async def rename_tool_instance( await session.commit() await session.refresh(instance) return instance - diff --git a/openspec/.pi-map.index.md b/openspec/.pi-map.index.md index 994f021..8ffb692 100644 --- a/openspec/.pi-map.index.md +++ b/openspec/.pi-map.index.md @@ -2,7 +2,7 @@ dir: openspec ## role -Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a project repository. +Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a Docker-based coding agent management platform. ## parent index: ./.pi-map.index.md map: ./.pi-map.md diff --git a/openspec/changes/.pi-map.index.md b/openspec/changes/.pi-map.index.md index b8b2e9a..2e0b58d 100644 --- a/openspec/changes/.pi-map.index.md +++ b/openspec/changes/.pi-map.index.md @@ -2,7 +2,7 @@ dir: openspec/changes ## role -Manages and tracks specification changes, diffs, and versioning history for OpenAPI documents. +Provides change-tracking and diffing utilities for OpenAPI specification evolution and version comparison. ## parent index: openspec/.pi-map.index.md map: openspec/.pi-map.md @@ -10,6 +10,9 @@ map: openspec/.pi-map.md - openspec/changes/archive index: openspec/changes/archive/.pi-map.index.md map: openspec/changes/archive/.pi-map.md +- openspec/changes/fix-container-status-false-positive + index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md + map: openspec/changes/fix-container-status-false-positive/.pi-map.md - openspec/changes/fix-pi-container-mount-permissions index: openspec/changes/fix-pi-container-mount-permissions/.pi-map.index.md map: openspec/changes/fix-pi-container-mount-permissions/.pi-map.md diff --git a/openspec/changes/fix-container-status-false-positive/.pi-map.index.md b/openspec/changes/fix-container-status-false-positive/.pi-map.index.md new file mode 100644 index 0000000..884168f --- /dev/null +++ b/openspec/changes/fix-container-status-false-positive/.pi-map.index.md @@ -0,0 +1,20 @@ +# openspec/changes/fix-container-status-false-positive (index) +dir: openspec/changes/fix-container-status-false-positive + +## role +Documents a bug fix for correcting container status false positives where failed containers incorrectly appeared as running in a dashboard/monitoring system. +## parent +index: openspec/changes/.pi-map.index.md +map: openspec/changes/.pi-map.md +## children +- +## files +- change.md +- tasks.md +## links +index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md +map: openspec/changes/fix-container-status-false-positive/.pi-map.md +## workflows +- +## dirty +- diff --git a/openspec/changes/fix-container-status-false-positive/.pi-map.md b/openspec/changes/fix-container-status-false-positive/.pi-map.md new file mode 100644 index 0000000..9c740ae --- /dev/null +++ b/openspec/changes/fix-container-status-false-positive/.pi-map.md @@ -0,0 +1,20 @@ +# openspec/changes/fix-container-status-false-positive +dir: openspec/changes/fix-container-status-false-positive + +index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md + +## role +Documents a bug fix for correcting container status false positives where failed containers incorrectly appeared as running in a dashboard/monitoring system. +## files +- change.md | Documents a bug fix for failed containers incorrectly displaying as running on a dashboard, detailing root cause analysis and the fix implementation across multiple files. +- tasks.md | Track progress of fixing a false-positive running status bug in a container/system management system +## arch +Issue-tracking documentation pattern using markdown files (change.md for technical specification, tasks.md for progress tracking) to manage a bug fix across multiple system components. +## tags +bug, fix, running, change, system, tasks, documents, failed +## symbols +- +## workflows +- +## dirty +- diff --git a/openspec/changes/fix-container-status-false-positive/change.md b/openspec/changes/fix-container-status-false-positive/change.md new file mode 100644 index 0000000..ccfad75 --- /dev/null +++ b/openspec/changes/fix-container-status-false-positive/change.md @@ -0,0 +1,32 @@ +# Fix: failed containers shown as running on dashboard + +## Problem + +Containers that fail during startup (e.g. `ln: failed to create symbolic link '/workspace': Permission denied`) are still displayed as "Running" on the web dashboard and session list. + +## Root cause + +1. `start_tool_instance` in `apps/api/src/services/tool/instance_service.py` sets `instance.status = "running"` after post-start setup without verifying the container is still up. +2. Generated compose files use `restart: unless-stopped`, so Docker immediately restarts a crashed container, putting it into the `restarting` state. +3. The background `HealthMonitor` treats `restarting` as a transient state and preserves the current DB status (`running`). + +## Fix + +1. Add a final `get_container_status` check in `start_tool_instance` immediately before writing `status = "running"`. If the container has stopped/exited, mark it as `error` and return the logs. +2. Update `HealthMonitor._derive_status` so that when the DB status is `running` and Docker reports `restarting`, the instance is marked as `error`. +3. Disable auto-restart for tool instances by changing `restart: unless-stopped` to `restart: "no"` in: + - `apps/api/src/services/build/manifest_compiler.py` + - `apps/api/src/services/tool/instance_service.py` (legacy dockerfile path) + - `apps/api/src/seeds/builtin_tool_types.py` + +## Affected files + +- `apps/api/src/services/tool/instance_service.py` +- `apps/api/src/services/instance/health_monitor.py` +- `apps/api/src/services/build/manifest_compiler.py` +- `apps/api/src/seeds/builtin_tool_types.py` + +## Verification + +- `pytest apps/api/tests/unit` +- `ruff`, `mypy` on changed files diff --git a/openspec/changes/fix-container-status-false-positive/tasks.md b/openspec/changes/fix-container-status-false-positive/tasks.md new file mode 100644 index 0000000..1f2de2f --- /dev/null +++ b/openspec/changes/fix-container-status-false-positive/tasks.md @@ -0,0 +1,8 @@ +# Tasks: fix false-positive running status + +- [x] Investigate status flow and identify root cause +- [x] Add final container status check before writing "running" +- [x] Treat restarting as error when DB status is already running +- [x] Disable auto-restart in manifest compiler, legacy path, and seed data +- [ ] Run quality gates +- [ ] Commit and push