fix: prevent failed containers from showing as running on dashboard
- Add final get_container_status check in start_tool_instance before writing status=running; mark as error and return logs if container stopped - Treat restarting as error in HealthMonitor when DB status was already running, so crash loops are surfaced instead of preserved - Disable auto-restart (restart: unless-stopped -> restart: no) for tool instances in manifest compiler, legacy dockerfile path, and built-in seeds Quality gates: - pytest tests/unit: 210 passed - ruff: clean on changed files - mypy: clean on changed files
This commit is contained in:
+2
-1
@@ -16,7 +16,7 @@ dir: .
|
||||
Trust boundary: index routes, map orients, source decides.
|
||||
|
||||
## role
|
||||
Infrastructure and deployment configuration for a self-hosted project management platform with containerized services, SSO integration, and AI agent development workflows.
|
||||
A self-hosted web-based project management and git repository platform with OAuth2 SSO authentication, containerized for Docker deployment.
|
||||
## parent
|
||||
-
|
||||
## children
|
||||
@@ -69,6 +69,7 @@ Infrastructure and deployment configuration for a self-hosted project management
|
||||
- CHANGELOG.md
|
||||
- Makefile
|
||||
- README.md
|
||||
- context.md
|
||||
- docker-compose.traefik.yml
|
||||
- docker-compose.yml
|
||||
- progress.md
|
||||
|
||||
+4
-3
@@ -18,7 +18,7 @@ index: ./.pi-map.index.md
|
||||
Trust boundary: index routes, map orients, source decides.
|
||||
|
||||
## role
|
||||
Infrastructure and deployment configuration for a self-hosted project management platform with containerized services, SSO integration, and AI agent development workflows.
|
||||
A self-hosted web-based project management and git repository platform with OAuth2 SSO authentication, containerized for Docker deployment.
|
||||
## files
|
||||
- .env.example | Provides a template of environment variables for configuring a Headquarter application with PostgreSQL, Redis, Authentik SSO, and Docker/Traefik deployment
|
||||
- .gitignore | Specifies files and directories for Git to ignore across a multi-language project with Python, Node, and custom tooling | dep: Git
|
||||
@@ -26,14 +26,15 @@ Infrastructure and deployment configuration for a self-hosted project management
|
||||
- CHANGELOG.md | Documents version history and notable changes for a Git-based project management web application
|
||||
- Makefile | Provides standard development commands for containerized web application lifecycle management via Docker Compose | dep: docker compose, alembic, pytest, ruff, mypy, playwright, npm, postgres, redis
|
||||
- README.md | A self-hosted platform for managing projects, git repositories, and development tools with OAuth2 authentication. | dep: FastAPI, SQLAlchemy, Pydantic, Alembic, python-jose, React, TypeScript, Vite, React Router, Docker, PostgreSQL, Traefik, Authentik, Git
|
||||
- context.md | Documents a bug investigation where crashed Docker containers incorrectly display "running" status on the dashboard due to optimistic status writes, auto-restart policies, and health monitor logic that preserves running status for restarting containers. | dep: docker, docker-compose, FastAPI/SQLAlchemy, SSE event bus, React/TypeScript frontend
|
||||
- docker-compose.traefik.yml | Deploys a multi-service web application (frontend, API, PostgreSQL, Redis) behind an existing Traefik reverse proxy with TLS termination and environment-configurable domains. | dep: docker, traefik, postgres, redis, authentik, docker-compose
|
||||
- docker-compose.yml | Defines a multi-service Docker Compose stack with PostgreSQL, Redis, API backend, and web frontend services for a "headquarter" application | dep: Docker, PostgreSQL, Redis, Vite, asyncpg, nginx
|
||||
- progress.md | Tracks completed and remaining tasks for a backend-frontend code refactoring project organized in 7 phases
|
||||
- swap-pane | Empty file with no functionality
|
||||
## arch
|
||||
Docker Compose-based microservices architecture with PostgreSQL/Redis data layer, Traefik reverse proxy, OAuth2 SSO via Authentik, and OpenSpec-driven AI agent orchestration.
|
||||
Full-stack containerized architecture with separate frontend/API services, PostgreSQL/Redis data layer, Traefik reverse proxy with TLS, environment-driven configuration, and health monitoring with auto-restart policies.
|
||||
## tags
|
||||
docker, redis, git, application, postgresql, compose, traefik, project
|
||||
docker, redis, git, compose, application, postgresql, traefik, project
|
||||
## symbols
|
||||
-
|
||||
## workflows
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: apps
|
||||
|
||||
## role
|
||||
Contains the deployable application entry points and main executables for the project.
|
||||
Contains the deployable application entry points and executable configurations for the project.
|
||||
## parent
|
||||
index: ./.pi-map.index.md
|
||||
map: ./.pi-map.md
|
||||
|
||||
+2
-2
@@ -4,10 +4,10 @@ dir: apps
|
||||
index: apps/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Contains the deployable application entry points and main executables for the project.
|
||||
Contains the deployable application entry points and executable configurations for the project.
|
||||
## files
|
||||
## arch
|
||||
Typically follows a multi-app workspace pattern where each subdirectory represents a standalone application with its own main.ts, configuration, and dependency injection setup, often sharing common libraries from other packages.
|
||||
Modular application structure where each subdirectory represents a separate deployable unit or service, following a monorepo or multi-app pattern.
|
||||
## tags
|
||||
-
|
||||
## symbols
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ FastAPI backend API that manages projects, git repositories, and development too
|
||||
- uv.lock | Lock file for the uv Python package manager that pins exact dependency versions and their artifact hashes for reproducible installations | dep: uv, Python 3.11+, aiosqlite, alembic, annotated-doc, annotated-types, anyio, ast-serialize, asyncpg, and many other PyPI packages
|
||||
- wait-for-db.sh | Wait for a PostgreSQL database to become available before executing a command, with configurable retry logic. | dep: nc (netcat), sh (POSIX shell), sleep
|
||||
## arch
|
||||
Async Python/FastAPI service using multi-stage Docker builds, Alembic migrations, uv package management, and container orchestration with external tunneling and database dependency waiting patterns.
|
||||
Async Python/FastAPI service with multi-stage Docker containerization, Alembic database migrations, uv package management, and Docker socket access for container orchestration.
|
||||
## tags
|
||||
docker, alembic, python, database, fastapi, postgresql, asyncpg, uvicorn
|
||||
## symbols
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: apps/api/src
|
||||
|
||||
## role
|
||||
Core API application package for the "Headquarter API" providing configuration, database infrastructure, logging, and FastAPI application initialization.
|
||||
Core API server package for the "Headquarter" backend, handling configuration, database connectivity, logging, and FastAPI application initialization.
|
||||
## parent
|
||||
index: apps/api/.pi-map.index.md
|
||||
map: apps/api/.pi-map.md
|
||||
|
||||
@@ -4,7 +4,7 @@ dir: apps/api/src
|
||||
index: apps/api/src/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Core API application package for the "Headquarter API" providing configuration, database infrastructure, logging, and FastAPI application initialization.
|
||||
Core API server package for the "Headquarter" backend, handling configuration, database connectivity, logging, and FastAPI application initialization.
|
||||
## files
|
||||
- __init__.py | Marks the directory as a Python package for the Headquarter API.
|
||||
- config.py | Defines application configuration settings with environment-based overrides using Pydantic, including database URLs, service domains, OAuth/Authentik integration, JWT/session settings, and computed properties for environment-specific behavior. | exp: class:Settings, func:build_database_url(user: str, password: str, host: str, port: int, database: str) → str | dep: pydantic, pydantic_settings
|
||||
@@ -12,7 +12,7 @@ Core API application package for the "Headquarter API" providing configuration,
|
||||
- logging_config.py | Configures structured JSON logging with correlation ID injection, custom formatters, and HTTP request/exception middleware for a FastAPI application. | exp: class:CorrelationIdFilter, method:filter(self, record: logging.LogRecord) → bool, call:get_correlation_id, class:JSONFormatter, method:format(self, record: logging.LogRecord) → str, call:self.formatTime, call:record.getMessage, call:getattr, call:self.formatException, call:json.dumps, method:formatTime(self, record: logging.LogRecord, datefmt) → str, call:time.strftime, call:time.gmtime, class:RequestLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:time.time, call:logger.info, call:call_next, call:int, call:logger.error, call:type, call:traceback.format_exc, class:ExceptionLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:call_next, call:logger.critical, call:traceback.format_exc, func:configure_logging(level) → None, call:JSONFormatter, call:logging.StreamHandler, call:console_handler.setFormatter, call:console_handler.addFilter, call:CorrelationIdFilter, call:root_logger.setLevel, call:logging.getLogger("uvicorn").setLevel, call:logging.getLogger("uvicorn.access").setLevel, call:logging.getLogger("sqlalchemy.engine").setLevel, call:logger.info, call:logging.getLevelName | dep: json, logging, sys, time, traceback, collections.abc, fastapi, starlette.middleware.base, src.services.shared.correlation
|
||||
- main.py | Initializes and configures a FastAPI application for the "Headquarter API" with database setup, middleware, routing, and background services. | exp: func:_sanitize_validation_errors(errors), call:error.get, call:str, call:ctx.items, call:isinstance, call:type, call:sanitized.append, func:validation_exception_handler(request: Request, exc: RequestValidationError), call:exc.errors, call:logger.warning, call:_sanitize_validation_errors, call:JSONResponse, func:on_startup(), call:logger.info, call:init_database, call:logger.error, call:sys.exit, call:_health_monitor.start, call:seed_builtin_tool_types, func:on_shutdown(), call:logger.info, call:_health_monitor.stop | dep: logging, os, fastapi, fastapi.exceptions, fastapi.middleware.cors, fastapi.responses, fastapi.staticfiles, src.api.config, src.api.project, src.api.system, src.api.tool, src.api.user, src.api.workspace, src.config, src.models, src.database, src.logging_config, src.seeds.builtin_tool_types, src.services.instance, src.services.shared, sys, src.api.*
|
||||
## arch
|
||||
Layered architecture with Pydantic-based config management, async SQLAlchemy with Alembic migrations, structured JSON logging with correlation IDs, and FastAPI middleware/routing pattern.
|
||||
Layered async architecture using FastAPI with Pydantic settings management, SQLAlchemy async ORM with Alembic migrations, structured JSON logging with correlation ID tracking, and environment-driven configuration with OAuth/Authentik integration.
|
||||
## tags
|
||||
src, database, logging, call:logger.info, api, middleware, fastapi, filter
|
||||
## symbols
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: apps/api/src/seeds
|
||||
|
||||
## role
|
||||
Provides database seeding utilities for initializing built-in tool type configurations in the API application.
|
||||
Provides database seeding utilities for initializing and synchronizing built-in data records in the API application.
|
||||
## parent
|
||||
index: apps/api/src/.pi-map.index.md
|
||||
map: apps/api/src/.pi-map.md
|
||||
|
||||
@@ -4,12 +4,12 @@ dir: apps/api/src/seeds
|
||||
index: apps/api/src/seeds/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Provides database seeding utilities for initializing built-in tool type configurations in the API application.
|
||||
Provides database seeding utilities for initializing and synchronizing built-in data records in the API application.
|
||||
## files
|
||||
- __init__.py | Marks the directory as a Python package for database seeding utilities.
|
||||
- builtin_tool_types.py | Seeds built-in tool types (code-server, jupyter-notebook, opencode) into a database with upsert logic, creating or updating Docker Compose-based development environment templates. | exp: func:_table_exists(session, table_name: str) → bool, call:session.execute, call:text, call:result.scalar, func:seed_builtin_tool_types(), call:SessionLocal, call:_table_exists, call:logger.warning, call:session.scalar, call:select(ToolType).where, call:ToolType, call:tool_data.get, call:session.add, call:logger.info, call:session.commit | dep: logging, sqlalchemy, src.database, src.models
|
||||
- builtin_tool_types.py | Seeds predefined built-in tool types (code-server, jupyter-notebook, opencode) into a database with upsert logic, creating them if missing or updating existing ones to match code changes. | exp: func:_table_exists(session, table_name: str) → bool, call:session.execute, call:text, call:result.scalar, func:seed_builtin_tool_types(), call:SessionLocal, call:_table_exists, call:logger.warning, call:session.scalar, call:select(ToolType).where, call:ToolType, call:tool_data.get, call:session.add, call:logger.info, call:session.commit | dep: logging, sqlalchemy, src.database, src.models
|
||||
## arch
|
||||
Simple procedural seeding script using SQLAlchemy upsert operations to populate reference data for containerized development environment templates.
|
||||
Simple imperative seeding scripts with upsert pattern for idempotent data initialization, using direct database operations without abstraction layers.
|
||||
## tags
|
||||
tool, types, table, exists, builtin, call:tool, database, init
|
||||
## symbols
|
||||
|
||||
@@ -65,7 +65,7 @@ services:
|
||||
- {{REPO_PATH}}:/config/workspace
|
||||
ports:
|
||||
- "8443:8443"
|
||||
restart: unless-stopped""",
|
||||
restart: 'no'""",
|
||||
"default_port": 8443,
|
||||
"required_variables": ["REPO_PATH", "TOOL_NAME"],
|
||||
},
|
||||
@@ -87,7 +87,7 @@ services:
|
||||
- {{REPO_PATH}}:/home/jovyan/work
|
||||
ports:
|
||||
- "8888:8888"
|
||||
restart: unless-stopped""",
|
||||
restart: 'no'""",
|
||||
"required_variables": ["REPO_PATH", "TOOL_NAME"],
|
||||
},
|
||||
{
|
||||
@@ -122,7 +122,7 @@ services:
|
||||
exec tail -f /dev/null"
|
||||
stdin_open: true
|
||||
tty: true
|
||||
restart: unless-stopped""",
|
||||
restart: 'no'""",
|
||||
"required_variables": ["REPO_PATH", "TOOL_NAME"],
|
||||
},
|
||||
]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: apps/api/src/services
|
||||
|
||||
## role
|
||||
Provides business logic and service layer abstractions for the API application.
|
||||
Marker package for the services layer in the API application
|
||||
## parent
|
||||
index: apps/api/src/.pi-map.index.md
|
||||
map: apps/api/src/.pi-map.md
|
||||
|
||||
@@ -4,11 +4,11 @@ dir: apps/api/src/services
|
||||
index: apps/api/src/services/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Provides business logic and service layer abstractions for the API application.
|
||||
Marker package for the services layer in the API application
|
||||
## files
|
||||
- __init__.py | Empty file with no functionality
|
||||
## arch
|
||||
Minimal or placeholder package structure with no implemented services yet, following standard Python package conventions.
|
||||
Standard Python package structure using __init__.py for namespace declaration
|
||||
## tags
|
||||
init, empty, functionality
|
||||
## symbols
|
||||
|
||||
@@ -411,7 +411,7 @@ def compile_compose(manifest: dict, variables: dict[str, Any]) -> str:
|
||||
service: dict[str, Any] = {
|
||||
"image": variables["IMAGE_TAG"],
|
||||
"container_name": variables["INSTANCE_NAME"],
|
||||
"restart": "unless-stopped",
|
||||
"restart": "no",
|
||||
}
|
||||
|
||||
# Terminal-specific fields
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: apps/api/src/services/instance
|
||||
|
||||
## role
|
||||
Provides infrastructure for managing tool instance lifecycle events, health monitoring, and asynchronous communication within the API service.
|
||||
Coordinates tool instance lifecycle events, health monitoring, and notifications across the API service.
|
||||
## parent
|
||||
index: apps/api/src/services/.pi-map.index.md
|
||||
map: apps/api/src/services/.pi-map.md
|
||||
|
||||
@@ -4,14 +4,14 @@ dir: apps/api/src/services/instance
|
||||
index: apps/api/src/services/instance/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Provides infrastructure for managing tool instance lifecycle events, health monitoring, and asynchronous communication within the API service.
|
||||
Coordinates tool instance lifecycle events, health monitoring, and notifications across the API service.
|
||||
## files
|
||||
- __init__.py | Exports public API for instance lifecycle services module | dep: src.services.instance.event_bus, src.services.instance.health_monitor, src.services.instance.lifecycle_hooks
|
||||
- event_bus.py | Implements a singleton in-memory typed event bus with publish/subscribe pattern for instance lifecycle and health events, supporting both sync and async callbacks with exception isolation. | exp: class:InstanceEventBus, method:__init__(self) → None, method:__new__(cls) → "InstanceEventBus", call:super().__new__, method:_reset_for_testing(self) → None, call:self._subscribers.clear, method:subscribe(self, event_type: str, callback: EventCallback) → Callable[[], None], call:str, call:uuid.uuid4, call:self._subscribers[event_type].append, call:self.unsubscribe, method:unsubscribe(self, event_type: str, callback_id: str) → None, method:unsubscribe_all(self, event_type: str) → None, call:self._subscribers.pop, method:publish(self, event_type: str, payload: InstanceEventPayload) → None, call:callbacks.extend, call:self._subscribers.get, call:inspect.iscoroutinefunction, call:callback, call:payload.get, call:logger.exception | dep: asyncio, inspect, logging, uuid, collections.abc, typing
|
||||
- health_monitor.py | Background health monitor that periodically polls Docker container and tunnel health for tool instances, publishing state change events and notifications. | exp: class:HealthSnapshot, class:HealthMonitor, method:__init__(self, event_bus: InstanceEventBus) → None, method:start(self) → None, call:self._task.done, call:asyncio.get_running_loop, call:loop.create_task, call:self._poll_loop, method:stop(self) → None, call:self._task.done, call:self._task.cancel, call:self._last_known_state.clear, method:_poll_loop(self) → None, call:asyncio.sleep, call:self._run_check_cycle, call:logger.exception, method:_run_check_cycle(self) → None, call:SessionLocal, call:session.execute, call:select(ToolInstance).where, call:ToolInstance.status.in_, call:result.scalars().all, call:self._check_instance, method:_check_instance(self, session: AsyncSession, instance: ToolInstance) → None, call:logger.debug, call:get_container_status, call:logger.exception, call:str, call:get_correlation_id, call:check_tunnel_health, call:tunnel_result.get, call:HealthSnapshot, call:self._last_known_state.get, call:self._derive_status, call:self._snapshots_equal, call:self._handle_state_change, method:_derive_status(self, snapshot: HealthSnapshot, previous: HealthSnapshot | None, current_status: str | None) → str, method:_snapshots_equal(self, a: HealthSnapshot, b: HealthSnapshot) → bool, method:_handle_state_change(self, session: AsyncSession, instance: ToolInstance, previous: HealthSnapshot | None, snapshot: HealthSnapshot, new_status: str) → None, call:HealthCheck, call:session.add, call:session.commit, call:get_correlation_id, call:str, call:datetime.now(timezone.utc).isoformat, call:self._event_bus.publish, call:notification_service.create_notification, call:logger.exception | dep: asyncio, logging, uuid, dataclasses, datetime, sqlalchemy, sqlalchemy.ext.asyncio, src.database, src.models, src.services.shared.correlation, src.services.docker, src.services.shared.tunnel, src.services.instance.event_bus, src.services.shared.notification_service
|
||||
- health_monitor.py | Background health monitor that polls Docker container and tunnel health for tool instances, publishes state change events, and creates notifications for errors/unhealthy states. | exp: class:HealthSnapshot, class:HealthMonitor, method:__init__(self, event_bus: InstanceEventBus) → None, method:start(self) → None, call:self._task.done, call:asyncio.get_running_loop, call:loop.create_task, call:self._poll_loop, method:stop(self) → None, call:self._task.done, call:self._task.cancel, call:self._last_known_state.clear, method:_poll_loop(self) → None, call:asyncio.sleep, call:self._run_check_cycle, call:logger.exception, method:_run_check_cycle(self) → None, call:SessionLocal, call:session.execute, call:select(ToolInstance).where, call:ToolInstance.status.in_, call:result.scalars().all, call:self._check_instance, method:_check_instance(self, session: AsyncSession, instance: ToolInstance) → None, call:logger.debug, call:get_container_status, call:logger.exception, call:str, call:get_correlation_id, call:check_tunnel_health, call:tunnel_result.get, call:HealthSnapshot, call:self._last_known_state.get, call:self._derive_status, call:self._snapshots_equal, call:self._handle_state_change, method:_derive_status(self, snapshot: HealthSnapshot, previous: HealthSnapshot | None, current_status: str | None) → str, method:_snapshots_equal(self, a: HealthSnapshot, b: HealthSnapshot) → bool, method:_handle_state_change(self, session: AsyncSession, instance: ToolInstance, previous: HealthSnapshot | None, snapshot: HealthSnapshot, new_status: str) → None, call:HealthCheck, call:session.add, call:session.commit, call:get_correlation_id, call:str, call:datetime.now(timezone.utc).isoformat, call:self._event_bus.publish, call:notification_service.create_notification, call:logger.exception | dep: asyncio, logging, uuid, dataclasses, datetime, sqlalchemy, sqlalchemy.ext.asyncio, src.database, src.models, src.services.shared.correlation, src.services.docker, src.services.shared.tunnel, src.services.instance.event_bus, src.services.shared.notification_service
|
||||
- lifecycle_hooks.py | Provides helpers to publish tool instance lifecycle events, persist audit records, and conditionally send user notifications. | exp: func:_derive_title(event_type: str) → str, call:mapping.get, call:event_type.replace("instance.", "").replace("_", " ").title, func:_should_notify(event_type: str, status: str | None) → bool, func:_build_payload(event_type: str, instance: ToolInstance, status, message, metadata) → InstanceEventPayload, call:str, call:datetime.now(timezone.utc).isoformat, call:get_correlation_id, func:_write_audit_row(session: AsyncSession, instance: ToolInstance, event_type: str, created_by, status, message, metadata) → InstanceEvent, call:InstanceEvent, call:event_type.replace, call:session.add, call:session.commit, func:publish_lifecycle_event(event_bus: InstanceEventBus, session: AsyncSession, instance: ToolInstance, event_type: str, created_by, status, message, metadata) → None, call:_build_payload, call:_write_audit_row, call:event_bus.publish, call:_should_notify, call:_derive_title, call:notification_service.create_notification, call:logger.exception, call:payload.get | dep: logging, uuid, datetime, sqlalchemy.ext.asyncio, src.models, src.services.shared.correlation, src.services.instance.event_bus, src.services.shared.notification_service
|
||||
## arch
|
||||
Event-driven architecture using a singleton in-memory pub/sub event bus with typed messages, background polling workers, and lifecycle hooks that bridge domain events to persistence and notifications with exception isolation between sync/async handlers.
|
||||
Observer pattern via typed singleton event bus with async/sync subscribers, background polling loops, and side-effect hooks for persistence and notifications.
|
||||
## tags
|
||||
call:self., instance, src, services, event, health, call:logger.exception, check
|
||||
## symbols
|
||||
|
||||
@@ -203,6 +203,11 @@ class HealthMonitor:
|
||||
# Transient states (created, restarting) — preserve current status
|
||||
# instead of treating them as an error. The next poll will resolve.
|
||||
if snapshot.container_status in ("created", "restarting"):
|
||||
# A container that was already running and is now restarting has
|
||||
# crashed (e.g. entrypoint failure / restart loop). Mark it failed
|
||||
# so the dashboard does not keep showing it as running.
|
||||
if current_status == "running":
|
||||
return "error"
|
||||
return current_status or "starting"
|
||||
|
||||
# Unknown/unexpected state (paused, etc.)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: apps/api/src/services/tool
|
||||
|
||||
## role
|
||||
Provides containerized execution environment for tools by managing Docker instances, git repositories, and compose orchestration.
|
||||
Provides Docker container lifecycle management for tool instances with git repository mounting, configuration resolution, and SSH tunnel connectivity.
|
||||
## parent
|
||||
index: apps/api/src/services/.pi-map.index.md
|
||||
map: apps/api/src/services/.pi-map.md
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -13,7 +13,14 @@ from fastapi import HTTPException, status
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from src.models import ConfigProfile, GitRepository, Project, SSHKey, ToolInstance, ToolType
|
||||
from src.models import (
|
||||
ConfigProfile,
|
||||
GitRepository,
|
||||
Project,
|
||||
SSHKey,
|
||||
ToolInstance,
|
||||
ToolType,
|
||||
)
|
||||
from src.schemas.tool import CreateInstanceRequest, StartInstanceRequest
|
||||
from src.services.git.clone import check_dirty_state, clone_repository
|
||||
from src.services.config.config_profile_resolver import (
|
||||
@@ -32,6 +39,7 @@ from src.services.docker import (
|
||||
get_container_id,
|
||||
get_container_ip_on_network,
|
||||
get_container_logs,
|
||||
get_container_status,
|
||||
is_container_on_network,
|
||||
render_compose_template,
|
||||
sort_volumes_by_specificity,
|
||||
@@ -774,7 +782,6 @@ def ensure_backend_network_in_compose(compose_path: str) -> None:
|
||||
logger.info("Injected backend network '%s' into compose file", network_name)
|
||||
|
||||
|
||||
|
||||
async def prepare_manifest_instance(
|
||||
session: AsyncSession,
|
||||
instance: ToolInstance,
|
||||
@@ -918,8 +925,6 @@ async def prepare_manifest_instance(
|
||||
return image_tag, compose_content, manifest, home_dir
|
||||
|
||||
|
||||
|
||||
|
||||
async def create_tool_instance(
|
||||
session: AsyncSession,
|
||||
user_id: uuid.UUID,
|
||||
@@ -1047,15 +1052,25 @@ async def create_tool_instance(
|
||||
repo_name = os.path.basename(os.path.normpath(repo_path))
|
||||
workspace_target = f"{home_dir}/{repo_name}"
|
||||
|
||||
compose_content = f"""version: "3.8"\nservices:\n app:\n image: {image_tag}\n container_name: {instance_name.lower()}\n stdin_open: true\n tty: true\n{ports_section} environment:\n - HOME={home_dir}\n volumes:\n - {repo_path}:{workspace_target}\n working_dir: {workspace_target}\n restart: unless-stopped\n"""
|
||||
compose_content = f"""version: "3.8"\nservices:
|
||||
app:
|
||||
image: {image_tag}
|
||||
container_name: {instance_name.lower()}
|
||||
stdin_open: true
|
||||
tty: true
|
||||
{ports_section} environment:
|
||||
- HOME={home_dir}
|
||||
volumes:
|
||||
- {repo_path}:{workspace_target}
|
||||
working_dir: {workspace_target}
|
||||
restart: "no"
|
||||
"""
|
||||
write_compose_file(instance_dir, compose_content)
|
||||
|
||||
elif tool_type.definition_type == "manifest":
|
||||
from src.models import ToolDefinitionManifest
|
||||
|
||||
manifest_def = await session.get(
|
||||
ToolDefinitionManifest, tool_type.manifest_id
|
||||
)
|
||||
manifest_def = await session.get(ToolDefinitionManifest, tool_type.manifest_id)
|
||||
if not manifest_def:
|
||||
raise RuntimeError("Manifest definition not found for this tool type")
|
||||
|
||||
@@ -1065,9 +1080,7 @@ async def create_tool_instance(
|
||||
ToolDefinitionManifest, manifest_def.base_definition_id
|
||||
)
|
||||
if base_def:
|
||||
manifest = resolve_base(
|
||||
deep_merge(dict(base_def.manifest), manifest)
|
||||
)
|
||||
manifest = resolve_base(deep_merge(dict(base_def.manifest), manifest))
|
||||
|
||||
image_tag = compute_image_tag(tool_type.name, manifest)
|
||||
|
||||
@@ -1105,9 +1118,7 @@ async def create_tool_instance(
|
||||
"WORKSPACE_NAME": os.path.basename(os.path.normpath(repo_path)),
|
||||
"HOME_DIRECTORY": tool_type.home_directory or "/home/user",
|
||||
}
|
||||
compose_content = render_compose_template(
|
||||
tool_type.compose_template, variables
|
||||
)
|
||||
compose_content = render_compose_template(tool_type.compose_template, variables)
|
||||
|
||||
write_compose_file(instance_dir, compose_content)
|
||||
|
||||
@@ -1674,6 +1685,41 @@ async def start_tool_instance(
|
||||
|
||||
logger.info("Readiness probe succeeded for instance %s", instance.id)
|
||||
|
||||
# Final stability check: the container must still be running after all
|
||||
# post-start setup. If it has already exited/restarted, mark it failed now
|
||||
# instead of optimistically reporting "running".
|
||||
if instance.container_id:
|
||||
final_check = get_container_status(instance.container_id)
|
||||
if final_check["status"] != "running":
|
||||
error_msg = (
|
||||
f"Container stopped during startup: status={final_check['status']}"
|
||||
)
|
||||
if final_check["exit_code"] is not None:
|
||||
error_msg += f", exit_code={final_check['exit_code']}"
|
||||
logs = get_container_logs(instance.container_id, tail=50)
|
||||
instance.status = "error"
|
||||
await session.commit()
|
||||
await publish_lifecycle_event(
|
||||
event_bus=_event_bus,
|
||||
session=session,
|
||||
instance=instance,
|
||||
event_type="instance.error",
|
||||
created_by=user_id,
|
||||
status="error",
|
||||
message=error_msg,
|
||||
metadata={
|
||||
"exit_code": final_check["exit_code"],
|
||||
"error_type": "container",
|
||||
},
|
||||
)
|
||||
logger.error(
|
||||
"Instance %s container stopped during startup: %s\nLogs:\n%s",
|
||||
instance.id,
|
||||
error_msg,
|
||||
logs,
|
||||
)
|
||||
return {"status": "error", "error": error_msg, "logs": logs}
|
||||
|
||||
instance.status = "running"
|
||||
await session.commit()
|
||||
await publish_lifecycle_event(
|
||||
@@ -2102,7 +2148,9 @@ async def stop_tool_instance(
|
||||
try:
|
||||
stop_tunnel(instance.name)
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to stop tunnel for instance %s: %s", instance.id, exc)
|
||||
logger.warning(
|
||||
"Failed to stop tunnel for instance %s: %s", instance.id, exc
|
||||
)
|
||||
|
||||
if instance.compose_path and os.path.exists(instance.compose_path):
|
||||
execute_compose_command(instance.compose_path, "stop")
|
||||
@@ -2144,4 +2192,3 @@ async def rename_tool_instance(
|
||||
await session.commit()
|
||||
await session.refresh(instance)
|
||||
return instance
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: openspec
|
||||
|
||||
## role
|
||||
Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a project repository.
|
||||
Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a Docker-based coding agent management platform.
|
||||
## parent
|
||||
index: ./.pi-map.index.md
|
||||
map: ./.pi-map.md
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
dir: openspec/changes
|
||||
|
||||
## role
|
||||
Manages and tracks specification changes, diffs, and versioning history for OpenAPI documents.
|
||||
Provides change-tracking and diffing utilities for OpenAPI specification evolution and version comparison.
|
||||
## parent
|
||||
index: openspec/.pi-map.index.md
|
||||
map: openspec/.pi-map.md
|
||||
@@ -10,6 +10,9 @@ map: openspec/.pi-map.md
|
||||
- openspec/changes/archive
|
||||
index: openspec/changes/archive/.pi-map.index.md
|
||||
map: openspec/changes/archive/.pi-map.md
|
||||
- openspec/changes/fix-container-status-false-positive
|
||||
index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md
|
||||
map: openspec/changes/fix-container-status-false-positive/.pi-map.md
|
||||
- openspec/changes/fix-pi-container-mount-permissions
|
||||
index: openspec/changes/fix-pi-container-mount-permissions/.pi-map.index.md
|
||||
map: openspec/changes/fix-pi-container-mount-permissions/.pi-map.md
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
# openspec/changes/fix-container-status-false-positive (index)
|
||||
dir: openspec/changes/fix-container-status-false-positive
|
||||
|
||||
## role
|
||||
Documents a bug fix for correcting container status false positives where failed containers incorrectly appeared as running in a dashboard/monitoring system.
|
||||
## parent
|
||||
index: openspec/changes/.pi-map.index.md
|
||||
map: openspec/changes/.pi-map.md
|
||||
## children
|
||||
-
|
||||
## files
|
||||
- change.md
|
||||
- tasks.md
|
||||
## links
|
||||
index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md
|
||||
map: openspec/changes/fix-container-status-false-positive/.pi-map.md
|
||||
## workflows
|
||||
-
|
||||
## dirty
|
||||
-
|
||||
@@ -0,0 +1,20 @@
|
||||
# openspec/changes/fix-container-status-false-positive
|
||||
dir: openspec/changes/fix-container-status-false-positive
|
||||
|
||||
index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md
|
||||
|
||||
## role
|
||||
Documents a bug fix for correcting container status false positives where failed containers incorrectly appeared as running in a dashboard/monitoring system.
|
||||
## files
|
||||
- change.md | Documents a bug fix for failed containers incorrectly displaying as running on a dashboard, detailing root cause analysis and the fix implementation across multiple files.
|
||||
- tasks.md | Track progress of fixing a false-positive running status bug in a container/system management system
|
||||
## arch
|
||||
Issue-tracking documentation pattern using markdown files (change.md for technical specification, tasks.md for progress tracking) to manage a bug fix across multiple system components.
|
||||
## tags
|
||||
bug, fix, running, change, system, tasks, documents, failed
|
||||
## symbols
|
||||
-
|
||||
## workflows
|
||||
-
|
||||
## dirty
|
||||
-
|
||||
@@ -0,0 +1,32 @@
|
||||
# Fix: failed containers shown as running on dashboard
|
||||
|
||||
## Problem
|
||||
|
||||
Containers that fail during startup (e.g. `ln: failed to create symbolic link '/workspace': Permission denied`) are still displayed as "Running" on the web dashboard and session list.
|
||||
|
||||
## Root cause
|
||||
|
||||
1. `start_tool_instance` in `apps/api/src/services/tool/instance_service.py` sets `instance.status = "running"` after post-start setup without verifying the container is still up.
|
||||
2. Generated compose files use `restart: unless-stopped`, so Docker immediately restarts a crashed container, putting it into the `restarting` state.
|
||||
3. The background `HealthMonitor` treats `restarting` as a transient state and preserves the current DB status (`running`).
|
||||
|
||||
## Fix
|
||||
|
||||
1. Add a final `get_container_status` check in `start_tool_instance` immediately before writing `status = "running"`. If the container has stopped/exited, mark it as `error` and return the logs.
|
||||
2. Update `HealthMonitor._derive_status` so that when the DB status is `running` and Docker reports `restarting`, the instance is marked as `error`.
|
||||
3. Disable auto-restart for tool instances by changing `restart: unless-stopped` to `restart: "no"` in:
|
||||
- `apps/api/src/services/build/manifest_compiler.py`
|
||||
- `apps/api/src/services/tool/instance_service.py` (legacy dockerfile path)
|
||||
- `apps/api/src/seeds/builtin_tool_types.py`
|
||||
|
||||
## Affected files
|
||||
|
||||
- `apps/api/src/services/tool/instance_service.py`
|
||||
- `apps/api/src/services/instance/health_monitor.py`
|
||||
- `apps/api/src/services/build/manifest_compiler.py`
|
||||
- `apps/api/src/seeds/builtin_tool_types.py`
|
||||
|
||||
## Verification
|
||||
|
||||
- `pytest apps/api/tests/unit`
|
||||
- `ruff`, `mypy` on changed files
|
||||
@@ -0,0 +1,8 @@
|
||||
# Tasks: fix false-positive running status
|
||||
|
||||
- [x] Investigate status flow and identify root cause
|
||||
- [x] Add final container status check before writing "running"
|
||||
- [x] Treat restarting as error when DB status is already running
|
||||
- [x] Disable auto-restart in manifest compiler, legacy path, and seed data
|
||||
- [ ] Run quality gates
|
||||
- [ ] Commit and push
|
||||
Reference in New Issue
Block a user