fix: prevent failed containers from showing as running on dashboard

- Add final get_container_status check in start_tool_instance before
  writing status=running; mark as error and return logs if container stopped
- Treat restarting as error in HealthMonitor when DB status was already
  running, so crash loops are surfaced instead of preserved
- Disable auto-restart (restart: unless-stopped -> restart: no) for tool
  instances in manifest compiler, legacy dockerfile path, and built-in seeds

Quality gates:
- pytest tests/unit: 210 passed
- ruff: clean on changed files
- mypy: clean on changed files
This commit is contained in:
Developer
2026-06-14 21:52:02 +00:00
parent a4e6c46a47
commit 089d802f1d
25 changed files with 185 additions and 48 deletions
+2 -1
View File
@@ -16,7 +16,7 @@ dir: .
Trust boundary: index routes, map orients, source decides.
## role
Infrastructure and deployment configuration for a self-hosted project management platform with containerized services, SSO integration, and AI agent development workflows.
A self-hosted web-based project management and git repository platform with OAuth2 SSO authentication, containerized for Docker deployment.
## parent
-
## children
@@ -69,6 +69,7 @@ Infrastructure and deployment configuration for a self-hosted project management
- CHANGELOG.md
- Makefile
- README.md
- context.md
- docker-compose.traefik.yml
- docker-compose.yml
- progress.md
+4 -3
View File
@@ -18,7 +18,7 @@ index: ./.pi-map.index.md
Trust boundary: index routes, map orients, source decides.
## role
Infrastructure and deployment configuration for a self-hosted project management platform with containerized services, SSO integration, and AI agent development workflows.
A self-hosted web-based project management and git repository platform with OAuth2 SSO authentication, containerized for Docker deployment.
## files
- .env.example | Provides a template of environment variables for configuring a Headquarter application with PostgreSQL, Redis, Authentik SSO, and Docker/Traefik deployment
- .gitignore | Specifies files and directories for Git to ignore across a multi-language project with Python, Node, and custom tooling | dep: Git
@@ -26,14 +26,15 @@ Infrastructure and deployment configuration for a self-hosted project management
- CHANGELOG.md | Documents version history and notable changes for a Git-based project management web application
- Makefile | Provides standard development commands for containerized web application lifecycle management via Docker Compose | dep: docker compose, alembic, pytest, ruff, mypy, playwright, npm, postgres, redis
- README.md | A self-hosted platform for managing projects, git repositories, and development tools with OAuth2 authentication. | dep: FastAPI, SQLAlchemy, Pydantic, Alembic, python-jose, React, TypeScript, Vite, React Router, Docker, PostgreSQL, Traefik, Authentik, Git
- context.md | Documents a bug investigation where crashed Docker containers incorrectly display "running" status on the dashboard due to optimistic status writes, auto-restart policies, and health monitor logic that preserves running status for restarting containers. | dep: docker, docker-compose, FastAPI/SQLAlchemy, SSE event bus, React/TypeScript frontend
- docker-compose.traefik.yml | Deploys a multi-service web application (frontend, API, PostgreSQL, Redis) behind an existing Traefik reverse proxy with TLS termination and environment-configurable domains. | dep: docker, traefik, postgres, redis, authentik, docker-compose
- docker-compose.yml | Defines a multi-service Docker Compose stack with PostgreSQL, Redis, API backend, and web frontend services for a "headquarter" application | dep: Docker, PostgreSQL, Redis, Vite, asyncpg, nginx
- progress.md | Tracks completed and remaining tasks for a backend-frontend code refactoring project organized in 7 phases
- swap-pane | Empty file with no functionality
## arch
Docker Compose-based microservices architecture with PostgreSQL/Redis data layer, Traefik reverse proxy, OAuth2 SSO via Authentik, and OpenSpec-driven AI agent orchestration.
Full-stack containerized architecture with separate frontend/API services, PostgreSQL/Redis data layer, Traefik reverse proxy with TLS, environment-driven configuration, and health monitoring with auto-restart policies.
## tags
docker, redis, git, application, postgresql, compose, traefik, project
docker, redis, git, compose, application, postgresql, traefik, project
## symbols
-
## workflows
+1 -1
View File
@@ -2,7 +2,7 @@
dir: apps
## role
Contains the deployable application entry points and main executables for the project.
Contains the deployable application entry points and executable configurations for the project.
## parent
index: ./.pi-map.index.md
map: ./.pi-map.md
+2 -2
View File
@@ -4,10 +4,10 @@ dir: apps
index: apps/.pi-map.index.md
## role
Contains the deployable application entry points and main executables for the project.
Contains the deployable application entry points and executable configurations for the project.
## files
## arch
Typically follows a multi-app workspace pattern where each subdirectory represents a standalone application with its own main.ts, configuration, and dependency injection setup, often sharing common libraries from other packages.
Modular application structure where each subdirectory represents a separate deployable unit or service, following a monorepo or multi-app pattern.
## tags
-
## symbols
+1 -1
View File
@@ -14,7 +14,7 @@ FastAPI backend API that manages projects, git repositories, and development too
- uv.lock | Lock file for the uv Python package manager that pins exact dependency versions and their artifact hashes for reproducible installations | dep: uv, Python 3.11+, aiosqlite, alembic, annotated-doc, annotated-types, anyio, ast-serialize, asyncpg, and many other PyPI packages
- wait-for-db.sh | Wait for a PostgreSQL database to become available before executing a command, with configurable retry logic. | dep: nc (netcat), sh (POSIX shell), sleep
## arch
Async Python/FastAPI service using multi-stage Docker builds, Alembic migrations, uv package management, and container orchestration with external tunneling and database dependency waiting patterns.
Async Python/FastAPI service with multi-stage Docker containerization, Alembic database migrations, uv package management, and Docker socket access for container orchestration.
## tags
docker, alembic, python, database, fastapi, postgresql, asyncpg, uvicorn
## symbols
+1 -1
View File
@@ -2,7 +2,7 @@
dir: apps/api/src
## role
Core API application package for the "Headquarter API" providing configuration, database infrastructure, logging, and FastAPI application initialization.
Core API server package for the "Headquarter" backend, handling configuration, database connectivity, logging, and FastAPI application initialization.
## parent
index: apps/api/.pi-map.index.md
map: apps/api/.pi-map.md
+2 -2
View File
@@ -4,7 +4,7 @@ dir: apps/api/src
index: apps/api/src/.pi-map.index.md
## role
Core API application package for the "Headquarter API" providing configuration, database infrastructure, logging, and FastAPI application initialization.
Core API server package for the "Headquarter" backend, handling configuration, database connectivity, logging, and FastAPI application initialization.
## files
- __init__.py | Marks the directory as a Python package for the Headquarter API.
- config.py | Defines application configuration settings with environment-based overrides using Pydantic, including database URLs, service domains, OAuth/Authentik integration, JWT/session settings, and computed properties for environment-specific behavior. | exp: class:Settings, func:build_database_url(user: str, password: str, host: str, port: int, database: str) → str | dep: pydantic, pydantic_settings
@@ -12,7 +12,7 @@ Core API application package for the "Headquarter API" providing configuration,
- logging_config.py | Configures structured JSON logging with correlation ID injection, custom formatters, and HTTP request/exception middleware for a FastAPI application. | exp: class:CorrelationIdFilter, method:filter(self, record: logging.LogRecord) → bool, call:get_correlation_id, class:JSONFormatter, method:format(self, record: logging.LogRecord) → str, call:self.formatTime, call:record.getMessage, call:getattr, call:self.formatException, call:json.dumps, method:formatTime(self, record: logging.LogRecord, datefmt) → str, call:time.strftime, call:time.gmtime, class:RequestLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:time.time, call:logger.info, call:call_next, call:int, call:logger.error, call:type, call:traceback.format_exc, class:ExceptionLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:call_next, call:logger.critical, call:traceback.format_exc, func:configure_logging(level) → None, call:JSONFormatter, call:logging.StreamHandler, call:console_handler.setFormatter, call:console_handler.addFilter, call:CorrelationIdFilter, call:root_logger.setLevel, call:logging.getLogger("uvicorn").setLevel, call:logging.getLogger("uvicorn.access").setLevel, call:logging.getLogger("sqlalchemy.engine").setLevel, call:logger.info, call:logging.getLevelName | dep: json, logging, sys, time, traceback, collections.abc, fastapi, starlette.middleware.base, src.services.shared.correlation
- main.py | Initializes and configures a FastAPI application for the "Headquarter API" with database setup, middleware, routing, and background services. | exp: func:_sanitize_validation_errors(errors), call:error.get, call:str, call:ctx.items, call:isinstance, call:type, call:sanitized.append, func:validation_exception_handler(request: Request, exc: RequestValidationError), call:exc.errors, call:logger.warning, call:_sanitize_validation_errors, call:JSONResponse, func:on_startup(), call:logger.info, call:init_database, call:logger.error, call:sys.exit, call:_health_monitor.start, call:seed_builtin_tool_types, func:on_shutdown(), call:logger.info, call:_health_monitor.stop | dep: logging, os, fastapi, fastapi.exceptions, fastapi.middleware.cors, fastapi.responses, fastapi.staticfiles, src.api.config, src.api.project, src.api.system, src.api.tool, src.api.user, src.api.workspace, src.config, src.models, src.database, src.logging_config, src.seeds.builtin_tool_types, src.services.instance, src.services.shared, sys, src.api.*
## arch
Layered architecture with Pydantic-based config management, async SQLAlchemy with Alembic migrations, structured JSON logging with correlation IDs, and FastAPI middleware/routing pattern.
Layered async architecture using FastAPI with Pydantic settings management, SQLAlchemy async ORM with Alembic migrations, structured JSON logging with correlation ID tracking, and environment-driven configuration with OAuth/Authentik integration.
## tags
src, database, logging, call:logger.info, api, middleware, fastapi, filter
## symbols
+1 -1
View File
@@ -2,7 +2,7 @@
dir: apps/api/src/seeds
## role
Provides database seeding utilities for initializing built-in tool type configurations in the API application.
Provides database seeding utilities for initializing and synchronizing built-in data records in the API application.
## parent
index: apps/api/src/.pi-map.index.md
map: apps/api/src/.pi-map.md
+3 -3
View File
@@ -4,12 +4,12 @@ dir: apps/api/src/seeds
index: apps/api/src/seeds/.pi-map.index.md
## role
Provides database seeding utilities for initializing built-in tool type configurations in the API application.
Provides database seeding utilities for initializing and synchronizing built-in data records in the API application.
## files
- __init__.py | Marks the directory as a Python package for database seeding utilities.
- builtin_tool_types.py | Seeds built-in tool types (code-server, jupyter-notebook, opencode) into a database with upsert logic, creating or updating Docker Compose-based development environment templates. | exp: func:_table_exists(session, table_name: str) → bool, call:session.execute, call:text, call:result.scalar, func:seed_builtin_tool_types(), call:SessionLocal, call:_table_exists, call:logger.warning, call:session.scalar, call:select(ToolType).where, call:ToolType, call:tool_data.get, call:session.add, call:logger.info, call:session.commit | dep: logging, sqlalchemy, src.database, src.models
- builtin_tool_types.py | Seeds predefined built-in tool types (code-server, jupyter-notebook, opencode) into a database with upsert logic, creating them if missing or updating existing ones to match code changes. | exp: func:_table_exists(session, table_name: str) → bool, call:session.execute, call:text, call:result.scalar, func:seed_builtin_tool_types(), call:SessionLocal, call:_table_exists, call:logger.warning, call:session.scalar, call:select(ToolType).where, call:ToolType, call:tool_data.get, call:session.add, call:logger.info, call:session.commit | dep: logging, sqlalchemy, src.database, src.models
## arch
Simple procedural seeding script using SQLAlchemy upsert operations to populate reference data for containerized development environment templates.
Simple imperative seeding scripts with upsert pattern for idempotent data initialization, using direct database operations without abstraction layers.
## tags
tool, types, table, exists, builtin, call:tool, database, init
## symbols
+3 -3
View File
@@ -65,7 +65,7 @@ services:
- {{REPO_PATH}}:/config/workspace
ports:
- "8443:8443"
restart: unless-stopped""",
restart: 'no'""",
"default_port": 8443,
"required_variables": ["REPO_PATH", "TOOL_NAME"],
},
@@ -87,7 +87,7 @@ services:
- {{REPO_PATH}}:/home/jovyan/work
ports:
- "8888:8888"
restart: unless-stopped""",
restart: 'no'""",
"required_variables": ["REPO_PATH", "TOOL_NAME"],
},
{
@@ -122,7 +122,7 @@ services:
exec tail -f /dev/null"
stdin_open: true
tty: true
restart: unless-stopped""",
restart: 'no'""",
"required_variables": ["REPO_PATH", "TOOL_NAME"],
},
]
+1 -1
View File
@@ -2,7 +2,7 @@
dir: apps/api/src/services
## role
Provides business logic and service layer abstractions for the API application.
Marker package for the services layer in the API application
## parent
index: apps/api/src/.pi-map.index.md
map: apps/api/src/.pi-map.md
+2 -2
View File
@@ -4,11 +4,11 @@ dir: apps/api/src/services
index: apps/api/src/services/.pi-map.index.md
## role
Provides business logic and service layer abstractions for the API application.
Marker package for the services layer in the API application
## files
- __init__.py | Empty file with no functionality
## arch
Minimal or placeholder package structure with no implemented services yet, following standard Python package conventions.
Standard Python package structure using __init__.py for namespace declaration
## tags
init, empty, functionality
## symbols
@@ -411,7 +411,7 @@ def compile_compose(manifest: dict, variables: dict[str, Any]) -> str:
service: dict[str, Any] = {
"image": variables["IMAGE_TAG"],
"container_name": variables["INSTANCE_NAME"],
"restart": "unless-stopped",
"restart": "no",
}
# Terminal-specific fields
@@ -2,7 +2,7 @@
dir: apps/api/src/services/instance
## role
Provides infrastructure for managing tool instance lifecycle events, health monitoring, and asynchronous communication within the API service.
Coordinates tool instance lifecycle events, health monitoring, and notifications across the API service.
## parent
index: apps/api/src/services/.pi-map.index.md
map: apps/api/src/services/.pi-map.md
+3 -3
View File
@@ -4,14 +4,14 @@ dir: apps/api/src/services/instance
index: apps/api/src/services/instance/.pi-map.index.md
## role
Provides infrastructure for managing tool instance lifecycle events, health monitoring, and asynchronous communication within the API service.
Coordinates tool instance lifecycle events, health monitoring, and notifications across the API service.
## files
- __init__.py | Exports public API for instance lifecycle services module | dep: src.services.instance.event_bus, src.services.instance.health_monitor, src.services.instance.lifecycle_hooks
- event_bus.py | Implements a singleton in-memory typed event bus with publish/subscribe pattern for instance lifecycle and health events, supporting both sync and async callbacks with exception isolation. | exp: class:InstanceEventBus, method:__init__(self) → None, method:__new__(cls) → "InstanceEventBus", call:super().__new__, method:_reset_for_testing(self) → None, call:self._subscribers.clear, method:subscribe(self, event_type: str, callback: EventCallback) → Callable[[], None], call:str, call:uuid.uuid4, call:self._subscribers[event_type].append, call:self.unsubscribe, method:unsubscribe(self, event_type: str, callback_id: str) → None, method:unsubscribe_all(self, event_type: str) → None, call:self._subscribers.pop, method:publish(self, event_type: str, payload: InstanceEventPayload) → None, call:callbacks.extend, call:self._subscribers.get, call:inspect.iscoroutinefunction, call:callback, call:payload.get, call:logger.exception | dep: asyncio, inspect, logging, uuid, collections.abc, typing
- health_monitor.py | Background health monitor that periodically polls Docker container and tunnel health for tool instances, publishing state change events and notifications. | exp: class:HealthSnapshot, class:HealthMonitor, method:__init__(self, event_bus: InstanceEventBus) → None, method:start(self) → None, call:self._task.done, call:asyncio.get_running_loop, call:loop.create_task, call:self._poll_loop, method:stop(self) → None, call:self._task.done, call:self._task.cancel, call:self._last_known_state.clear, method:_poll_loop(self) → None, call:asyncio.sleep, call:self._run_check_cycle, call:logger.exception, method:_run_check_cycle(self) → None, call:SessionLocal, call:session.execute, call:select(ToolInstance).where, call:ToolInstance.status.in_, call:result.scalars().all, call:self._check_instance, method:_check_instance(self, session: AsyncSession, instance: ToolInstance) → None, call:logger.debug, call:get_container_status, call:logger.exception, call:str, call:get_correlation_id, call:check_tunnel_health, call:tunnel_result.get, call:HealthSnapshot, call:self._last_known_state.get, call:self._derive_status, call:self._snapshots_equal, call:self._handle_state_change, method:_derive_status(self, snapshot: HealthSnapshot, previous: HealthSnapshot | None, current_status: str | None) → str, method:_snapshots_equal(self, a: HealthSnapshot, b: HealthSnapshot) → bool, method:_handle_state_change(self, session: AsyncSession, instance: ToolInstance, previous: HealthSnapshot | None, snapshot: HealthSnapshot, new_status: str) → None, call:HealthCheck, call:session.add, call:session.commit, call:get_correlation_id, call:str, call:datetime.now(timezone.utc).isoformat, call:self._event_bus.publish, call:notification_service.create_notification, call:logger.exception | dep: asyncio, logging, uuid, dataclasses, datetime, sqlalchemy, sqlalchemy.ext.asyncio, src.database, src.models, src.services.shared.correlation, src.services.docker, src.services.shared.tunnel, src.services.instance.event_bus, src.services.shared.notification_service
- health_monitor.py | Background health monitor that polls Docker container and tunnel health for tool instances, publishes state change events, and creates notifications for errors/unhealthy states. | exp: class:HealthSnapshot, class:HealthMonitor, method:__init__(self, event_bus: InstanceEventBus) → None, method:start(self) → None, call:self._task.done, call:asyncio.get_running_loop, call:loop.create_task, call:self._poll_loop, method:stop(self) → None, call:self._task.done, call:self._task.cancel, call:self._last_known_state.clear, method:_poll_loop(self) → None, call:asyncio.sleep, call:self._run_check_cycle, call:logger.exception, method:_run_check_cycle(self) → None, call:SessionLocal, call:session.execute, call:select(ToolInstance).where, call:ToolInstance.status.in_, call:result.scalars().all, call:self._check_instance, method:_check_instance(self, session: AsyncSession, instance: ToolInstance) → None, call:logger.debug, call:get_container_status, call:logger.exception, call:str, call:get_correlation_id, call:check_tunnel_health, call:tunnel_result.get, call:HealthSnapshot, call:self._last_known_state.get, call:self._derive_status, call:self._snapshots_equal, call:self._handle_state_change, method:_derive_status(self, snapshot: HealthSnapshot, previous: HealthSnapshot | None, current_status: str | None) → str, method:_snapshots_equal(self, a: HealthSnapshot, b: HealthSnapshot) → bool, method:_handle_state_change(self, session: AsyncSession, instance: ToolInstance, previous: HealthSnapshot | None, snapshot: HealthSnapshot, new_status: str) → None, call:HealthCheck, call:session.add, call:session.commit, call:get_correlation_id, call:str, call:datetime.now(timezone.utc).isoformat, call:self._event_bus.publish, call:notification_service.create_notification, call:logger.exception | dep: asyncio, logging, uuid, dataclasses, datetime, sqlalchemy, sqlalchemy.ext.asyncio, src.database, src.models, src.services.shared.correlation, src.services.docker, src.services.shared.tunnel, src.services.instance.event_bus, src.services.shared.notification_service
- lifecycle_hooks.py | Provides helpers to publish tool instance lifecycle events, persist audit records, and conditionally send user notifications. | exp: func:_derive_title(event_type: str) → str, call:mapping.get, call:event_type.replace("instance.", "").replace("_", " ").title, func:_should_notify(event_type: str, status: str | None) → bool, func:_build_payload(event_type: str, instance: ToolInstance, status, message, metadata) → InstanceEventPayload, call:str, call:datetime.now(timezone.utc).isoformat, call:get_correlation_id, func:_write_audit_row(session: AsyncSession, instance: ToolInstance, event_type: str, created_by, status, message, metadata) → InstanceEvent, call:InstanceEvent, call:event_type.replace, call:session.add, call:session.commit, func:publish_lifecycle_event(event_bus: InstanceEventBus, session: AsyncSession, instance: ToolInstance, event_type: str, created_by, status, message, metadata) → None, call:_build_payload, call:_write_audit_row, call:event_bus.publish, call:_should_notify, call:_derive_title, call:notification_service.create_notification, call:logger.exception, call:payload.get | dep: logging, uuid, datetime, sqlalchemy.ext.asyncio, src.models, src.services.shared.correlation, src.services.instance.event_bus, src.services.shared.notification_service
## arch
Event-driven architecture using a singleton in-memory pub/sub event bus with typed messages, background polling workers, and lifecycle hooks that bridge domain events to persistence and notifications with exception isolation between sync/async handlers.
Observer pattern via typed singleton event bus with async/sync subscribers, background polling loops, and side-effect hooks for persistence and notifications.
## tags
call:self., instance, src, services, event, health, call:logger.exception, check
## symbols
@@ -203,6 +203,11 @@ class HealthMonitor:
# Transient states (created, restarting) — preserve current status
# instead of treating them as an error. The next poll will resolve.
if snapshot.container_status in ("created", "restarting"):
# A container that was already running and is now restarting has
# crashed (e.g. entrypoint failure / restart loop). Mark it failed
# so the dashboard does not keep showing it as running.
if current_status == "running":
return "error"
return current_status or "starting"
# Unknown/unexpected state (paused, etc.)
+1 -1
View File
@@ -2,7 +2,7 @@
dir: apps/api/src/services/tool
## role
Provides containerized execution environment for tools by managing Docker instances, git repositories, and compose orchestration.
Provides Docker container lifecycle management for tool instances with git repository mounting, configuration resolution, and SSH tunnel connectivity.
## parent
index: apps/api/src/services/.pi-map.index.md
map: apps/api/src/services/.pi-map.md
File diff suppressed because one or more lines are too long
+63 -16
View File
@@ -13,7 +13,14 @@ from fastapi import HTTPException, status
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from src.models import ConfigProfile, GitRepository, Project, SSHKey, ToolInstance, ToolType
from src.models import (
ConfigProfile,
GitRepository,
Project,
SSHKey,
ToolInstance,
ToolType,
)
from src.schemas.tool import CreateInstanceRequest, StartInstanceRequest
from src.services.git.clone import check_dirty_state, clone_repository
from src.services.config.config_profile_resolver import (
@@ -32,6 +39,7 @@ from src.services.docker import (
get_container_id,
get_container_ip_on_network,
get_container_logs,
get_container_status,
is_container_on_network,
render_compose_template,
sort_volumes_by_specificity,
@@ -774,7 +782,6 @@ def ensure_backend_network_in_compose(compose_path: str) -> None:
logger.info("Injected backend network '%s' into compose file", network_name)
async def prepare_manifest_instance(
session: AsyncSession,
instance: ToolInstance,
@@ -918,8 +925,6 @@ async def prepare_manifest_instance(
return image_tag, compose_content, manifest, home_dir
async def create_tool_instance(
session: AsyncSession,
user_id: uuid.UUID,
@@ -1047,15 +1052,25 @@ async def create_tool_instance(
repo_name = os.path.basename(os.path.normpath(repo_path))
workspace_target = f"{home_dir}/{repo_name}"
compose_content = f"""version: "3.8"\nservices:\n app:\n image: {image_tag}\n container_name: {instance_name.lower()}\n stdin_open: true\n tty: true\n{ports_section} environment:\n - HOME={home_dir}\n volumes:\n - {repo_path}:{workspace_target}\n working_dir: {workspace_target}\n restart: unless-stopped\n"""
compose_content = f"""version: "3.8"\nservices:
app:
image: {image_tag}
container_name: {instance_name.lower()}
stdin_open: true
tty: true
{ports_section} environment:
- HOME={home_dir}
volumes:
- {repo_path}:{workspace_target}
working_dir: {workspace_target}
restart: "no"
"""
write_compose_file(instance_dir, compose_content)
elif tool_type.definition_type == "manifest":
from src.models import ToolDefinitionManifest
manifest_def = await session.get(
ToolDefinitionManifest, tool_type.manifest_id
)
manifest_def = await session.get(ToolDefinitionManifest, tool_type.manifest_id)
if not manifest_def:
raise RuntimeError("Manifest definition not found for this tool type")
@@ -1065,9 +1080,7 @@ async def create_tool_instance(
ToolDefinitionManifest, manifest_def.base_definition_id
)
if base_def:
manifest = resolve_base(
deep_merge(dict(base_def.manifest), manifest)
)
manifest = resolve_base(deep_merge(dict(base_def.manifest), manifest))
image_tag = compute_image_tag(tool_type.name, manifest)
@@ -1105,9 +1118,7 @@ async def create_tool_instance(
"WORKSPACE_NAME": os.path.basename(os.path.normpath(repo_path)),
"HOME_DIRECTORY": tool_type.home_directory or "/home/user",
}
compose_content = render_compose_template(
tool_type.compose_template, variables
)
compose_content = render_compose_template(tool_type.compose_template, variables)
write_compose_file(instance_dir, compose_content)
@@ -1674,6 +1685,41 @@ async def start_tool_instance(
logger.info("Readiness probe succeeded for instance %s", instance.id)
# Final stability check: the container must still be running after all
# post-start setup. If it has already exited/restarted, mark it failed now
# instead of optimistically reporting "running".
if instance.container_id:
final_check = get_container_status(instance.container_id)
if final_check["status"] != "running":
error_msg = (
f"Container stopped during startup: status={final_check['status']}"
)
if final_check["exit_code"] is not None:
error_msg += f", exit_code={final_check['exit_code']}"
logs = get_container_logs(instance.container_id, tail=50)
instance.status = "error"
await session.commit()
await publish_lifecycle_event(
event_bus=_event_bus,
session=session,
instance=instance,
event_type="instance.error",
created_by=user_id,
status="error",
message=error_msg,
metadata={
"exit_code": final_check["exit_code"],
"error_type": "container",
},
)
logger.error(
"Instance %s container stopped during startup: %s\nLogs:\n%s",
instance.id,
error_msg,
logs,
)
return {"status": "error", "error": error_msg, "logs": logs}
instance.status = "running"
await session.commit()
await publish_lifecycle_event(
@@ -2102,7 +2148,9 @@ async def stop_tool_instance(
try:
stop_tunnel(instance.name)
except Exception as exc:
logger.warning("Failed to stop tunnel for instance %s: %s", instance.id, exc)
logger.warning(
"Failed to stop tunnel for instance %s: %s", instance.id, exc
)
if instance.compose_path and os.path.exists(instance.compose_path):
execute_compose_command(instance.compose_path, "stop")
@@ -2144,4 +2192,3 @@ async def rename_tool_instance(
await session.commit()
await session.refresh(instance)
return instance
+1 -1
View File
@@ -2,7 +2,7 @@
dir: openspec
## role
Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a project repository.
Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a Docker-based coding agent management platform.
## parent
index: ./.pi-map.index.md
map: ./.pi-map.md
+4 -1
View File
@@ -2,7 +2,7 @@
dir: openspec/changes
## role
Manages and tracks specification changes, diffs, and versioning history for OpenAPI documents.
Provides change-tracking and diffing utilities for OpenAPI specification evolution and version comparison.
## parent
index: openspec/.pi-map.index.md
map: openspec/.pi-map.md
@@ -10,6 +10,9 @@ map: openspec/.pi-map.md
- openspec/changes/archive
index: openspec/changes/archive/.pi-map.index.md
map: openspec/changes/archive/.pi-map.md
- openspec/changes/fix-container-status-false-positive
index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md
map: openspec/changes/fix-container-status-false-positive/.pi-map.md
- openspec/changes/fix-pi-container-mount-permissions
index: openspec/changes/fix-pi-container-mount-permissions/.pi-map.index.md
map: openspec/changes/fix-pi-container-mount-permissions/.pi-map.md
@@ -0,0 +1,20 @@
# openspec/changes/fix-container-status-false-positive (index)
dir: openspec/changes/fix-container-status-false-positive
## role
Documents a bug fix for correcting container status false positives where failed containers incorrectly appeared as running in a dashboard/monitoring system.
## parent
index: openspec/changes/.pi-map.index.md
map: openspec/changes/.pi-map.md
## children
-
## files
- change.md
- tasks.md
## links
index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md
map: openspec/changes/fix-container-status-false-positive/.pi-map.md
## workflows
-
## dirty
-
@@ -0,0 +1,20 @@
# openspec/changes/fix-container-status-false-positive
dir: openspec/changes/fix-container-status-false-positive
index: openspec/changes/fix-container-status-false-positive/.pi-map.index.md
## role
Documents a bug fix for correcting container status false positives where failed containers incorrectly appeared as running in a dashboard/monitoring system.
## files
- change.md | Documents a bug fix for failed containers incorrectly displaying as running on a dashboard, detailing root cause analysis and the fix implementation across multiple files.
- tasks.md | Track progress of fixing a false-positive running status bug in a container/system management system
## arch
Issue-tracking documentation pattern using markdown files (change.md for technical specification, tasks.md for progress tracking) to manage a bug fix across multiple system components.
## tags
bug, fix, running, change, system, tasks, documents, failed
## symbols
-
## workflows
-
## dirty
-
@@ -0,0 +1,32 @@
# Fix: failed containers shown as running on dashboard
## Problem
Containers that fail during startup (e.g. `ln: failed to create symbolic link '/workspace': Permission denied`) are still displayed as "Running" on the web dashboard and session list.
## Root cause
1. `start_tool_instance` in `apps/api/src/services/tool/instance_service.py` sets `instance.status = "running"` after post-start setup without verifying the container is still up.
2. Generated compose files use `restart: unless-stopped`, so Docker immediately restarts a crashed container, putting it into the `restarting` state.
3. The background `HealthMonitor` treats `restarting` as a transient state and preserves the current DB status (`running`).
## Fix
1. Add a final `get_container_status` check in `start_tool_instance` immediately before writing `status = "running"`. If the container has stopped/exited, mark it as `error` and return the logs.
2. Update `HealthMonitor._derive_status` so that when the DB status is `running` and Docker reports `restarting`, the instance is marked as `error`.
3. Disable auto-restart for tool instances by changing `restart: unless-stopped` to `restart: "no"` in:
- `apps/api/src/services/build/manifest_compiler.py`
- `apps/api/src/services/tool/instance_service.py` (legacy dockerfile path)
- `apps/api/src/seeds/builtin_tool_types.py`
## Affected files
- `apps/api/src/services/tool/instance_service.py`
- `apps/api/src/services/instance/health_monitor.py`
- `apps/api/src/services/build/manifest_compiler.py`
- `apps/api/src/seeds/builtin_tool_types.py`
## Verification
- `pytest apps/api/tests/unit`
- `ruff`, `mypy` on changed files
@@ -0,0 +1,8 @@
# Tasks: fix false-positive running status
- [x] Investigate status flow and identify root cause
- [x] Add final container status check before writing "running"
- [x] Treat restarting as error when DB status is already running
- [x] Disable auto-restart in manifest compiler, legacy path, and seed data
- [ ] Run quality gates
- [ ] Commit and push