diff --git a/.pi-map.index.md b/.pi-map.index.md index b8ab15b..478b78d 100644 --- a/.pi-map.index.md +++ b/.pi-map.index.md @@ -16,7 +16,7 @@ dir: . Trust boundary: index routes, map orients, source decides. ## role -Infrastructure and deployment configuration for a self-hosted project management platform with OAuth2 authentication, providing Docker orchestration, environment setup, and development tooling. +A self-hosted web-based project management and Git repository platform with OAuth2 authentication, designed for deployment via Docker Compose with Traefik reverse proxy. ## parent - ## children diff --git a/.pi-map.md b/.pi-map.md index 4b7a2b2..1ed326a 100644 --- a/.pi-map.md +++ b/.pi-map.md @@ -18,7 +18,7 @@ index: ./.pi-map.index.md Trust boundary: index routes, map orients, source decides. ## role -Infrastructure and deployment configuration for a self-hosted project management platform with OAuth2 authentication, providing Docker orchestration, environment setup, and development tooling. +A self-hosted web-based project management and Git repository platform with OAuth2/SSO authentication, containerized for Docker deployment. ## files - .env.example | Provides a template of environment variables for configuring a Headquarter application with PostgreSQL, Redis, Authentik SSO, and Docker/Traefik deployment - .gitignore | Specifies files and directories for Git to ignore across a multi-language project with Python, Node, and custom tooling | dep: Git @@ -31,7 +31,7 @@ Infrastructure and deployment configuration for a self-hosted project management - progress.md | Tracks completed and remaining tasks for a backend-frontend code refactoring project organized in 7 phases - swap-pane | Empty file with no functionality ## arch -Containerized microservices architecture using Docker Compose with PostgreSQL/Redis backend services, Traefik reverse proxy with TLS termination, environment-based configuration, and Makefile-driven development lifecycle management. +Multi-service containerized architecture (PostgreSQL, Redis, API backend, web frontend) with reverse proxy integration (Traefik), environment-driven configuration, and AI-agent-guided development workflows via OpenSpec. ## tags docker, redis, git, application, postgresql, compose, traefik, project ## symbols diff --git a/apps/.pi-map.index.md b/apps/.pi-map.index.md index 3f1ff04..f19c62d 100644 --- a/apps/.pi-map.index.md +++ b/apps/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps ## role -Contains the deployable application entry points and executable configurations for the project. +Contains the main application entry points and top-level configurations for different deployable targets. ## parent index: ./.pi-map.index.md map: ./.pi-map.md diff --git a/apps/.pi-map.md b/apps/.pi-map.md index 9765054..71fa511 100644 --- a/apps/.pi-map.md +++ b/apps/.pi-map.md @@ -4,10 +4,10 @@ dir: apps index: apps/.pi-map.index.md ## role -Contains the deployable application entry points and executable configurations for the project. +Contains the main deployable applications or entry points for the project. ## files ## arch -Modular application structure with separate, self-contained app modules that compose shared libraries into runnable binaries. +Typically follows a multi-app workspace pattern where each subdirectory is an independent executable or service sharing common libraries. ## tags - ## symbols diff --git a/apps/api/.pi-map.index.md b/apps/api/.pi-map.index.md index 52461c9..75ac041 100644 --- a/apps/api/.pi-map.index.md +++ b/apps/api/.pi-map.index.md @@ -2,17 +2,23 @@ dir: apps/api ## role -FastAPI backend API that manages projects, git repositories, and development tools via Docker instances. +FastAPI backend API that manages projects, git repositories, and development tools through Docker containers with PostgreSQL persistence. ## parent index: apps/.pi-map.index.md map: apps/.pi-map.md ## children +- apps/api/.mypy_cache + index: apps/api/.mypy_cache/.pi-map.index.md + map: apps/api/.mypy_cache/.pi-map.md - apps/api/.pi-lens index: apps/api/.pi-lens/.pi-map.index.md map: apps/api/.pi-lens/.pi-map.md - apps/api/.pytest_cache index: apps/api/.pytest_cache/.pi-map.index.md map: apps/api/.pytest_cache/.pi-map.md +- apps/api/.ruff_cache + index: apps/api/.ruff_cache/.pi-map.index.md + map: apps/api/.ruff_cache/.pi-map.md - apps/api/.venv-test index: apps/api/.venv-test/.pi-map.index.md map: apps/api/.venv-test/.pi-map.md diff --git a/apps/api/.pi-map.md b/apps/api/.pi-map.md index a742201..7b6202c 100644 --- a/apps/api/.pi-map.md +++ b/apps/api/.pi-map.md @@ -4,7 +4,7 @@ dir: apps/api index: apps/api/.pi-map.index.md ## role -FastAPI backend API that manages projects, git repositories, and development tools via Docker instances. +Self-hosted FastAPI backend API that manages projects, git repositories, and development tools via Docker instances. ## files - .dockerignore | Specifies files and directories to exclude from Docker build context to reduce image size and avoid copying unnecessary files into containers. | dep: Docker - Dockerfile | Multi-stage Docker build for a Python application with Docker socket access, Cloudflare tunneling, and database dependency waiting | dep: python:3.11-slim, gcc, libpq-dev, docker-ce-cli, docker-compose-plugin, cloudflared, uvicorn, pyproject.toml dependencies @@ -14,7 +14,7 @@ FastAPI backend API that manages projects, git repositories, and development too - uv.lock | Lock file for the uv Python package manager that pins exact dependency versions and their artifact hashes for reproducible installations | dep: uv, Python 3.11+, aiosqlite, alembic, annotated-doc, annotated-types, anyio, ast-serialize, asyncpg, and many other PyPI packages - wait-for-db.sh | Wait for a PostgreSQL database to become available before executing a command, with configurable retry logic. | dep: nc (netcat), sh (POSIX shell), sleep ## arch -Async FastAPI with PostgreSQL (Alembic migrations), multi-stage Docker containerization with Docker socket access and Cloudflare tunneling, uv package management. +Multi-stage containerized architecture with async PostgreSQL database, Alembic migrations, Cloudflare tunneling, Docker socket access, and uv-based Python dependency management. ## tags docker, alembic, python, database, fastapi, postgresql, asyncpg, uvicorn ## symbols diff --git a/apps/api/alembic/.pi-map.index.md b/apps/api/alembic/.pi-map.index.md index 13b0cf5..be6472d 100644 --- a/apps/api/alembic/.pi-map.index.md +++ b/apps/api/alembic/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/alembic ## role -Database migration infrastructure for the API application, providing version-controlled schema evolution with async SQLAlchemy support. +Database migration infrastructure for the API application, enabling version-controlled schema changes with async SQLAlchemy support. ## parent index: apps/api/.pi-map.index.md map: apps/api/.pi-map.md diff --git a/apps/api/alembic/.pi-map.md b/apps/api/alembic/.pi-map.md index b5ef2d9..06d29fb 100644 --- a/apps/api/alembic/.pi-map.md +++ b/apps/api/alembic/.pi-map.md @@ -4,12 +4,12 @@ dir: apps/api/alembic index: apps/api/alembic/.pi-map.index.md ## role -Database migration infrastructure for the API application, providing version-controlled schema evolution with async SQLAlchemy support. +Database migration infrastructure for the API application, enabling version-controlled schema changes with async SQLAlchemy support. ## files - env.py | Configures Alembic database migration environment with async SQLAlchemy support for a project. | exp: func:run_migrations_offline() → None, call:context.configure, call:context.begin_transaction, call:context.run_migrations, func:do_run_migrations(connection: Connection) → None, call:context.configure, call:context.begin_transaction, call:context.run_migrations, func:run_async_migrations() → None, call:async_engine_from_config, call:config.get_section, call:connectable.connect, call:connection.run_sync, call:connectable.dispose, func:run_migrations_online() → None, call:asyncio.run, call:run_async_migrations | dep: logging.config, alembic, sqlalchemy, sqlalchemy.engine, sqlalchemy.ext.asyncio, src.config, src.models, asyncio - script.py.mako | Alembic database migration script template that generates upgrade/downgrade functions for SQLAlchemy schema migrations | dep: alembic, sqlalchemy ## arch -Standard Alembic migration framework with async SQLAlchemy integration, using Mako templating for migration script generation and environment-based configuration for database connection management. +Alembic migration framework with Mako templating for script generation, async SQLAlchemy engine configuration, and environment-based database connection management. ## tags migrations, run, sqlalchemy, async, alembic, call:context.configure, call:context.begin, transaction ## symbols diff --git a/apps/api/alembic/versions/.pi-map.index.md b/apps/api/alembic/versions/.pi-map.index.md index b157d86..da46b43 100644 --- a/apps/api/alembic/versions/.pi-map.index.md +++ b/apps/api/alembic/versions/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/alembic/versions ## role -Contains sequential Alembic database migration scripts that evolve the API's relational schema over time, tracking structural changes from initial tables through complex features like tool instances, config profiles, workspaces, manifests, and monitoring. +Database schema evolution and versioning management for the API application using Alembic migration scripts. ## parent index: apps/api/alembic/.pi-map.index.md map: apps/api/alembic/.pi-map.md @@ -48,6 +48,7 @@ map: apps/api/alembic/.pi-map.md - 2026_05_29_remove_ssh_keys_mount_from_manifest.py - 2026_06_01_add_workspaces.py - 2026_06_13_make_clone_mode_nullable.py +- 2026_06_14_104415_add_tool_type_home_directory.py - 398082499c30_add_tool_config_fields.py - 6fc7bfcf199f_merge_remove_is_builtin_and_add_config_.py - 86cec91fdb00_merge_profile_resolver_and_workspaces_.py @@ -55,6 +56,7 @@ map: apps/api/alembic/.pi-map.md - 8ed7dd80973d_create_config_folders_table.py - af8512103d67_add_tool_type_fields.py - f3d2dc90ba3a_merge_single_interface_and_clone_mode.py +- fc8f1a20cbf6_merge_home_directory_and_pi_agent_mount_.py ## links index: apps/api/alembic/versions/.pi-map.index.md map: apps/api/alembic/versions/.pi-map.md diff --git a/apps/api/alembic/versions/.pi-map.md b/apps/api/alembic/versions/.pi-map.md index e7017cf..60c928c 100644 --- a/apps/api/alembic/versions/.pi-map.md +++ b/apps/api/alembic/versions/.pi-map.md @@ -4,9 +4,9 @@ dir: apps/api/alembic/versions index: apps/api/alembic/versions/.pi-map.index.md ## role -Contains sequential Alembic database migration scripts that evolve the API's relational schema over time, tracking structural changes from initial tables through complex features like tool instances, config profiles, workspaces, manifests, and monitoring. +Database schema evolution and versioning management for the API application using Alembic migration scripts. ## files -- 0001_initial_schema.py | Defines the initial database schema migration creating five tables (users, ssh_keys, projects, git_repositories, user_configs) with relationships, indexes, and constraints using Alembic. | dep: sqlalchemy, alembic, postgresql dialect +- 0001_initial_schema.py | Defines the initial database schema migration creating five tables (users, ssh_keys, projects, git_repositories, user_configs) with relationships, indexes, and constraints using Alembic. | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.String, call:postgresql.UUID, call:sa.DateTime, call:sa.func.now, call:sa.PrimaryKeyConstraint, call:sa.UniqueConstraint, call:op.create_index, call:op.f, call:sa.Text, call:sa.ForeignKeyConstraint, call:sa.Boolean, call:postgresql.JSONB, func:downgrade() → None, call:op.drop_table, call:op.drop_index, call:op.f | dep: alembic, sqlalchemy.dialects, sqlalchemy, postgresql dialect - 0002_refresh_tokens.py | Alembic database migration that creates a refresh_tokens table with indexes for user authentication token management | exp: func:upgrade() → None, call:op.get_bind, call:sa.inspect, call:inspector.has_table, call:op.create_table, call:sa.Column, call:postgresql.UUID, call:sa.String, call:sa.DateTime, call:sa.ForeignKeyConstraint, call:sa.PrimaryKeyConstraint, call:sa.UniqueConstraint, call:inspector.get_indexes, call:op.f, call:op.create_index, func:downgrade() → None, call:op.get_bind, call:sa.inspect, call:inspector.has_table, call:inspector.get_indexes, call:op.f, call:op.drop_index, call:op.drop_table | dep: alembic, sqlalchemy.dialects, sqlalchemy, sqlalchemy.dialects.postgresql - 0003_user_configs.py | Alembic database migration that creates a user_configs table with JSON configuration storage linked to users | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.UUID, call:sa.JSON, call:sa.DateTime, call:sa.text, call:sa.ForeignKeyConstraint, call:sa.PrimaryKeyConstraint, call:sa.UniqueConstraint, func:downgrade() → None, call:op.drop_table | dep: typing, alembic, sqlalchemy - 0004_tool_types.py | Alembic database migration that creates a tool_types table with metadata, templates, and versioning columns for a tool management system. | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.Uuid, call:sa.String, call:sa.Text, call:sa.JSON, call:sa.Boolean, call:sa.ForeignKey, call:sa.DateTime, call:sa.text, func:downgrade() → None, call:op.drop_table | dep: typing, alembic, sqlalchemy @@ -33,7 +33,7 @@ Contains sequential Alembic database migration scripts that evolve the API's rel - 2026_05_27_external_repos.py | Alembic database migration that makes project_id nullable in git_repositories table to support external repositories and expands alembic_version version_num column to 64 characters. | exp: func:upgrade() → None, call:op.execute, call:op.alter_column, call:sa.UUID, func:downgrade() → None, call:op.alter_column, call:sa.UUID, call:op.execute | dep: typing, alembic, sqlalchemy - 2026_05_28_add_monitoring_tables.py | Alembic database migration that creates monitoring tables (instance_events and health_checks) with indexes for tracking tool instance events and health checks | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.Uuid, call:sa.String, call:sa.Text, call:sa.JSON, call:sa.DateTime, call:sa.func.now, call:sa.ForeignKeyConstraint, call:sa.PrimaryKeyConstraint, call:op.create_index, call:sa.Boolean, call:sa.Integer, func:downgrade() → None, call:op.drop_index, call:op.drop_table | dep: collections.abc, alembic, sqlalchemy - 2026_05_28_add_terminal_sessions_table.py | Alembic database migration that creates a terminal_sessions table with tracking columns and foreign key to tool_instances | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.UUID, call:sa.String, call:sa.DateTime, call:sa.text, call:sa.ForeignKeyConstraint, call:sa.PrimaryKeyConstraint, call:op.create_index, call:op.f, func:downgrade() → None, call:op.drop_index, call:op.f, call:op.drop_table | dep: collections.abc, alembic, sqlalchemy -- 2026_05_28_add_tool_definition_manifests.py | Alembic database migration that creates a tool_definition_manifests table, adds manifest-related columns to tool_types and tool_instances, and migrates the pi-agent tool from legacy dockerfile/compose templates to a JSON-based manifest system with a base Ubuntu image definition. | exp: func:upgrade() → None, call:op.get_bind, call:op.create_table, call:sa.Column, call:sa.UUID, call:sa.String, call:sa.Text, call:sa.JSON, call:sa.Boolean, call:sa.TIMESTAMP, call:sa.func.now, call:sa.PrimaryKeyConstraint, call:sa.UniqueConstraint, call:sa.ForeignKeyConstraint, call:sa.CheckConstraint, call:conn.execute, call:sa.text, call:result.fetchone, call:op.add_column, call:op.create_foreign_key, call:op.drop_constraint, call:op.execute, call:json.dumps, call:str, func:downgrade() → None, call:op.get_bind, call:conn.execute, call:sa.text, call:result.fetchone, call:op.drop_column, call:op.drop_constraint, call:op.drop_table | dep: json, uuid, typing, alembic, sqlalchemy +- 2026_05_28_add_tool_definition_manifests.py | Alembic database migration that creates a tool_definition_manifests table, adds manifest support to existing tool_types and tool_instances tables, and seeds initial data with a base Ubuntu image and pi-agent manifest while migrating the legacy pi-agent from Dockerfile templates to the new manifest system. | exp: func:upgrade() → None, call:op.get_bind, call:op.create_table, call:sa.Column, call:sa.UUID, call:sa.String, call:sa.Text, call:sa.JSON, call:sa.Boolean, call:sa.TIMESTAMP, call:sa.func.now, call:sa.PrimaryKeyConstraint, call:sa.UniqueConstraint, call:sa.ForeignKeyConstraint, call:sa.CheckConstraint, call:conn.execute, call:sa.text, call:result.fetchone, call:op.add_column, call:op.create_foreign_key, call:op.drop_constraint, call:op.execute, call:json.dumps, call:str, func:downgrade() → None, call:op.get_bind, call:conn.execute, call:sa.text, call:result.fetchone, call:op.drop_column, call:op.drop_constraint, call:op.drop_table | dep: json, uuid, typing, alembic, sqlalchemy - 2026_05_28_drop_tool_configs_and_config_folders.py | Alembic database migration that drops `tool_configs` and `config_folders` tables with conditional existence checks and full downgrade recreation | exp: func:upgrade() → None, call:op.get_bind, call:conn.execute, call:sa.text, call:result.fetchone, call:op.drop_table, func:downgrade() → None, call:op.create_table, call:sa.Column, call:sa.UUID, call:sa.String, call:sa.Text, call:sa.JSON, call:sa.Boolean, call:sa.TIMESTAMP, call:sa.func.now, call:sa.PrimaryKeyConstraint, call:sa.Integer | dep: typing, alembic, sqlalchemy - 2026_05_29_add_notifications_table.py | Alembic database migration that creates a notifications table with user-linked, categorized, severity-graded messages supporting read/dismissed tracking and optimized querying indexes. | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.Uuid, call:sa.String, call:sa.Text, call:sa.JSON, call:sa.DateTime, call:sa.func.now, call:sa.ForeignKeyConstraint, call:sa.PrimaryKeyConstraint, call:op.create_index, call:sa.text, func:downgrade() → None, call:op.drop_index, call:op.drop_table | dep: collections.abc, alembic, sqlalchemy - 2026_05_29_add_ssh_key_ids_to_tool_instances.py | Alembic database migration that adds a JSON column named ssh_key_ids to the tool_instances table | exp: func:upgrade() → None, call:op.add_column, call:sa.Column, call:sa.JSON, func:downgrade() → None, call:op.drop_column | dep: alembic, sqlalchemy @@ -45,6 +45,7 @@ Contains sequential Alembic database migration scripts that evolve the API's rel - 2026_05_29_remove_ssh_keys_mount_from_manifest.py | Alembic database migration that removes (or restores) the ssh_keys mount from a JSON manifest stored in the tool_definition_manifests table for the pi-agent tool definition. | exp: func:upgrade() → None, call:op.get_bind, call:conn.execute, call:sa.text, call:result.fetchone, call:isinstance, call:json.loads, call:manifest.get, call:len, call:m.get, call:json.dumps, func:downgrade() → None, call:op.get_bind, call:conn.execute, call:sa.text, call:result.fetchone, call:isinstance, call:json.loads, call:manifest.get, call:any, call:m.get, call:mounts.append, call:json.dumps | dep: json, typing, alembic, sqlalchemy - 2026_06_01_add_workspaces.py | Alembic database migration that creates a workspaces table with foreign keys to git_repositories and users, adds indexes, and adds a workspace_id column to tool_instances | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:sa.Uuid, call:sa.String, call:sa.ForeignKey, call:sa.DateTime, call:sa.text, call:sa.UniqueConstraint, call:op.create_index, call:op.add_column, func:downgrade() → None, call:op.drop_index, call:op.drop_column, call:op.drop_table | dep: collections.abc, alembic, sqlalchemy - 2026_06_13_make_clone_mode_nullable.py | Alembic database migration that makes the `clone_mode` column in `tool_instances` table nullable to support workspace-first cleanup workflow. | exp: func:upgrade() → None, call:op.alter_column, call:sa.String, func:downgrade() → None, call:op.alter_column, call:sa.String | dep: alembic, sqlalchemy +- 2026_06_14_104415_add_tool_type_home_directory.py | Alembic database migration that adds a `home_directory` column to `tool_types` table and updates template strings to use a configurable workspace path instead of hardcoded `/workspace` | exp: func:upgrade() → None, call:op.add_column, call:sa.Column, call:sa.String, call:op.execute, call:sa.update(tool_types) .where(tool_types.c.compose_template.is_not(None)) .values, call:tool_types.c.compose_template.is_not, call:sa.func.replace, call:sa.update(tool_types) .where(tool_types.c.dockerfile_template.is_not(None)) .values, call:tool_types.c.dockerfile_template.is_not, func:downgrade() → None, call:op.execute, call:sa.update(tool_types) .where(tool_types.c.compose_template.is_not(None)) .values, call:tool_types.c.compose_template.is_not, call:sa.func.replace, call:sa.update(tool_types) .where(tool_types.c.dockerfile_template.is_not(None)) .values, call:tool_types.c.dockerfile_template.is_not, call:op.drop_column | dep: typing, alembic, sqlalchemy.sql, sqlalchemy - 398082499c30_add_tool_config_fields.py | Alembic database migration that adds five new columns (port_override, start_command, working_directory, environment_variables, volumes) to the tool_configs table with a port range check constraint. | exp: func:upgrade() → None, call:op.add_column, call:sa.Column, call:sa.Integer, call:sa.Text, call:postgresql.JSONB, call:op.create_check_constraint, call:sa.text, func:downgrade() → None, call:op.drop_constraint, call:op.drop_column | dep: alembic, sqlalchemy.dialects, sqlalchemy, sqlalchemy.dialects.postgresql - 6fc7bfcf199f_merge_remove_is_builtin_and_add_config_.py | Alembic database migration that merges two parallel revision branches (removing is_builtin and adding config_profiles) into a single history line | exp: func:upgrade() → None, func:downgrade() → None | dep: alembic - 86cec91fdb00_merge_profile_resolver_and_workspaces_.py | Alembic database migration that merges two divergent migration branches (profile resolver and workspaces) into a single head | exp: func:upgrade() → None, func:downgrade() → None | dep: alembic @@ -52,8 +53,9 @@ Contains sequential Alembic database migration scripts that evolve the API's rel - 8ed7dd80973d_create_config_folders_table.py | Alembic database migration that creates a config_folders table with user-owned configuration folders supporting JSONB file storage and project overrides | exp: func:upgrade() → None, call:op.create_table, call:sa.Column, call:postgresql.UUID, call:sa.text, call:sa.ForeignKey, call:sa.String, call:sa.Text, call:postgresql.JSONB, call:sa.Boolean, call:sa.DateTime, call:sa.UniqueConstraint, call:op.create_index, func:downgrade() → None, call:op.drop_index, call:op.drop_table | dep: alembic, sqlalchemy.dialects, sqlalchemy, sqlalchemy.dialects.postgresql - af8512103d67_add_tool_type_fields.py | Alembic database migration that adds new columns (definition_type, dockerfile_template, build_context, readiness_probe) to the tool_types table with a CHECK constraint on definition_type. | exp: func:upgrade() → None, call:op.add_column, call:sa.Column, call:sa.String, call:sa.Text, call:postgresql.JSONB, call:op.create_check_constraint, call:sa.text, func:downgrade() → None, call:op.drop_constraint, call:op.drop_column | dep: alembic, sqlalchemy.dialects, sqlalchemy, sqlalchemy.dialects.postgresql - f3d2dc90ba3a_merge_single_interface_and_clone_mode.py | Alembic database migration that merges two prior revisions (single_interface and clone_mode) into a single migration path | exp: func:upgrade() → None, func:downgrade() → None | dep: typing, alembic +- fc8f1a20cbf6_merge_home_directory_and_pi_agent_mount_.py | Alembic database migration that merges two revision branches by declaring them as down revisions without performing any schema changes. | exp: func:upgrade() → None, func:downgrade() → None | dep: alembic, sqlalchemy ## arch -Linear migration history with periodic merge heads to reconcile divergent branches; each migration is self-contained with upgrade/downgrade operations, using declarative SQLAlchemy operations, conditional checks for idempotency, and occasional data migrations alongside schema changes. +Sequential and branched migration pattern with merge resolution, using Alembic's revision-based approach with upgrade/downgrade functions, including data migrations, conditional schema changes, and cross-dialect support (PostgreSQL/SQLite). ## tags column, table, call:op.drop, alembic, downgrade, upgrade, key, call:sa.text ## symbols diff --git a/apps/api/alembic/versions/2026_06_14_104415_add_tool_type_home_directory.py b/apps/api/alembic/versions/2026_06_14_104415_add_tool_type_home_directory.py new file mode 100644 index 0000000..b84639f --- /dev/null +++ b/apps/api/alembic/versions/2026_06_14_104415_add_tool_type_home_directory.py @@ -0,0 +1,89 @@ +"""add_tool_type_home_directory + +Revision ID: 2026_06_14_104415 +Revises: f3d2dc90ba3a +Create Date: 2026-06-14 10:44:15.000000 + +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from alembic import op +from sqlalchemy.sql import column, table + +# revision identifiers, used by Alembic. +revision: str = "2026_06_14_104415" +down_revision: Union[str, Sequence[str], None] = "f3d2dc90ba3a" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +tool_types = table( + "tool_types", + column("id", sa.UUID), + column("home_directory", sa.String), + column("compose_template", sa.Text), + column("dockerfile_template", sa.Text), +) + +OLD_WORKSPACE = "/workspace" +NEW_WORKSPACE = "/home/user/{{WORKSPACE_NAME}}" + + +def upgrade() -> None: + # Add the new column with a default that applies to existing rows. + op.add_column( + "tool_types", + sa.Column( + "home_directory", + sa.String(255), + nullable=False, + server_default="/home/user", + ), + ) + + # Rewrite legacy templates that mount the workspace at /workspace so they + # use the new configurable home directory and preserve the workspace name. + op.execute( + sa.update(tool_types) + .where(tool_types.c.compose_template.is_not(None)) + .values( + compose_template=sa.func.replace( + tool_types.c.compose_template, OLD_WORKSPACE, NEW_WORKSPACE + ) + ) + ) + op.execute( + sa.update(tool_types) + .where(tool_types.c.dockerfile_template.is_not(None)) + .values( + dockerfile_template=sa.func.replace( + tool_types.c.dockerfile_template, OLD_WORKSPACE, NEW_WORKSPACE + ) + ) + ) + + +def downgrade() -> None: + # Restore the original /workspace strings before dropping the column. + op.execute( + sa.update(tool_types) + .where(tool_types.c.compose_template.is_not(None)) + .values( + compose_template=sa.func.replace( + tool_types.c.compose_template, NEW_WORKSPACE, OLD_WORKSPACE + ) + ) + ) + op.execute( + sa.update(tool_types) + .where(tool_types.c.dockerfile_template.is_not(None)) + .values( + dockerfile_template=sa.func.replace( + tool_types.c.dockerfile_template, NEW_WORKSPACE, OLD_WORKSPACE + ) + ) + ) + + op.drop_column("tool_types", "home_directory") diff --git a/apps/api/alembic/versions/fc8f1a20cbf6_merge_home_directory_and_pi_agent_mount_.py b/apps/api/alembic/versions/fc8f1a20cbf6_merge_home_directory_and_pi_agent_mount_.py new file mode 100644 index 0000000..6b73e95 --- /dev/null +++ b/apps/api/alembic/versions/fc8f1a20cbf6_merge_home_directory_and_pi_agent_mount_.py @@ -0,0 +1,23 @@ +"""merge home directory and pi agent mount cleanup heads + +Revision ID: fc8f1a20cbf6 +Revises: 2026_06_14_104415, 8c6d1dbd4798 +Create Date: 2026-06-14 11:08:41.273502 +""" + + + + +# revision identifiers, used by Alembic. +revision = 'fc8f1a20cbf6' +down_revision = ('2026_06_14_104415', '8c6d1dbd4798') +branch_labels = None +depends_on = None + + +def upgrade() -> None: + pass + + +def downgrade() -> None: + pass diff --git a/apps/api/src/.pi-map.index.md b/apps/api/src/.pi-map.index.md index 6048666..69ebca6 100644 --- a/apps/api/src/.pi-map.index.md +++ b/apps/api/src/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src ## role -Core API application package that initializes and configures the Headquarter FastAPI backend with database, authentication, logging, and middleware infrastructure. +Core application bootstrap and infrastructure layer for the Headquarter API, providing configuration, database connectivity, logging, and FastAPI application initialization. ## parent index: apps/api/.pi-map.index.md map: apps/api/.pi-map.md diff --git a/apps/api/src/.pi-map.md b/apps/api/src/.pi-map.md index 7c8205c..5b786aa 100644 --- a/apps/api/src/.pi-map.md +++ b/apps/api/src/.pi-map.md @@ -4,7 +4,7 @@ dir: apps/api/src index: apps/api/src/.pi-map.index.md ## role -Core API application package that initializes and configures the Headquarter FastAPI backend with database, authentication, logging, and middleware infrastructure. +Core API application package for the Headquarter service, providing configuration, database connectivity, structured logging, and FastAPI application initialization. ## files - __init__.py | Marks the directory as a Python package for the Headquarter API. - config.py | Defines application configuration settings with environment-based overrides using Pydantic, including database URLs, service domains, OAuth/Authentik integration, JWT/session settings, and computed properties for environment-specific behavior. | exp: class:Settings, func:build_database_url(user: str, password: str, host: str, port: int, database: str) → str | dep: pydantic, pydantic_settings @@ -12,7 +12,7 @@ Core API application package that initializes and configures the Headquarter Fas - logging_config.py | Configures structured JSON logging with correlation ID injection, custom formatters, and HTTP request/exception middleware for a FastAPI application. | exp: class:CorrelationIdFilter, method:filter(self, record: logging.LogRecord) → bool, call:get_correlation_id, class:JSONFormatter, method:format(self, record: logging.LogRecord) → str, call:self.formatTime, call:record.getMessage, call:getattr, call:self.formatException, call:json.dumps, method:formatTime(self, record: logging.LogRecord, datefmt) → str, call:time.strftime, call:time.gmtime, class:RequestLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:time.time, call:logger.info, call:call_next, call:int, call:logger.error, call:type, call:traceback.format_exc, class:ExceptionLoggingMiddleware, method:dispatch(self, request: Request, call_next: Callable) → Response, call:call_next, call:logger.critical, call:traceback.format_exc, func:configure_logging(level) → None, call:JSONFormatter, call:logging.StreamHandler, call:console_handler.setFormatter, call:console_handler.addFilter, call:CorrelationIdFilter, call:root_logger.setLevel, call:logging.getLogger("uvicorn").setLevel, call:logging.getLogger("uvicorn.access").setLevel, call:logging.getLogger("sqlalchemy.engine").setLevel, call:logger.info, call:logging.getLevelName | dep: json, logging, sys, time, traceback, collections.abc, fastapi, starlette.middleware.base, src.services.shared.correlation - main.py | Initializes and configures a FastAPI application for the "Headquarter API" with database setup, middleware, routing, and background services. | exp: func:_sanitize_validation_errors(errors), call:error.get, call:str, call:ctx.items, call:isinstance, call:type, call:sanitized.append, func:validation_exception_handler(request: Request, exc: RequestValidationError), call:exc.errors, call:logger.warning, call:_sanitize_validation_errors, call:JSONResponse, func:on_startup(), call:logger.info, call:init_database, call:logger.error, call:sys.exit, call:_health_monitor.start, call:seed_builtin_tool_types, func:on_shutdown(), call:logger.info, call:_health_monitor.stop | dep: logging, os, fastapi, fastapi.exceptions, fastapi.middleware.cors, fastapi.responses, fastapi.staticfiles, src.api.config, src.api.project, src.api.system, src.api.tool, src.api.user, src.api.workspace, src.config, src.models, src.database, src.logging_config, src.seeds.builtin_tool_types, src.services.instance, src.services.shared, sys, src.api.* ## arch -Layered architecture using Pydantic for environment-based configuration, async SQLAlchemy with Alembic migrations, structured JSON logging with correlation IDs, and FastAPI middleware/routing pattern for a service-oriented backend. +Layered architecture with environment-based Pydantic configuration, async SQLAlchemy with Alembic migrations, structured JSON logging with correlation IDs, and FastAPI middleware/routing pattern with background services integration. ## tags src, database, logging, call:logger.info, api, middleware, fastapi, filter ## symbols diff --git a/apps/api/src/models/.pi-map.index.md b/apps/api/src/models/.pi-map.index.md index 405e810..ee42ab8 100644 --- a/apps/api/src/models/.pi-map.index.md +++ b/apps/api/src/models/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/models ## role -Centralized database model definitions and shared infrastructure for the API's data layer. +Provides the SQLAlchemy ORM data models and database schema definitions for the API application. ## parent index: apps/api/src/.pi-map.index.md map: apps/api/src/.pi-map.md diff --git a/apps/api/src/models/.pi-map.md b/apps/api/src/models/.pi-map.md index 91e73b5..272caa0 100644 --- a/apps/api/src/models/.pi-map.md +++ b/apps/api/src/models/.pi-map.md @@ -4,12 +4,12 @@ dir: apps/api/src/models index: apps/api/src/models/.pi-map.index.md ## role -Centralized database model definitions and shared infrastructure for the API's data layer. +Provides the SQLAlchemy ORM data models and database schema definitions for the API application. ## files - __init__.py | Re-exports model classes from submodules to provide a centralized public API for the src.models package | dep: src.models.base, src.models.config.config_profile, src.models.project.git_repository, src.models.project.project, src.models.project.workspace, src.models.system.health_check, src.models.system.instance_event, src.models.system.notification, src.models.system.terminal_session, src.models.tool.tool_definition_manifest, src.models.tool.tool_instance, src.models.tool.tool_type, src.models.user.ssh_key, src.models.user.user, src.models.user.user_config - base.py | Defines SQLAlchemy base model and reusable mixins for UUID primary keys and automatic timestamp tracking in database models. | exp: class:Base, class:UUIDPrimaryKeyMixin, class:TimestampMixin | dep: uuid, datetime, sqlalchemy, sqlalchemy.orm ## arch -SQLAlchemy ORM with declarative base, mixin-based composition for cross-cutting concerns (UUIDs, timestamps), and explicit package-level re-exports for clean public API surface. +Layered repository pattern with declarative SQLAlchemy base, UUID/timestamp mixins for reusable model traits, and package-level facade pattern via __init__.py re-exports to centralize model access. ## tags models, src, base, project, system, user, mixin, tool ## symbols diff --git a/apps/api/src/models/tool/.pi-map.index.md b/apps/api/src/models/tool/.pi-map.index.md index 59ac10f..2aa7f20 100644 --- a/apps/api/src/models/tool/.pi-map.index.md +++ b/apps/api/src/models/tool/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/models/tool ## role -Database models for containerized tool lifecycle management, covering tool definitions, deployment instances, and categorization types. +Provides SQLAlchemy ORM models for managing containerized tool definitions, types, and deployed instances in the API. ## parent index: apps/api/src/models/.pi-map.index.md map: apps/api/src/models/.pi-map.md diff --git a/apps/api/src/models/tool/.pi-map.md b/apps/api/src/models/tool/.pi-map.md index 0244275..079fe79 100644 --- a/apps/api/src/models/tool/.pi-map.md +++ b/apps/api/src/models/tool/.pi-map.md @@ -4,16 +4,16 @@ dir: apps/api/src/models/tool index: apps/api/src/models/tool/.pi-map.index.md ## role -Database models for containerized tool lifecycle management, covering tool definitions, deployment instances, and categorization types. +Provides SQLAlchemy ORM models for managing containerized tool definitions, types, and deployed instances in the API. ## files - __init__.py | Exports the public API for the tool models module by re-exporting three key classes. | dep: src.models.tool.tool_definition_manifest, src.models.tool.tool_instance, src.models.tool.tool_type - tool_definition_manifest.py | Defines a SQLAlchemy ORM model for storing tool definition manifests that compile to Dockerfiles and Compose files, supporting both base definitions and tool-specific definitions with inheritance. | exp: class:ToolDefinitionManifest | dep: uuid, typing, sqlalchemy, sqlalchemy.orm, src.models.base, src.models.user - tool_instance.py | Defines a SQLAlchemy ORM model for tool instances that represent deployed tools with container metadata, status tracking, and relationships to users, projects, workspaces, and other entities. | exp: class:ToolInstance | dep: uuid, datetime, typing, sqlalchemy, sqlalchemy.orm, src.models.base, src.models, src.models.project, src.models.user, src.models (ConfigProfile, GitRepository, Project, ToolType, User, Workspace) -- tool_type.py | Defines a SQLAlchemy ORM model for tool types that represent configurable categories of tools with deployment templates, manifest references, and metadata. | exp: class:ToolType | dep: uuid, typing, sqlalchemy, sqlalchemy.orm, src.models.base, src.models.tool.tool_definition_manifest, src.models.user +- tool_type.py | Defines a SQLAlchemy ORM model for tool types that specify configuration templates and metadata for deployable tools in a containerized environment. | exp: class:ToolType | dep: uuid, typing, sqlalchemy, sqlalchemy.orm, src.models.base, src.models.tool.tool_definition_manifest, src.models.user ## arch -SQLAlchemy ORM with declarative models using inheritance hierarchies, relationship mappings, and polymorphic manifest compilation for Docker/Compose deployment. +Domain-driven data models using SQLAlchemy ORM with declarative base pattern, entity relationships, and inheritance support for tool manifest definitions. ## tags -tool, models, src, sqlalchemy, orm, definition, manifest, base +tool, models, src, sqlalchemy, orm, definition, base, manifest ## symbols - ToolDefinitionManifest - ToolInstance diff --git a/apps/api/src/models/tool/tool_type.py b/apps/api/src/models/tool/tool_type.py index aea5cb9..7948339 100644 --- a/apps/api/src/models/tool/tool_type.py +++ b/apps/api/src/models/tool/tool_type.py @@ -39,6 +39,9 @@ class ToolType(UUIDPrimaryKeyMixin, TimestampMixin, Base): ) readiness_probe: Mapped[dict | None] = mapped_column(JSON, nullable=True) startup_command: Mapped[str | None] = mapped_column(Text, nullable=True) + home_directory: Mapped[str] = mapped_column( + String(255), nullable=False, default="/home/user" + ) required_variables: Mapped[list[str]] = mapped_column( JSON, default=list, nullable=False ) diff --git a/apps/api/src/services/.pi-map.index.md b/apps/api/src/services/.pi-map.index.md index 6a4454b..5f5120a 100644 --- a/apps/api/src/services/.pi-map.index.md +++ b/apps/api/src/services/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/services ## role -Marks the services directory as a Python package for business logic layer components. +Marks the services directory as a Python package for organizing business logic and service-layer abstractions. ## parent index: apps/api/src/.pi-map.index.md map: apps/api/src/.pi-map.md diff --git a/apps/api/src/services/.pi-map.md b/apps/api/src/services/.pi-map.md index d7e5d8c..690d899 100644 --- a/apps/api/src/services/.pi-map.md +++ b/apps/api/src/services/.pi-map.md @@ -4,11 +4,11 @@ dir: apps/api/src/services index: apps/api/src/services/.pi-map.index.md ## role -Provides a Python package namespace for organizing service-layer modules in the API application. +This directory serves as a Python package namespace for organizing service-layer modules in the API application. ## files - __init__.py | Empty file with no functionality ## arch -Standard Python package structure using __init__.py for directory-based module organization, following conventional layered architecture patterns. +Standard Python package structure using __init__.py to define an importable module directory, following conventional layered architecture patterns. ## tags init, empty, functionality ## symbols diff --git a/apps/api/src/services/build/manifest_compiler.py b/apps/api/src/services/build/manifest_compiler.py index 343dcaa..28e85ae 100644 --- a/apps/api/src/services/build/manifest_compiler.py +++ b/apps/api/src/services/build/manifest_compiler.py @@ -8,6 +8,7 @@ from typing import Any import yaml +from src.services.config.config_profile_resolver import expand_container_path from src.services.docker import sort_volumes_by_specificity @@ -158,6 +159,8 @@ def compile_dockerfile(manifest: dict) -> str: # User creation user = manifest.get("user") + home_dir = get_manifest_home_dir(manifest) + workspace_name = manifest.get("workspace_name", "{{WORKSPACE_NAME}}") if user: name = user["name"] uid = user["uid"] @@ -168,22 +171,21 @@ def compile_dockerfile(manifest: dict) -> str: lines.append(f" useradd -u {uid} -g {gid} {create_home}-s {shell} {name}") lines.append("") # Set HOME and USER for runtime compatibility - home = f"/home/{name}" - lines.append(f"ENV HOME={home}") + lines.append(f"ENV HOME={home_dir}") lines.append(f"ENV USER={name}") lines.append("") # Ensure home directory exists and is writable by the user. # Recursively chown so any files copied from /etc/skel by useradd -m # (e.g. .bashrc, .config) are owned by the container user. lines.append( - f"RUN mkdir -p {home} && chown -R {name}:{name} {home} && chmod 755 {home}" + f"RUN mkdir -p {home_dir} && chown -R {name}:{name} {home_dir} && chmod 755 {home_dir}" ) # Pre-create common config directories so apps like ranger can write # their configs on first run without permission errors. common_dirs = [".config", ".local/share", ".cache"] for d in common_dirs: lines.append( - f"RUN mkdir -p {home}/{d} && chown -R {name}:{name} {home}/{d}" + f"RUN mkdir -p {home_dir}/{d} && chown -R {name}:{name} {home_dir}/{d}" ) lines.append("") @@ -208,10 +210,12 @@ def compile_dockerfile(manifest: dict) -> str: # After build scripts, ensure everything in home is owned by the user if user and build_scripts: - lines.append(f"RUN chown -R {name}:{name} {home}") + lines.append(f"RUN chown -R {name}:{name} {home_dir}") lines.append("") - # Create mount target directories + # Create mount target directories and /workspace compatibility symlink. + # The symlink target includes the workspace/repo name so legacy scripts + # that cd into /workspace still land on the right project. mounts = manifest.get("mounts", []) if mounts: dirs = [mount["target"] for mount in mounts] @@ -221,6 +225,16 @@ def compile_dockerfile(manifest: dict) -> str: lines.append(f"RUN chown -R {user['name']}:{user['name']} {dir_str}") lines.append("") + workspace_target = f"{home_dir}/{workspace_name}" + lines.append(f"RUN mkdir -p {workspace_target}") + if user: + lines.append( + f"RUN ln -sfn {workspace_target} /workspace && chown -R {user['name']}:{user['name']} {home_dir}" + ) + else: + lines.append(f"RUN ln -sfn {workspace_target} /workspace") + lines.append("") + # Entrypoint for startup scripts startup_scripts = manifest.get("scripts", {}).get("startup", []) if startup_scripts: @@ -233,11 +247,18 @@ def compile_dockerfile(manifest: dict) -> str: # Switch to runtime user if user: lines.append(f"USER {user['name']}") - lines.append(f"WORKDIR /home/{user['name']}") - lines.append("") + + # Set WORKDIR to the configured home directory unless runtime.working_dir + # explicitly overrides it. + runtime = manifest.get("runtime", {}) + working_dir = runtime.get("working_dir") + if working_dir: + lines.append(f"WORKDIR {expand_container_path(working_dir, home_dir)}") + else: + lines.append(f"WORKDIR {home_dir}") + lines.append("") # Entrypoint and CMD - runtime = manifest.get("runtime", {}) if startup_scripts: lines.append('ENTRYPOINT ["/usr/local/bin/headquarter-entrypoint"]') @@ -251,6 +272,11 @@ def compile_dockerfile(manifest: dict) -> str: def compile_entrypoint(manifest: dict) -> str: """Generate the startup entrypoint script from startup scripts. + Injects a permission-fixer preamble that runs as root (or via sudo) before + any user-defined startup script. It chowns the home directory and a safe + subset of mount parents to the container user, creates the /workspace + compatibility symlink, and avoids recursive chown of large repo subtrees. + Args: manifest: Fully resolved manifest JSON. @@ -259,6 +285,70 @@ def compile_entrypoint(manifest: dict) -> str: """ lines = ["#!/bin/bash", "set -e", ""] + user = manifest.get("user") + home_dir = get_manifest_home_dir(manifest) + workspace_name = manifest.get("workspace_name", "{{WORKSPACE_NAME}}") + workspace_target = f"{home_dir}/{workspace_name}" + + # Permission fixer preamble: run as root when possible, else fall back to + # passwordless sudo configured in the Dockerfile. + lines.append("# Permission fixer preamble") + lines.append("CONTAINER_USER=''") + lines.append('if [ "$(id -u)" = '"'"'0'"'"' ]; then') + if user: + lines.append(f" CONTAINER_USER='{user['name']}'") + lines.append("else") + lines.append(" # Try passwordless sudo; ignore failure so the container still starts") + lines.append(" if sudo -n true 2>/dev/null; then") + lines.append(" SUDO='sudo'") + lines.append(" else") + lines.append(" SUDO=''") + lines.append(" fi") + lines.append("fi") + lines.append("") + + if user: + name = user["name"] + uid = user["uid"] + gid = user["gid"] + lines.append(f"USER_NAME='{name}'") + lines.append(f"USER_UID='{uid}'") + lines.append(f"USER_GID='{gid}'") + lines.append(f"HOME_DIR='{home_dir}'") + lines.append(f"WORKSPACE_TARGET='{workspace_target}'") + lines.append("") + lines.append("fix_owner() {") + lines.append(" local path=\"$1\"") + lines.append(' [ -e "$path" ] || return 0') + lines.append(' if [ -n "$SUDO" ]; then') + lines.append(' sudo chown "$USER_UID:$USER_GID" "$path" 2>/dev/null || true') + lines.append(' elif [ "$(id -u)" = "0" ]; then') + lines.append(' chown "$USER_UID:$USER_GID" "$path" 2>/dev/null || true') + lines.append(' fi') + lines.append("}") + lines.append("") + lines.append("# Ensure home directory exists and is owned by the container user") + lines.append('mkdir -p "$HOME_DIR"') + lines.append('fix_owner "$HOME_DIR"') + lines.append("") + lines.append("# Ensure workspace target exists and is owned by the container user") + lines.append('mkdir -p "$WORKSPACE_TARGET"') + lines.append('fix_owner "$WORKSPACE_TARGET"') + lines.append("") + lines.append("# Create /workspace compatibility symlink") + lines.append('ln -sfn "$WORKSPACE_TARGET" /workspace') + lines.append("") + lines.append("# Fix ownership of declared mount targets (top-level only)") + for mount in manifest.get("mounts", []): + target = mount.get("target") + if not target: + continue + # Expand any ~/$HOME placeholders in the mount target. + expanded = target.replace("~", home_dir).replace("$HOME", home_dir) + if expanded.startswith(home_dir) and not mount.get("readonly", False): + lines.append(f'fix_owner "{expanded}"') + lines.append("") + startup_scripts = manifest.get("scripts", {}).get("startup", []) for script in startup_scripts: lines.append(script) @@ -281,6 +371,13 @@ def compile_compose(manifest: dict, variables: dict[str, Any]) -> str: runtime = manifest.get("runtime", {}) user = manifest.get("user") interface_type = manifest["interface_type"] + home_dir = get_manifest_home_dir(manifest) + + # Determine the workspace/repo name from variables when available. + workspace_name = variables.get( + "WORKSPACE_NAME", + variables.get("REPO_NAME", "workspace"), + ) service: dict[str, Any] = { "image": variables["IMAGE_TAG"], @@ -294,7 +391,9 @@ def compile_compose(manifest: dict, variables: dict[str, Any]) -> str: if runtime.get("tty", False): service["tty"] = True if runtime.get("working_dir"): - service["working_dir"] = runtime["working_dir"] + service["working_dir"] = expand_container_path( + runtime["working_dir"], home_dir + ) # User override if user: @@ -319,14 +418,24 @@ def compile_compose(manifest: dict, variables: dict[str, Any]) -> str: # Volumes from mount schema volumes = [] + has_explicit_repo_mount = False for mount in manifest.get("mounts", []): source = resolve_mount_source(mount, variables) if not source: continue - target = mount["target"] + if mount.get("source_type") == "repo": + has_explicit_repo_mount = True + target = expand_container_path(mount["target"], home_dir) readonly = ":ro" if mount.get("readonly", False) else "" volumes.append(f"{source}:{target}{readonly}") + # Synthesize a default repo/workspace mount when the manifest does not + # declare an explicit repo mount. This preserves the repo root directory + # name under the configured home directory. + if not has_explicit_repo_mount and variables.get("REPO_PATH"): + target = f"{home_dir}/{workspace_name}" + volumes.append(f"{variables['REPO_PATH']}:{target}") + # Append extra volumes from tool config / config profile for vol in variables.get("EXTRA_VOLUMES", []): vol_str = f"{vol['source']}:{vol['target']}" @@ -385,7 +494,12 @@ def resolve_mount_source(mount: dict, variables: dict[str, Any]) -> str: def get_manifest_home_dir(manifest: dict) -> str: - """Get the home directory for a container based on manifest user config. + """Get the home directory for a container based on manifest config. + + Precedence: + 1. manifest["home_directory"] if present and non-empty. + 2. /home/{user.name} if manifest.user.name is present. + 3. /root otherwise. Args: manifest: Fully resolved manifest JSON. @@ -393,6 +507,10 @@ def get_manifest_home_dir(manifest: dict) -> str: Returns: Home directory path (e.g., /home/user or /root). """ + home_directory = manifest.get("home_directory") + if home_directory and isinstance(home_directory, str) and home_directory.strip(): + return home_directory.strip() + user = manifest.get("user") if user and user.get("name"): return f"/home/{user['name']}" diff --git a/apps/api/src/services/docker/.pi-map.index.md b/apps/api/src/services/docker/.pi-map.index.md index 4b6fdde..16f23a5 100644 --- a/apps/api/src/services/docker/.pi-map.index.md +++ b/apps/api/src/services/docker/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/services/docker ## role -Provides Docker infrastructure services for container lifecycle management, compose orchestration, configuration deployment, and secure tunneling to expose internal services. +Provides Docker infrastructure services for container lifecycle management, compose orchestration, secure configuration deployment, and external tunnel exposure. ## parent index: apps/api/src/services/.pi-map.index.md map: apps/api/src/services/.pi-map.md diff --git a/apps/api/src/services/docker/.pi-map.md b/apps/api/src/services/docker/.pi-map.md index e9bf8f0..51c80f6 100644 --- a/apps/api/src/services/docker/.pi-map.md +++ b/apps/api/src/services/docker/.pi-map.md @@ -4,15 +4,15 @@ dir: apps/api/src/services/docker index: apps/api/src/services/docker/.pi-map.index.md ## role -Provides Docker infrastructure services for container lifecycle management, compose orchestration, configuration deployment, and secure tunneling to expose internal services. +Provides Docker infrastructure services for container lifecycle management, compose orchestration, secure configuration deployment, and external tunnel exposure. ## files - __init__.py | Package initialization file that exposes Docker-related service functions for container operations, compose management, configuration staging, and tunnel management. | dep: src.services.docker.compose, src.services.docker.config_staging, src.services.docker.container, src.services.docker.tunnel -- compose.py | Generates, renders, and executes Docker Compose commands for container orchestration with volume sorting and template substitution. | exp: func:sort_volumes_by_specificity(volumes: list[str]) → list[str], call:vol.split, call:len, call:parts[1].rstrip, call:target.count, call:targets.append, call:Counter(targets).items, call:logger.warning, call:sorted, func:_target_depth(vol: str) → int, call:vol.split, call:len, call:parts[1].rstrip, call:target.count, func:render_compose_template(template: str, variables: dict[str, Any]) → str, call:variables.items, call:result.replace, call:str, func:write_compose_file(instance_dir: str, content: str) → str, call:Path, call:compose_path.write_text, call:str, func:execute_compose_command(compose_path: str, action: str, timeout, env_file) → tuple[int, str, str], call:Path, call:cmd.extend, call:cmd.append, call:subprocess.run, call:str, raise:ValueError | dep: logging, subprocess, collections, pathlib, typing, collections.Counter, pathlib.Path, typing.Any +- compose.py | Generates, renders, and executes Docker Compose files with volume sorting and template variable substitution. | exp: func:sort_volumes_by_specificity(volumes: list[str]) → list[str], call:vol.split, call:len, call:parts[1].rstrip, call:target.count, call:targets.append, call:Counter(targets).items, call:logger.warning, call:sorted, func:_target_depth(vol: str) → int, call:vol.split, call:len, call:parts[1].rstrip, call:target.count, func:render_compose_template(template: str, variables: dict[str, Any]) → str, call:variables.items, call:result.replace, call:str, call:aliases.items, call:variables.get, func:write_compose_file(instance_dir: str, content: str) → str, call:Path, call:compose_path.write_text, call:str, func:execute_compose_command(compose_path: str, action: str, timeout, env_file) → tuple[int, str, str], call:Path, call:cmd.extend, call:cmd.append, call:subprocess.run, call:str, raise:ValueError | dep: logging, subprocess, collections, pathlib, typing, collections.Counter, pathlib.Path, typing.Any - config_staging.py | Stages configuration files into instance directories with security checks for path traversal. | exp: func:ensure_instance_directory(instance_id: str, base_path) → str, call:Settings, call:Path, call:instance_dir.mkdir, call:str, call:instance_dir.absolute, func:write_env_file(instance_dir: str, env_vars: dict[str, str]) → str, call:Path, call:env_vars.items, call:env_path.write_text, call:"\n".join, call:str, func:write_config_files(instance_dir: str, files: dict[str, str]) → None, call:Path, call:files.items, call:full_path.resolve().relative_to, call:instance_path.resolve, call:full_path.parent.mkdir, call:full_path.write_text, raise:ValueError | dep: logging, pathlib, src.config, src.config.Settings - container.py | Provides Docker container runtime queries and network management utilities via subprocess calls to the Docker CLI. | exp: func:get_container_id(instance_name: str) → str | None, call:instance_name.lower, call:subprocess.run, call:result.stdout.strip, call:ps_result.stdout.strip().splitlines, call:line.split, call:len, call:name.lower, func:get_container_name(instance_name: str) → str | None, call:subprocess.run, call:instance_name.lower, call:result.stdout.strip().lstrip, func:get_backend_network_name() → str, call:subprocess.run, call:result.stdout.strip().split, call:net.lower, func:connect_container_to_network(container_name: str, network_name) → bool, call:get_backend_network_name, call:subprocess.run, func:get_container_ip_on_network(container_id: str, network_name) → str | None, call:get_backend_network_name, call:subprocess.run, call:result.stdout.strip, func:is_container_on_network(container_id: str, network_name) → bool, call:get_backend_network_name, call:subprocess.run, func:get_container_status(container_id: str) → dict[str, Any], call:subprocess.run, call:result.stdout.strip().split, call:int, call:len, call:parts[1].isdigit, func:wait_for_container_running(container_id: str, timeout, interval) → dict[str, Any], call:time.time, call:get_container_status, call:time.sleep, func:get_container_logs(container_id: str, tail) → str, call:subprocess.run, call:str, func:find_free_port(start, end) → int, call:range, call:socket.socket, call:s.connect_ex, raise:RuntimeError | dep: logging, subprocess, time, typing, socket - tunnel.py | Manages Cloudflare tunnels by orchestrating cloudflared Docker containers to expose internal services via temporary public URLs. | exp: func:_tunnel_container_name(instance_name: str) → str, call:instance_name.lower, func:_ensure_image() → None, call:subprocess.run, call:result.stdout.strip, call:logger.info, call:logger.warning, func:_cleanup_stale_tunnel(tunnel_name: str) → None, call:subprocess.run, func:_get_tunnel_logs(tunnel_name: str) → tuple[str, str], call:subprocess.run, func:_get_tunnel_exit_code(tunnel_name: str) → int | None, call:subprocess.run, call:int, call:result.stdout.strip, func:start_tunnel(instance_name: str, container_port: int, timeout, target_url) → dict[str, str], call:_ensure_image, call:_tunnel_container_name, call:_cleanup_stale_tunnel, call:instance_name.lower, call:get_backend_network_name, call:logger.debug, call:" ".join, call:subprocess.run, call:proc.stdout.strip, call:re.compile, call:__import__("time").time, call:_get_tunnel_logs, call:url_pattern.search, call:match.group, call:_get_tunnel_exit_code, call:__import__("time").sleep, call:logger.info, raise:RuntimeError, func:stop_tunnel(instance_name: str) → None, call:_tunnel_container_name, call:_cleanup_stale_tunnel, call:logger.debug, func:recreate_tunnel(instance_name: str, container_port: int, target_url) → dict[str, str], call:stop_tunnel, call:start_tunnel, func:check_tunnel_health(url: str, timeout) → dict[str, Any], call:subprocess.run, call:int, call:result.stdout.strip, call:str(exc).lower, call:any | dep: logging, re, subprocess, typing, src.services.docker.container ## arch -Subprocess-based CLI wrapper architecture around Docker/cloudflared tools with template rendering, path-traversal-safe file staging, and functional decomposition into single-responsibility modules. +Service-oriented utility modules with subprocess-based Docker CLI integration, Jinja2 templating for compose generation, and security-hardened file operations with path traversal validation. ## tags tunnel, container, call:subprocess.run, get, name, call:, network, call:result.stdout.strip ## symbols diff --git a/apps/api/src/services/docker/compose.py b/apps/api/src/services/docker/compose.py index 660b2b0..2472eaa 100644 --- a/apps/api/src/services/docker/compose.py +++ b/apps/api/src/services/docker/compose.py @@ -61,6 +61,19 @@ def render_compose_template(template: str, variables: dict[str, Any]) -> str: for key, value in variables.items(): placeholder = f"{{{{{key}}}}}" result = result.replace(placeholder, str(value)) + + # Convenience aliases so legacy and migrated templates can use lowercase + # placeholders without changing every stored template. + aliases = { + "{{workspace_name}}": "WORKSPACE_NAME", + "{{home_directory}}": "HOME_DIRECTORY", + } + for alias_placeholder, key in aliases.items(): + if alias_placeholder in result: + result = result.replace( + alias_placeholder, str(variables.get(key, "workspace")) + ) + return result diff --git a/apps/api/src/services/tool/.pi-map.index.md b/apps/api/src/services/tool/.pi-map.index.md index 11d468d..2673b9a 100644 --- a/apps/api/src/services/tool/.pi-map.index.md +++ b/apps/api/src/services/tool/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/src/services/tool ## role -Provides backend infrastructure for provisioning and managing isolated development tool instances with their dependencies and network access. +Orchestrates end-to-end deployment and runtime management of development tool instances via containerized environments with remote access capabilities. ## parent index: apps/api/src/services/.pi-map.index.md map: apps/api/src/services/.pi-map.md diff --git a/apps/api/src/services/tool/.pi-map.md b/apps/api/src/services/tool/.pi-map.md index f1deace..ab3b000 100644 --- a/apps/api/src/services/tool/.pi-map.md +++ b/apps/api/src/services/tool/.pi-map.md @@ -4,11 +4,11 @@ dir: apps/api/src/services/tool index: apps/api/src/services/tool/.pi-map.index.md ## role -Provides backend infrastructure for provisioning and managing isolated development tool instances with their dependencies and network access. +Orchestrates end-to-end deployment and runtime management of development tool instances via containerized environments with remote access capabilities. ## files -- instance_service.py | Manages tool instance lifecycle operations including git repository mounting, config profile validation, Docker container orchestration, and SSH tunnel management for a development environment platform. | exp: func:_chown_path(path: str, uid: int, gid: int) → None, call:os.path.isdir, call:os.walk, call:os.path.join, call:contextlib.suppress, call:os.chown, call:logger.warning, func:_chown_staged_mounts(extra_volumes: list[dict], instance_dir: str, uid: int, gid: int) → None, call:vol.get, call:source.startswith, call:_chown_path, func:resolve_git_mounts(session: AsyncSession, resolved: ResolvedProfile, instance_dir, working_directory, home_dir) → list[dict], call:tasks.append, call:resolve_single_git_mount, call:asyncio.gather, call:isinstance, call:logger.warning, call:volume_mounts.extend, func:normalize_git_mount(entry: dict) → dict, call:dict, call:entry.get, call:entry.pop, func:clone_git_repo(remote_url: str, branch: str | None, clone_parent: str) → str, call:hashlib.md5(remote_url.encode()).hexdigest, call:remote_url.encode, call:remote_url.split("/")[-1].replace, call:os.path.join, call:os.path.exists, call:os.makedirs, call:clone_repository, call:logger.debug, call:logger.warning, call:pull_repository_updates, call:checkout_branch, func:resolve_git_mount_mappings(repo_path: str, mappings: list[dict], working_directory: str | None, home_dir) → list[dict], call:mapping.get, call:logger.warning, call:expand_container_path, call:target_path.startswith, call:os.path.join, call:expand_glob_source, call:os.path.exists, call:len, call:os.path.relpath, call:volume_mounts.append, call:logger.debug, func:resolve_single_git_mount(session: AsyncSession, git_mount: dict, instance_dir, working_directory, home_dir) → list[dict], call:normalize_git_mount, call:git_mount.get, call:logger.warning, call:asyncio.to_thread, call:resolve_git_mount_mappings, func:checkout_branch(repo_path: str, branch: str) → bool, call:subprocess.run, call:logger.warning, call:result.stderr.strip, func:pull_repository_updates(repo_path: str, remote_url: str) → None, call:subprocess.run, raise:RuntimeError, func:expand_glob_source(source_path: str, repo_path: str) → list[str], call:any, call:os.path.exists, call:glob_module.glob, call:len, call:os.path.abspath, call:abs_path.startswith, call:results.append, call:logger.warning, func:validate_config_profile(session: AsyncSession, profile_id: str | None, user_id: uuid.UUID, project_id: uuid.UUID, tool_type_id: uuid.UUID) → uuid.UUID | None, call:uuid.UUID, call:session.get, raise:HTTPException, func:sanitize_compose_file(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:isinstance, call:port_mapping.split, call:len, call:valid_ports.append, call:compose_file.write_text, call:yaml.dump, func:modify_compose_file(compose_path: str, port_override, start_command, working_directory, extra_volumes, home_dir) → None, call:Path, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:enumerate, call:isinstance, call:port_mapping.split, call:expand_container_path, call:vol.get, call:service_config["volumes"].append, call:service_config.get, call:sort_volumes_by_specificity, call:compose_file.write_text, call:yaml.dump, func:ensure_container_name_in_compose(compose_path: str, container_name: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:container_name.lower, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:ensure_web_bind_address(compose_path: str, tool_type_name: str, default_port: int) → None, call:KNOWN_BIND_FIXES.get, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].values, call:service_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.warning, call:logger.info, func:ensure_backend_network_in_compose(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:get_backend_network_name, call:compose_data["services"].values, call:svc_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:prepare_manifest_instance(session: AsyncSession, instance: ToolInstance, instance_dir: str, repo_path: str, env_vars: dict, extra_volumes: list, working_directory: str | None) → tuple[str, str, dict, str], call:session.get, call:dict, call:resolve_base, call:deep_merge, call:logger.warning, call:merge_with_config, call:manifest.pop, call:env_vars.update, call:extra_volumes.extend, call:compute_image_tag, call:subprocess.run, call:check.stdout.strip, call:compile_dockerfile, call:compile_entrypoint, call:logger.debug, call:len, call:asyncio.to_thread, call:logger.info, call:resolve_profile, call:gm.get, call:instance.name.lower, call:compile_compose, call:manifest.get, call:datetime.now, call:get_manifest_home_dir, raise:RuntimeError, func:create_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, data: "CreateInstanceRequest") → ToolInstance, call:session.get, call:uuid.UUID, call:validate_config_profile, call:uuid.uuid4, call:select(ToolInstance) .where(ToolInstance.workspace_id == workspace_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:select(ToolInstance) .where(ToolInstance.repository_id == repo_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:session.execute, call:len, call:result.scalars().all, call:ensure_instance_directory, call:os.path.join, call:find_free_port, call:f"headquarter/{instance_name}:latest".lower, call:asyncio.to_thread, call:logger.error, call:logger.info, call:instance_name.lower, call:write_compose_file, call:dict, call:resolve_base, call:deep_merge, call:compute_image_tag, call:compile_compose, call:str, call:render_compose_template, call:ToolInstance, call:session.add, call:session.commit, call:session.refresh, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:start_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, data: "StartInstanceRequest | None") → dict, call:session.get, call:validate_config_profile, call:session.commit, call:os.path.exists, call:logger.info, call:dict, call:resolve_base, call:deep_merge, call:get_manifest_home_dir, call:manifest.get, call:user_cfg.get, call:logger.debug, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:env_vars.update, call:config_files.update, call:extra_volumes.extend, call:resolve_git_mounts, call:profile_hints.get, call:len, call:logger.error, call:write_env_file, call:write_config_files, call:uuid.UUID, call:ssh_keys_to_mount.append, call:logger.warning, call:os.path.join, call:os.makedirs, call:_sanitize_filename, call:key_filenames.append, call:prepare_ssh_key_files, call:write_ssh_config, call:extra_volumes.append, call:_chown_staged_mounts, call:prepare_manifest_instance, call:write_compose_file, call:logger.exception, call:modify_compose_file, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:instance.name.lower, call:get_container_id, call:publish_lifecycle_event, call:wait_for_container_running, call:get_container_logs, call:resolved_manifest.get, call:apply_mount_permissions, call:home_dir.startswith, call:apply_ssh_permissions, call:probe_config.get, call:execute_probe, call:datetime.now().isoformat, call:"\n".join, call:start_tunnel, call:str, call:traceback.format_exc, raise:ValueError, raise:RuntimeError, func:restart_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.debug, call:logger.warning, call:os.path.exists, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:write_env_file, call:logger.error, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:datetime.now, call:session.commit, call:start_tunnel, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:delete_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, force) → None, call:session.get, call:os.path.dirname, call:os.path.join, call:os.path.exists, call:check_dirty_state, call:stop_tunnel, call:logger.debug, call:logger.warning, call:execute_compose_command, call:shutil.rmtree, call:publish_lifecycle_event, call:session.delete, call:session.commit, raise:ValueError, raise:RuntimeError, func:recreate_instance_tunnel(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:instance.name.lower, call:logger.info, call:get_container_id, call:logger.error, call:get_backend_network_name, call:is_container_on_network, call:connect_container_to_network, call:get_container_ip_on_network, call:logger.warning, call:recreate_tunnel, call:check_tunnel_health, call:health.get, call:subprocess.run, call:probe.stdout.strip, call:session.commit, call:logger.exception, raise:ValueError, raise:RuntimeError, func:stop_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.warning, call:os.path.exists, call:execute_compose_command, call:datetime.now, call:session.commit, call:publish_lifecycle_event, raise:ValueError, func:rename_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, display_name: str) → ToolInstance, call:session.get, call:display_name.strip, call:session.commit, call:session.refresh, raise:ValueError | dep: asyncio, contextlib, logging, os, subprocess, uuid, datetime, httpx, fastapi, sqlalchemy, sqlalchemy.ext.asyncio, src.models, src.schemas.tool, src.services.git.clone, src.services.config.config_profile_resolver, src.services.docker, src.services.shared.tunnel, src.services.build.docker_build, src.services.build.manifest_compiler, src.services.shared.permission_fixer, src.services.shared.readiness_probe, src.services.shared.ssh_keys, src.services.instance.event_bus, src.services.instance.lifecycle_hooks, src.auth.dependencies, hashlib, yaml, pathlib, traceback, shutil, glob +- instance_service.py | Manages tool instance lifecycle operations including git repository mounting, config profile resolution, Docker compose file generation, container orchestration, and SSH tunnel management for development environments. | exp: func:_chown_path(path: str, uid: int, gid: int) → None, call:os.path.isdir, call:os.walk, call:os.path.join, call:contextlib.suppress, call:os.chown, call:logger.warning, func:_chown_staged_mounts(extra_volumes: list[dict], instance_dir: str, uid: int, gid: int) → None, call:vol.get, call:source.startswith, call:_chown_path, func:resolve_git_mounts(session: AsyncSession, resolved: ResolvedProfile, instance_dir, working_directory, home_dir) → list[dict], call:tasks.append, call:resolve_single_git_mount, call:asyncio.gather, call:isinstance, call:logger.warning, call:volume_mounts.extend, func:normalize_git_mount(entry: dict) → dict, call:dict, call:entry.get, call:entry.pop, func:clone_git_repo(remote_url: str, branch: str | None, clone_parent: str) → str, call:hashlib.md5(remote_url.encode()).hexdigest, call:remote_url.encode, call:remote_url.split("/")[-1].replace, call:os.path.join, call:os.path.exists, call:os.makedirs, call:clone_repository, call:logger.debug, call:logger.warning, call:pull_repository_updates, call:checkout_branch, func:resolve_git_mount_mappings(repo_path: str, mappings: list[dict], working_directory: str | None, home_dir) → list[dict], call:mapping.get, call:logger.warning, call:expand_container_path, call:target_path.startswith, call:os.path.join, call:expand_glob_source, call:os.path.exists, call:len, call:os.path.relpath, call:volume_mounts.append, call:logger.debug, func:resolve_single_git_mount(session: AsyncSession, git_mount: dict, instance_dir, working_directory, home_dir) → list[dict], call:normalize_git_mount, call:git_mount.get, call:logger.warning, call:asyncio.to_thread, call:resolve_git_mount_mappings, func:checkout_branch(repo_path: str, branch: str) → bool, call:subprocess.run, call:logger.warning, call:result.stderr.strip, func:pull_repository_updates(repo_path: str, remote_url: str) → None, call:subprocess.run, raise:RuntimeError, func:expand_glob_source(source_path: str, repo_path: str) → list[str], call:any, call:os.path.exists, call:glob_module.glob, call:len, call:os.path.abspath, call:abs_path.startswith, call:results.append, call:logger.warning, func:validate_config_profile(session: AsyncSession, profile_id: str | None, user_id: uuid.UUID, project_id: uuid.UUID, tool_type_id: uuid.UUID) → uuid.UUID | None, call:uuid.UUID, call:session.get, raise:HTTPException, func:sanitize_compose_file(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:isinstance, call:port_mapping.split, call:len, call:valid_ports.append, call:compose_file.write_text, call:yaml.dump, func:modify_compose_file(compose_path: str, port_override, start_command, working_directory, extra_volumes, home_dir) → None, call:Path, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:enumerate, call:isinstance, call:port_mapping.split, call:expand_container_path, call:vol.get, call:service_config["volumes"].append, call:service_config.get, call:sort_volumes_by_specificity, call:compose_file.write_text, call:yaml.dump, func:ensure_container_name_in_compose(compose_path: str, container_name: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].items, call:container_name.lower, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:ensure_web_bind_address(compose_path: str, tool_type_name: str, default_port: int) → None, call:KNOWN_BIND_FIXES.get, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:compose_data["services"].values, call:service_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.warning, call:logger.info, func:ensure_backend_network_in_compose(compose_path: str) → None, call:Path, call:compose_file.exists, call:compose_file.read_text, call:yaml.safe_load, call:get_backend_network_name, call:compose_data["services"].values, call:svc_config.get, call:compose_file.write_text, call:yaml.dump, call:logger.info, func:prepare_manifest_instance(session: AsyncSession, instance: ToolInstance, instance_dir: str, repo_path: str, env_vars: dict, extra_volumes: list, working_directory: str | None) → tuple[str, str, dict, str], call:session.get, call:dict, call:resolve_base, call:deep_merge, call:logger.warning, call:merge_with_config, call:manifest.pop, call:env_vars.update, call:extra_volumes.extend, call:compute_image_tag, call:subprocess.run, call:check.stdout.strip, call:compile_dockerfile, call:compile_entrypoint, call:logger.debug, call:len, call:asyncio.to_thread, call:logger.info, call:resolve_profile, call:gm.get, call:instance.name.lower, call:compile_compose, call:manifest.get, call:datetime.now, call:get_manifest_home_dir, raise:RuntimeError, func:create_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, data: "CreateInstanceRequest") → ToolInstance, call:session.get, call:uuid.UUID, call:validate_config_profile, call:uuid.uuid4, call:select(ToolInstance) .where(ToolInstance.workspace_id == workspace_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:select(ToolInstance) .where(ToolInstance.repository_id == repo_id) .where(ToolInstance.tool_type_id == tool_type_id) .where, call:session.execute, call:len, call:result.scalars().all, call:ensure_instance_directory, call:os.path.join, call:find_free_port, call:f"headquarter/{instance_name}:latest".lower, call:asyncio.to_thread, call:logger.error, call:logger.info, call:os.path.basename, call:os.path.normpath, call:instance_name.lower, call:write_compose_file, call:dict, call:resolve_base, call:deep_merge, call:compute_image_tag, call:compile_compose, call:str, call:render_compose_template, call:ToolInstance, call:session.add, call:session.commit, call:session.refresh, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:start_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, data: "StartInstanceRequest | None") → dict, call:session.get, call:validate_config_profile, call:session.commit, call:os.path.exists, call:logger.info, call:dict, call:resolve_base, call:deep_merge, call:get_manifest_home_dir, call:manifest.get, call:user_cfg.get, call:logger.debug, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:env_vars.update, call:config_files.update, call:extra_volumes.extend, call:resolve_git_mounts, call:profile_hints.get, call:len, call:logger.error, call:write_env_file, call:write_config_files, call:uuid.UUID, call:ssh_keys_to_mount.append, call:logger.warning, call:os.path.join, call:os.makedirs, call:_sanitize_filename, call:key_filenames.append, call:prepare_ssh_key_files, call:write_ssh_config, call:extra_volumes.append, call:_chown_staged_mounts, call:prepare_manifest_instance, call:write_compose_file, call:logger.exception, call:modify_compose_file, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:instance.name.lower, call:get_container_id, call:publish_lifecycle_event, call:wait_for_container_running, call:get_container_logs, call:resolved_manifest.get, call:apply_mount_permissions, call:home_dir.startswith, call:apply_ssh_permissions, call:probe_config.get, call:execute_probe, call:datetime.now().isoformat, call:"\n".join, call:start_tunnel, call:str, call:traceback.format_exc, raise:ValueError, raise:RuntimeError, func:restart_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.debug, call:logger.warning, call:os.path.exists, call:os.path.dirname, call:resolve_profile, call:apply_resolved_profile, call:write_env_file, call:logger.error, call:sanitize_compose_file, call:ensure_web_bind_address, call:ensure_container_name_in_compose, call:ensure_backend_network_in_compose, call:execute_compose_command, call:datetime.now, call:session.commit, call:start_tunnel, call:publish_lifecycle_event, raise:ValueError, raise:RuntimeError, func:delete_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, force) → None, call:session.get, call:os.path.dirname, call:os.path.join, call:os.path.exists, call:check_dirty_state, call:stop_tunnel, call:logger.debug, call:logger.warning, call:execute_compose_command, call:shutil.rmtree, call:publish_lifecycle_event, call:session.delete, call:session.commit, raise:ValueError, raise:RuntimeError, func:recreate_instance_tunnel(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:instance.name.lower, call:logger.info, call:get_container_id, call:logger.error, call:get_backend_network_name, call:is_container_on_network, call:connect_container_to_network, call:get_container_ip_on_network, call:logger.warning, call:recreate_tunnel, call:check_tunnel_health, call:health.get, call:subprocess.run, call:probe.stdout.strip, call:session.commit, call:logger.exception, raise:ValueError, raise:RuntimeError, func:stop_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID) → dict, call:session.get, call:stop_tunnel, call:logger.warning, call:os.path.exists, call:execute_compose_command, call:datetime.now, call:session.commit, call:publish_lifecycle_event, raise:ValueError, func:rename_tool_instance(session: AsyncSession, user_id: uuid.UUID, project_id: uuid.UUID, repo_id: uuid.UUID, instance_id: uuid.UUID, display_name: str) → ToolInstance, call:session.get, call:display_name.strip, call:session.commit, call:session.refresh, raise:ValueError | dep: asyncio, contextlib, logging, os, subprocess, uuid, datetime, fastapi, sqlalchemy, sqlalchemy.ext.asyncio, src.models, src.schemas.tool, src.services.git.clone, src.services.config.config_profile_resolver, src.services.docker, src.services.shared.tunnel, src.services.build.docker_build, src.services.build.manifest_compiler, src.services.shared.permission_fixer, src.services.shared.readiness_probe, src.services.shared.ssh_keys, src.services.instance.event_bus, src.services.instance.lifecycle_hooks, hashlib, yaml, pathlib, traceback, shutil ## arch -Service-oriented architecture with a single service class handling multiple infrastructure concerns (git, Docker, SSH, config) through direct orchestration. +Service-layer facade combining infrastructure-as-code (Docker Compose generation), GitOps (repository mounting), external configuration resolution, and network tunneling (SSH) into a unified lifecycle management pattern. ## tags compose, call:compose, call:logger.warning, error, instance, container, git, text ## symbols diff --git a/apps/api/src/services/tool/instance_service.py b/apps/api/src/services/tool/instance_service.py index ca3d5c2..94f8141 100644 --- a/apps/api/src/services/tool/instance_service.py +++ b/apps/api/src/services/tool/instance_service.py @@ -9,7 +9,6 @@ import subprocess import uuid from datetime import datetime -import httpx from fastapi import HTTPException, status from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession @@ -33,7 +32,6 @@ from src.services.docker import ( get_container_id, get_container_ip_on_network, get_container_logs, - get_container_status, is_container_on_network, render_compose_template, sort_volumes_by_specificity, @@ -64,10 +62,9 @@ from src.services.shared.permission_fixer import ( apply_ssh_permissions, ) from src.services.shared.readiness_probe import execute_probe -from src.services.shared.ssh_keys import cleanup_ssh_key_files, prepare_ssh_key_files +from src.services.shared.ssh_keys import prepare_ssh_key_files from src.services.instance.event_bus import InstanceEventBus from src.services.instance.lifecycle_hooks import publish_lifecycle_event -from src.auth.dependencies import _get_owned_project, _get_user logger = logging.getLogger(__name__) _event_bus = InstanceEventBus() @@ -139,7 +136,7 @@ async def resolve_git_mounts( if isinstance(result, Exception): logger.warning("Git mount failed: %s", result) continue - if result: + if isinstance(result, list): volume_mounts.extend(result) return volume_mounts @@ -1043,7 +1040,11 @@ async def create_tool_instance( else "" ) - compose_content = f"""version: "3.8"\nservices:\n app:\n image: {image_tag}\n container_name: {instance_name.lower()}\n stdin_open: true\n tty: true\n{ports_section} volumes:\n - {repo_path}:/workspace\n restart: unless-stopped\n""" + home_dir = tool_type.home_directory or "/home/user" + repo_name = os.path.basename(os.path.normpath(repo_path)) + workspace_target = f"{home_dir}/{repo_name}" + + compose_content = f"""version: "3.8"\nservices:\n app:\n image: {image_tag}\n container_name: {instance_name.lower()}\n stdin_open: true\n tty: true\n{ports_section} environment:\n - HOME={home_dir}\n volumes:\n - {repo_path}:{workspace_target}\n working_dir: {workspace_target}\n restart: unless-stopped\n""" write_compose_file(instance_dir, compose_content) elif tool_type.definition_type == "manifest": @@ -1095,6 +1096,8 @@ async def create_tool_instance( "TOOL_PORT": tool_port, "USER_ID": str(user_id), "PROJECT_ID": str(project_id), + "WORKSPACE_NAME": os.path.basename(os.path.normpath(repo_path)), + "HOME_DIRECTORY": tool_type.home_directory or "/home/user", } compose_content = render_compose_template( tool_type.compose_template, variables diff --git a/apps/api/tests/.pi-map.index.md b/apps/api/tests/.pi-map.index.md index 77f726f..be886e4 100644 --- a/apps/api/tests/.pi-map.index.md +++ b/apps/api/tests/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/tests ## role -Provides shared test infrastructure and fixtures for the FastAPI API application. +Provides shared test infrastructure and fixtures for the API application's test suite. ## parent index: apps/api/.pi-map.index.md map: apps/api/.pi-map.md diff --git a/apps/api/tests/.pi-map.md b/apps/api/tests/.pi-map.md index 2a35dc9..8661644 100644 --- a/apps/api/tests/.pi-map.md +++ b/apps/api/tests/.pi-map.md @@ -4,11 +4,11 @@ dir: apps/api/tests index: apps/api/tests/.pi-map.index.md ## role -Provides shared test infrastructure and fixtures for the FastAPI API application. +Provides shared test infrastructure and fixtures for the API application's test suite. ## files - conftest.py | Provides shared pytest fixtures for testing a FastAPI application with async SQLite database, authenticated clients, and test data setup. | exp: func:test_client() → Generator[TestClient, None, None], call:create_async_engine, call:engine.begin, call:conn.run_sync, call:asyncio.run, call:init_db, call:async_sessionmaker, call:patch, call:TestClient, call:app.dependency_overrides.pop, call:engine.dispose, func:init_db(), call:engine.begin, call:conn.run_sync, func:override_get_db_session() → AsyncGenerator[AsyncSession, None], call:async_sessionmaker, func:db_session(test_client) → AsyncGenerator[AsyncSession, None], call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:gen.aclose, call:create_async_engine, call:engine.begin, call:conn.run_sync, call:async_sessionmaker, call:engine.dispose, func:authenticated_client(test_client) → Generator[TestClient, None, None], call:str, call:uuid.uuid4, call:Settings, call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:User, call:uuid.UUID, call:session.add, call:session.commit, call:gen.aclose, call:asyncio.run, call:create_test_user, call:create_session_cookie, call:test_client.cookies.set, func:create_test_user(), call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:User, call:uuid.UUID, call:session.add, call:session.commit, call:gen.aclose, func:test_project_and_repo(authenticated_client) → tuple[str, str], call:uuid.uuid4, call:Settings, call:authenticated_client.cookies.get, call:decode_session_cookie, call:uuid.UUID, call:asyncio.run, call:get_user_id, call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:Project, call:session.add, call:GitRepository, call:session.commit, call:gen.aclose, call:create_project_and_repo, call:str, raise:RuntimeError, func:get_user_id(), call:Settings, call:authenticated_client.cookies.get, call:decode_session_cookie, call:uuid.UUID, func:create_project_and_repo(), call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:Project, call:session.add, call:GitRepository, call:session.commit, call:gen.aclose, func:admin_client(test_client) → Generator[TestClient, None, None], call:str, call:uuid.uuid4, call:Settings, call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:User, call:uuid.UUID, call:session.add, call:session.commit, call:gen.aclose, call:asyncio.run, call:create_admin_user, call:create_session_cookie, call:test_client.cookies.set, func:create_admin_user(), call:app.dependency_overrides.get, call:override_fn, call:gen.asend, call:User, call:uuid.UUID, call:session.add, call:session.commit, call:gen.aclose | dep: asyncio, os, typing, unittest.mock, pytest, pytest_asyncio, fastapi.testclient, sqlalchemy.ext.asyncio, src.config, src.models.base, src.main, src.auth.dependencies, uuid, src.auth.session, src.models.user.user, src.models.project.project, src.models.project.git_repository, fastapi, sqlalchemy, aiosqlite, src.models, src.auth ## arch -Pytest fixture-based testing architecture using async SQLite in-memory database, dependency injection overrides, and async HTTP client setup for isolated integration tests. +Pytest plugin architecture with dependency-injected async fixtures for database, HTTP client, and authentication state management. ## tags call:app.dependency, call:create, overrides.get, call:override, fn, call:gen.asend, call:gen.aclose, user ## symbols diff --git a/apps/api/tests/integration/.pi-map.index.md b/apps/api/tests/integration/.pi-map.index.md index 220ef5e..0c42ca2 100644 --- a/apps/api/tests/integration/.pi-map.index.md +++ b/apps/api/tests/integration/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/tests/integration ## role -Integration test suite for the API backend, covering authentication, workspaces, projects, notifications, configuration, and database models with real PostgreSQL and git repositories. +Integration and unit test suite for the API backend, covering authentication, CRUD APIs, workspace operations, git integration, notifications, and database model validation against real PostgreSQL and mocked dependencies. ## parent index: apps/api/tests/.pi-map.index.md map: apps/api/tests/.pi-map.md @@ -21,6 +21,7 @@ map: apps/api/tests/.pi-map.md - test_projects_api.py - test_seed.py - test_ssh_keys_api.py +- test_tool_instance_lifecycle.py - test_tool_types_api.py - test_tool_types_api_extended.py - test_users_api.py diff --git a/apps/api/tests/integration/.pi-map.md b/apps/api/tests/integration/.pi-map.md index 839421b..7ea11a9 100644 --- a/apps/api/tests/integration/.pi-map.md +++ b/apps/api/tests/integration/.pi-map.md @@ -4,7 +4,7 @@ dir: apps/api/tests/integration index: apps/api/tests/integration/.pi-map.index.md ## role -Integration test suite for the API backend, covering authentication, workspaces, projects, notifications, configuration, and database models with real PostgreSQL and git repositories. +Integration and unit test suite for the API backend, covering authentication, CRUD APIs, workspace operations, git integration, notifications, and database model validation against real PostgreSQL and mocked dependencies. ## files - __init__.py | Empty file with no functionality - test_auth_api.py | Integration tests for authentication API endpoints using a real PostgreSQL database | exp: func:_postgres_available() → bool, call:asyncpg.connect, call:conn.close, call:asyncio.run, call:_check, func:_check() → bool, call:asyncpg.connect, call:conn.close, func:_prepare_auth_test_db() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, func:_load_app(), call:importlib.reload, func:_insert_test_user(user_id: str) → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, func:test_login_redirects_to_authentik_authorize_endpoint() → None, call:_prepare_auth_test_db, call:_load_app, call:TestClient, call:client.get, func:test_me_returns_401_without_session_cookie() → None, call:_prepare_auth_test_db, call:_load_app, call:TestClient, call:client.get, func:test_me_returns_user_with_valid_session() → None, call:_prepare_auth_test_db, call:_insert_test_user, call:_load_app, call:Settings, call:create_session_cookie, call:TestClient, call:client.get, call:response.json, func:test_logout_clears_session_cookie() → None, call:_prepare_auth_test_db, call:_load_app, call:TestClient, call:client.post, call:response.headers.get | dep: uuid, asyncio, importlib, fastapi.testclient, pytest, sqlalchemy, sqlalchemy.ext.asyncio, src.auth.session, src.config, src.models, src.models.user.user, asyncpg, fastapi @@ -18,6 +18,7 @@ Integration test suite for the API backend, covering authentication, workspaces, - test_projects_api.py | Integration tests for a FastAPI projects API endpoint, verifying authentication, CRUD operations, and ownership-based authorization against a real PostgreSQL database. | exp: func:_postgres_available() → bool, call:asyncpg.connect, call:conn.close, call:asyncio.run, call:_check, func:_check() → bool, call:asyncpg.connect, call:conn.close, func:_prepare_test_db() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, func:_load_app(), call:hasattr, call:asyncio.run, call:database_module.engine.dispose, call:importlib.reload, func:_mint_token(user_id: str) → str, call:Settings, call:create_session_cookie, func:_insert_user(user_id: str, email) → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, func:_insert_project(project_id: str, owner_id: str, name) → None, call:create_async_engine, call:build_database_url, call:async_sessionmaker, call:session_factory, call:Project, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:async_sessionmaker, call:session_factory, call:Project, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, func:test_create_project_requires_authentication() → None, call:_prepare_test_db, call:_load_app, call:TestClient, call:client.post, func:test_create_project_successfully() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.post, call:response.json, func:test_list_projects_returns_only_owned_projects() → None, call:_prepare_test_db, call:_insert_user, call:_insert_project, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.get, call:response.json, call:len, func:test_update_project_requires_ownership() → None, call:_prepare_test_db, call:_insert_user, call:_insert_project, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.patch, func:test_update_project_successfully() → None, call:_prepare_test_db, call:_insert_user, call:_insert_project, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.patch, call:response.json, func:test_delete_project_requires_ownership() → None, call:_prepare_test_db, call:_insert_user, call:_insert_project, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.delete, func:test_delete_project_successfully() → None, call:_prepare_test_db, call:_insert_user, call:_insert_project, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.delete, func:test_set_default_ssh_key_requires_ownership() → None, call:_prepare_test_db, call:_insert_user, call:_insert_project, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.patch | dep: uuid, datetime, asyncio, pytest, fastapi.testclient, sqlalchemy, sqlalchemy.ext.asyncio, src.auth.session, src.config, src.models, src.models.project.project, src.models.user.user, asyncpg, importlib, fastapi, src.database, src.api.user.auth, src.api.project.projects, src.main - test_seed.py | Tests deterministic seed user generation and database seeding functionality for development environments. | exp: func:test_build_seed_user_returns_deterministic_payload() → None, call:build_seed_user, func:test_seed_database_creates_development_user(db_session: AsyncSession) → None, call:seed_database, call:db_session.scalar, call:select(User).where | dep: pytest, sqlalchemy, sqlalchemy.ext.asyncio, src.models.user, src.scripts.seed - test_ssh_keys_api.py | Integration tests verifying SSH key API endpoints require authentication | exp: func:test_create_ssh_key_requires_authentication(test_client: TestClient) → None, call:test_client.post, func:test_list_ssh_keys_requires_authentication(test_client: TestClient) → None, call:test_client.get | dep: pytest, fastapi.testclient +- test_tool_instance_lifecycle.py | Integration test placeholder for verifying Docker availability and container home directory lifecycle behavior | exp: func:_docker_available() → bool, call:subprocess.run, func:test_docker_available_placeholder() → None, call:_docker_available | dep: subprocess, pytest - test_tool_types_api.py | Integration tests for a FastAPI tool types REST API endpoint using a real PostgreSQL database. | exp: func:_postgres_available() → bool, call:asyncpg.connect, call:conn.close, call:asyncio.run, call:_check, func:_check() → bool, call:asyncpg.connect, call:conn.close, func:_prepare_test_db() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, func:_load_app(), call:hasattr, call:asyncio.run, call:database_module.engine.dispose, call:importlib.reload, func:_mint_token(user_id: str) → str, call:Settings, call:create_session_cookie, call:datetime.now, call:timedelta, func:_insert_user(user_id: str, email) → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, func:_insert_tool_type(tool_type_id: str, name: str, display_name: str, compose_template: str, created_by_id) → None, call:create_async_engine, call:build_database_url, call:async_sessionmaker, call:session_factory, call:ToolType, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:async_sessionmaker, call:session_factory, call:ToolType, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, func:test_list_tool_types_requires_authentication() → None, call:_prepare_test_db, call:_load_app, call:TestClient, call:client.get, func:test_list_tool_types_returns_all_types() → None, call:_prepare_test_db, call:_insert_user, call:_insert_tool_type, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.get, call:response.json, call:len, call:next, func:test_get_tool_type_by_id() → None, call:_prepare_test_db, call:_insert_user, call:_insert_tool_type, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.get, call:response.json, func:test_get_tool_type_not_found() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.get, func:test_create_tool_type_successfully() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.post, call:response.json, func:test_create_tool_type_duplicate_name() → None, call:_prepare_test_db, call:_insert_user, call:_insert_tool_type, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.post, func:test_create_tool_type_invalid_yaml() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.post, func:test_create_tool_type_missing_services() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.post, func:test_create_tool_type_missing_required_variable() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.post, func:test_update_tool_type_successfully() → None, call:_prepare_test_db, call:_insert_user, call:_insert_tool_type, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.put, call:response.json, func:test_update_tool_type_not_found() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.put, func:test_delete_tool_type_successfully() → None, call:_prepare_test_db, call:_insert_user, call:_insert_tool_type, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.delete, call:client.get, func:test_delete_tool_type_not_found() → None, call:_prepare_test_db, call:_insert_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_mint_token, call:client.delete | dep: uuid, datetime, asyncio, pytest, fastapi.testclient, sqlalchemy, sqlalchemy.ext.asyncio, src.auth.session, src.config, src.models, src.models.tool.tool_type, src.models.user.user, asyncpg, importlib, fastapi - test_tool_types_api_extended.py | Integration tests for a FastAPI tool types API endpoint covering CRUD operations with extended fields like dockerfile templates, readiness probes, startup commands, and validation. | exp: class:TestToolTypesAPIExtended, method:test_create_tool_type_with_dockerfile(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_create_tool_type_with_readiness_probe(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_create_tool_type_invalid_definition_type(self, authenticated_client: TestClient) → None, call:authenticated_client.post, method:test_create_tool_type_dockerfile_without_template(self, authenticated_client: TestClient) → None, call:authenticated_client.post, method:test_update_tool_type_with_new_fields(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:create_response.json, call:authenticated_client.put, call:response.json, method:test_validate_tool_type_compose(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_validate_tool_type_invalid_compose(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_validate_tool_type_dockerfile(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_get_tool_type_returns_new_fields(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:create_response.json, call:authenticated_client.get, call:response.json, method:test_create_tool_type_without_port_fails(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, call:str, method:test_create_tool_type_with_port_mismatch_fails(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_create_tool_type_with_startup_command(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:response.json, method:test_update_tool_type_startup_command(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:create_response.json, call:authenticated_client.put, call:response.json, method:test_get_tool_type_returns_startup_command(self, authenticated_client: TestClient) → None, call:authenticated_client.post, call:create_response.json, call:authenticated_client.get, call:response.json | dep: pytest, fastapi.testclient, fastapi.testclient.TestClient - test_users_api.py | Integration tests for user profile API endpoints using a real PostgreSQL database | exp: func:_postgres_available() → bool, call:asyncpg.connect, call:conn.close, call:asyncio.run, call:_check, func:_check() → bool, call:asyncpg.connect, call:conn.close, func:_prepare_users_test_db() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:connection.execute, call:text, call:engine.dispose, func:_load_app(), call:importlib.reload, func:_insert_test_user(user_id: str) → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, call:asyncio.run, call:_run, func:_run() → None, call:create_async_engine, call:build_database_url, call:engine.begin, call:connection.run_sync, call:async_sessionmaker, call:session_factory, call:User, call:uuid.UUID, call:session.merge, call:session.commit, call:engine.dispose, func:_create_auth_cookie(user_id: str) → str, call:Settings, call:create_session_cookie, func:test_get_profile_returns_401_without_cookie() → None, call:_prepare_users_test_db, call:_load_app, call:TestClient, call:client.get, func:test_get_profile_returns_user_data() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.get, call:response.json, func:test_update_profile_changes_name_and_email() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.put, call:response.json, func:test_update_profile_rejects_empty_name() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.put, func:test_update_profile_rejects_invalid_email() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.put, func:test_upload_avatar_updates_avatar_url() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.post, call:io.BytesIO, call:response.json, call:data["avatar_url"].startswith, func:test_upload_avatar_rejects_invalid_file_type() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.post, call:io.BytesIO, func:test_upload_avatar_rejects_oversized_file() → None, call:_prepare_users_test_db, call:_insert_test_user, call:_load_app, call:TestClient, call:client.cookies.set, call:_create_auth_cookie, call:client.post, call:io.BytesIO | dep: uuid, datetime, asyncio, io, fastapi.testclient, pytest, sqlalchemy, sqlalchemy.ext.asyncio, src.auth.session, src.config, src.models, src.models.user.user, asyncpg, importlib, fastapi, src.database, src.api.user.users, src.main @@ -26,7 +27,7 @@ Integration test suite for the API backend, covering authentication, workspaces, - test_workspace_instances.py | Integration tests for FastAPI workspace instance endpoints (list and create) using an authenticated test client and mocked database fixtures. | exp: class:TestListWorkspaceInstances, method:test_list_empty(self, authenticated_client: TestClient, test_workspace_with_tool_type), call:authenticated_client.get, call:response.json, method:test_list_instances(self, authenticated_client: TestClient, db_session: AsyncSession, test_workspace_with_tool_type), call:ToolInstance, call:db_session.add, call:db_session.commit, call:asyncio.run, call:_create_instance, call:authenticated_client.get, call:response.json, call:len, class:TestCreateWorkspaceInstance, method:test_create_instance(self, authenticated_client: TestClient, test_workspace_with_tool_type), call:ToolInstance, call:uuid.uuid4, call:datetime.now, call:patch, call:authenticated_client.post, call:str, call:response.json, func:_get_user_id(client: TestClient) → uuid.UUID, call:Settings, call:client.cookies.get, call:decode_session_cookie, call:uuid.UUID, raise:RuntimeError, func:test_workspace_with_tool_type(db_session: AsyncSession, authenticated_client: TestClient), call:_get_user_id, call:Project, call:db_session.add, call:db_session.flush, call:GitRepository, call:tempfile.mkdtemp, call:Workspace, call:ToolType, call:db_session.commit, call:db_session.refresh, call:asyncio.run, call:_create, func:_create(), call:Project, call:db_session.add, call:db_session.flush, call:GitRepository, call:tempfile.mkdtemp, call:Workspace, call:ToolType, call:db_session.commit, call:db_session.refresh | dep: asyncio, tempfile, uuid, datetime, unittest.mock, pytest, fastapi.testclient, sqlalchemy.ext.asyncio, src.models, src.auth.session, src.config, src.api.workspace.workspace_instances - test_workspaces_api.py | Integration tests for workspace API endpoints including list, create, delete, and sync operations with repository/project scoping | exp: class:TestListWorkspaces, method:test_list_empty(self, authenticated_client: TestClient, test_repo: GitRepository), call:authenticated_client.get, call:response.json, method:test_list_with_workspaces(self, authenticated_client: TestClient, db_session: AsyncSession, test_repo: GitRepository), call:Workspace, call:db_session.add, call:db_session.commit, call:asyncio.run, call:_commit, call:authenticated_client.get, call:response.json, call:len, class:TestCreateWorkspace, method:test_create_success(self, authenticated_client: TestClient, test_repo: GitRepository), call:Workspace, call:uuid.uuid4, call:patch.object, call:authenticated_client.post, call:response.json, call:mock_create.assert_called_once, method:test_create_missing_name(self, authenticated_client: TestClient, test_repo: GitRepository), call:authenticated_client.post, call:response.json, method:test_create_duplicate_name(self, authenticated_client: TestClient, db_session: AsyncSession, test_repo: GitRepository), call:Workspace, call:db_session.add, call:db_session.commit, call:asyncio.run, call:_commit, call:patch.object, call:Exception, call:authenticated_client.post, class:TestDeleteWorkspace, method:test_delete_without_instances(self, authenticated_client: TestClient, db_session: AsyncSession, test_repo: GitRepository), call:Workspace, call:db_session.add, call:db_session.commit, call:db_session.refresh, call:asyncio.run, call:_commit_refresh, call:patch.object, call:authenticated_client.delete, call:response.json, method:test_delete_with_instances_force(self, authenticated_client: TestClient, db_session: AsyncSession, test_repo: GitRepository), call:Workspace, call:db_session.add, call:db_session.commit, call:db_session.refresh, call:asyncio.run, call:_commit_refresh, call:patch.object, call:authenticated_client.delete, class:TestSyncWorkspace, method:test_sync_success(self, authenticated_client: TestClient, db_session: AsyncSession, test_repo: GitRepository), call:Workspace, call:db_session.add, call:db_session.commit, call:db_session.refresh, call:asyncio.run, call:_commit_refresh, call:patch.object, call:MagicMock, call:authenticated_client.post, call:response.json, method:test_sync_branch_deleted(self, authenticated_client: TestClient, db_session: AsyncSession, test_repo: GitRepository), call:Workspace, call:db_session.add, call:db_session.commit, call:db_session.refresh, call:asyncio.run, call:_commit_refresh, call:patch.object, call:MagicMock, call:authenticated_client.post, call:response.json, func:_get_user_id_from_client(client: TestClient) → uuid.UUID, call:Settings, call:client.cookies.get, call:decode_session_cookie, call:uuid.UUID, raise:RuntimeError, func:test_repo(db_session: AsyncSession, authenticated_client: TestClient), call:_get_user_id_from_client, call:Project, call:db_session.add, call:db_session.flush, call:GitRepository, call:db_session.commit, call:db_session.refresh, call:asyncio.run, call:_create, func:_create(), call:Project, call:db_session.add, call:db_session.flush, call:GitRepository, call:db_session.commit, call:db_session.refresh | dep: asyncio, uuid, unittest.mock, pytest, fastapi.testclient, sqlalchemy.ext.asyncio, src.models, src.services.shared.workspace_manager, src.auth.session, src.config, fastapi, sqlalchemy ## arch -Pytest-based integration testing using real external dependencies (PostgreSQL, git repos), FastAPI TestClient with authenticated fixtures, mocked database fixtures for isolation, and security validation patterns (cookie/ OIDC/ SSE auth). +Pytest-based integration testing with FastAPI TestClient, SQLAlchemy async sessions, real PostgreSQL database fixtures, and selective mocking; follows pattern of endpoint-per-test-file with authenticated client fixtures and seeded database state. ## tags test, call:, call:db, call:authenticated, user, create, call:response.json, call:create ## symbols diff --git a/apps/api/tests/integration/test_tool_instance_lifecycle.py b/apps/api/tests/integration/test_tool_instance_lifecycle.py new file mode 100644 index 0000000..321d13f --- /dev/null +++ b/apps/api/tests/integration/test_tool_instance_lifecycle.py @@ -0,0 +1,34 @@ +"""Integration tests for tool container home directory behavior. + +These tests exercise container lifecycle behavior and require Docker and a +running PostgreSQL database. They are skipped when Docker is unavailable. +""" + +import subprocess + +import pytest + + +def _docker_available() -> bool: + try: + result = subprocess.run( + ["docker", "info"], + capture_output=True, + text=True, + timeout=10, + ) + return result.returncode == 0 + except Exception: + return False + + +@pytest.mark.integration +@pytest.mark.skipif(not _docker_available(), reason="Docker not available") +def test_docker_available_placeholder() -> None: + """Placeholder to keep the test file valid when Docker is present. + + Real lifecycle tests (container starts with HOME=/home/user, /workspace + symlink works, git mounts are writable) should be added here once the + test harness can start the API and Docker services. + """ + assert _docker_available() diff --git a/apps/api/tests/unit/.pi-map.index.md b/apps/api/tests/unit/.pi-map.index.md index 51f736d..238fc48 100644 --- a/apps/api/tests/unit/.pi-map.index.md +++ b/apps/api/tests/unit/.pi-map.index.md @@ -2,7 +2,7 @@ dir: apps/api/tests/unit ## role -Contains comprehensive unit tests for the API backend services covering configuration, Docker operations, Git integration, file management, health monitoring, notifications, and SSH key handling. +Unit test suite for the API application covering core services, utilities, and infrastructure components. ## parent index: apps/api/tests/.pi-map.index.md map: apps/api/tests/.pi-map.md @@ -10,6 +10,7 @@ map: apps/api/tests/.pi-map.md - ## files - __init__.py +- test_alembic_migrations.py - test_config.py - test_config_profile_resolver.py - test_docker_build.py @@ -21,6 +22,7 @@ map: apps/api/tests/.pi-map.md - test_git_url_parser.py - test_health_monitor.py - test_home_path_expansion.py +- test_instance_service.py - test_lifecycle_hooks.py - test_manifest_compiler.py - test_migration_metadata.py @@ -35,7 +37,7 @@ index: apps/api/tests/unit/.pi-map.index.md map: apps/api/tests/unit/.pi-map.md ## workflows - change unit behavior - read: __init__.py, test_config.py, test_config_profile_resolver.py + read: __init__.py, test_alembic_migrations.py, test_config.py - change unit config read: test_config.py, test_config_profile_resolver.py ## dirty diff --git a/apps/api/tests/unit/.pi-map.md b/apps/api/tests/unit/.pi-map.md index 920b089..c2e00d7 100644 --- a/apps/api/tests/unit/.pi-map.md +++ b/apps/api/tests/unit/.pi-map.md @@ -4,9 +4,10 @@ dir: apps/api/tests/unit index: apps/api/tests/unit/.pi-map.index.md ## role -Contains comprehensive unit tests for the API backend services covering configuration, Docker operations, Git integration, file management, health monitoring, notifications, and SSH key handling. +Unit test suite for the API application covering core services, utilities, and infrastructure components. ## files - __init__.py | Empty file with no functionality +- test_alembic_migrations.py | Unit tests that verify Alembic database migrations are importable, have correct identifiers, and declare expected dependencies without requiring a live database. | exp: func:test_home_directory_migration_imports_and_rewrites() → None, call:Path, call:migration_path.exists, call:importlib.util.spec_from_file_location, call:importlib.util.module_from_spec, call:spec.loader.exec_module, call:callable, func:test_merge_migration_resolves_heads() → None, call:Path, call:migration_path.exists, call:importlib.util.spec_from_file_location, call:importlib.util.module_from_spec, call:spec.loader.exec_module, call:callable | dep: importlib.util, pathlib, pytest, importlib - test_config.py | Tests configuration settings and database URL building for an application, verifying defaults, environment variable overrides, and environment-specific behavior. | exp: func:test_settings_default_database_url_uses_asyncpg(monkeypatch) → None, call:monkeypatch.delenv, call:Settings, func:test_build_database_url_uses_explicit_values() → None, call:build_database_url, func:test_settings_prefers_explicit_database_url_env(monkeypatch) → None, call:monkeypatch.setenv, call:Settings, func:test_auth_settings_have_secure_defaults() → None, call:Settings, call:settings.resolved_authentik_authorize_url.endswith, call:settings.resolved_authentik_token_url.endswith, call:settings.resolved_authentik_jwks_url.endswith, func:test_cookie_policy_is_strict_in_production(monkeypatch) → None, call:monkeypatch.setenv, call:Settings, func:test_cookie_policy_is_relaxed_for_local_dev(monkeypatch) → None, call:monkeypatch.setenv, call:Settings | dep: pytest, src.config, src.database - test_config_profile_resolver.py | Tests the config profile resolution system including merge helpers, profile inheritance with cycle detection, and git mount normalization | exp: class:TestMergeFunctions, method:test_merge_env_vars_basic(self) → None, call:_merge_env_vars, method:test_merge_env_vars_tracks_overrides(self) → None, call:_merge_env_vars, method:test_merge_runtime_hints_basic(self) → None, call:_merge_runtime_hints, method:test_merge_files_basic(self) → None, call:_merge_files, method:test_merge_mounts_basic(self) → None, call:_merge_mounts, method:test_merge_mounts_file_override(self) → None, call:_merge_mounts, call:ResolvedMount, method:test_merge_mounts_mode_conflict(self) → None, call:_merge_mounts, call:ResolvedMount, method:test_merge_git_mounts_basic(self) → None, call:_merge_git_mounts, call:len, method:test_merge_git_mounts_concatenate_same_repo_branch(self) → None, call:_merge_git_mounts, call:len, method:test_merge_git_mounts_dedup_same_mapping(self) → None, call:_merge_git_mounts, call:len, method:test_merge_git_mounts_different_repos(self) → None, call:_merge_git_mounts, call:len, method:test_merge_git_mounts_different_branches(self) → None, call:_merge_git_mounts, call:len, call:m.get, class:TestResolveProfile, class:TestApplyResolvedProfile, method:test_mounts_directory_not_individual_files(self, tmp_path) → None, call:ResolvedProfile, call:uuid.uuid4, call:ResolvedMount, call:apply_resolved_profile, call:str, call:len, call:Path(volumes[0]["source"]).is_dir, call:(Path(volumes[0]["source"]) / "config.json").exists, call:(Path(volumes[0]["source"]) / "nested" / "file.txt").exists, method:test_directory_mount_target(self, tmp_path) → None, call:ResolvedProfile, call:uuid.uuid4, call:ResolvedMount, call:apply_resolved_profile, call:str, call:len, call:Path, call:(Path(volumes[0]["source"]) / "z.json").exists, method:test_empty_mount_produces_no_volumes(self, tmp_path) → None, call:ResolvedProfile, call:uuid.uuid4, call:ResolvedMount, call:apply_resolved_profile, call:str, method:test_home_expansion_in_directory_mount_target(self, tmp_path) → None, call:ResolvedProfile, call:uuid.uuid4, call:ResolvedMount, call:apply_resolved_profile, call:str, call:len, call:(Path(volumes[0]["source"]) / "app.toml").exists, call:Path, method:test_readonly_mount_sets_readonly_flag(self, tmp_path) → None, call:ResolvedProfile, call:uuid.uuid4, call:ResolvedMount, call:apply_resolved_profile, call:str, call:len, call:volumes[0].get, method:test_writable_mount_does_not_set_readonly_flag(self, tmp_path) → None, call:ResolvedProfile, call:uuid.uuid4, call:ResolvedMount, call:apply_resolved_profile, call:str, call:len, call:volumes[0].get, class:TestCheckIncludeCycle | dep: uuid, pathlib, pytest, sqlalchemy.ext.asyncio, src.models.config.config_profile, src.services.config.config_profile_resolver - test_docker_build.py | Unit tests for a Docker image build service that verifies Dockerfile creation, command structure, context file handling, path traversal prevention, and error handling. | exp: class:TestBuildImage | dep: subprocess, tempfile, pathlib, unittest.mock, pytest, src.services.build.docker_build @@ -18,8 +19,9 @@ Contains comprehensive unit tests for the API backend services covering configur - test_git_url_parser.py | Tests for git URL parsing utilities that extract base repository URLs, validate clone URLs, and parse various git URL formats across GitHub, GitLab, and Bitbucket. | exp: class:TestExtractBaseRepoUrl, method:test_github_tree_url(self), call:extract_base_repo_url, method:test_github_blob_url(self), call:extract_base_repo_url, method:test_github_pull_url(self), call:extract_base_repo_url, method:test_github_issues_url(self), call:extract_base_repo_url, method:test_github_valid_url(self), call:extract_base_repo_url, method:test_github_url_with_query_params(self), call:extract_base_repo_url, method:test_gitlab_tree_url(self), call:extract_base_repo_url, method:test_gitlab_blob_url(self), call:extract_base_repo_url, method:test_gitlab_merge_request_url(self), call:extract_base_repo_url, method:test_gitlab_valid_url(self), call:extract_base_repo_url, method:test_bitbucket_src_url(self), call:extract_base_repo_url, method:test_bitbucket_valid_url(self), call:extract_base_repo_url, method:test_ssh_url(self), call:extract_base_repo_url, method:test_ssh_url_without_git_suffix(self), call:extract_base_repo_url, method:test_invalid_url(self), call:extract_base_repo_url, method:test_empty_url(self), call:extract_base_repo_url, class:TestIsValidCloneUrl, method:test_valid_ssh_url(self), call:is_valid_clone_url, method:test_valid_https_url(self), call:is_valid_clone_url, method:test_browser_url(self), call:is_valid_clone_url, method:test_url_without_git_suffix(self), call:is_valid_clone_url, method:test_invalid_url(self), call:is_valid_clone_url, class:TestParseGitUrl, method:test_valid_git_url(self), call:parse_git_url, method:test_browser_url(self), call:parse_git_url, method:test_invalid_url(self), call:parse_git_url, method:test_empty_url(self), call:parse_git_url, method:test_ssh_url(self), call:parse_git_url | dep: src.utils.git_url_parser, pytest - test_health_monitor.py | Unit tests for HealthMonitor state-transition logic covering container crash detection, tunnel failure detection, recovery detection, write deduplication, exception resilience, and start/stop lifecycle. | exp: func:event_bus() → InstanceEventBus, call:InstanceEventBus, call:bus._reset_for_testing, func:health_monitor(event_bus: InstanceEventBus) → HealthMonitor, call:HealthMonitor, func:_create_running_instance(db_session) → ToolInstance, call:User, call:uuid.uuid4, call:db_session.add, call:db_session.commit, call:ToolInstance, func:test_detects_container_crash(db_session, event_bus: InstanceEventBus, health_monitor: HealthMonitor) → None, call:_create_running_instance, call:events_captured.append, call:event_bus.subscribe, call:patch, call:health_monitor._check_instance, call:db_session.refresh, call:len, call:db_session.execute, call:select(HealthCheck).where, call:result.scalar_one, func:capture_event(payload: InstanceEventPayload) → None, call:events_captured.append, func:test_detects_tunnel_failure(db_session, event_bus: InstanceEventBus, health_monitor: HealthMonitor) → None, call:_create_running_instance, call:events_captured.append, call:event_bus.subscribe, call:patch, call:health_monitor._check_instance, call:db_session.refresh, call:len, call:db_session.execute, call:select(HealthCheck).where, call:result.scalar_one, func:capture_event(payload: InstanceEventPayload) → None, call:events_captured.append, func:test_detects_recovery(db_session, event_bus: InstanceEventBus, health_monitor: HealthMonitor) → None, call:_create_running_instance, call:db_session.commit, call:HealthSnapshot, call:events_captured.append, call:event_bus.subscribe, call:patch, call:health_monitor._check_instance, call:db_session.refresh, call:len, call:db_session.execute, call:select(HealthCheck).where, call:result.scalar_one, func:capture_event(payload: InstanceEventPayload) → None, call:events_captured.append, func:test_skips_writes_when_no_state_change(db_session, event_bus: InstanceEventBus, health_monitor: HealthMonitor) → None, call:_create_running_instance, call:HealthSnapshot, call:patch, call:health_monitor._check_instance, call:db_session.execute, call:select(HealthCheck).where, call:len, call:result.scalars().all, func:test_docker_exception_resilience(db_session, event_bus: InstanceEventBus, health_monitor: HealthMonitor) → None, call:_create_running_instance, call:events_captured.append, call:event_bus.subscribe, call:patch, call:RuntimeError, call:health_monitor._check_instance, call:db_session.execute, call:select(HealthCheck).where, call:result.scalar_one_or_none, func:capture_event(payload: InstanceEventPayload) → None, call:events_captured.append, func:test_monitor_start_stop(health_monitor: HealthMonitor) → None, call:health_monitor.start, call:task.done, call:health_monitor.stop, call:suppress, call:task.cancelled | dep: asyncio, uuid, contextlib, unittest.mock, pytest, sqlalchemy, src.models.system.health_check, src.models.tool.tool_instance, src.models.user.user, src.services.instance.event_bus, src.services.instance.health_monitor - test_home_path_expansion.py | Unit tests for home directory path expansion (~ and $HOME) in container paths and manifest home directory resolution. | exp: class:TestExpandContainerPath, method:test_tilde_slash_expands(self) → None, call:expand_container_path, method:test_tilde_alone_expands(self) → None, call:expand_container_path, method:test_dollar_home_slash_expands(self) → None, call:expand_container_path, method:test_dollar_home_alone_expands(self) → None, call:expand_container_path, method:test_absolute_path_unchanged(self) → None, call:expand_container_path, method:test_relative_path_unchanged(self) → None, call:expand_container_path, method:test_tilde_in_middle_unchanged(self) → None, call:expand_container_path, method:test_dollar_home_in_middle_unchanged(self) → None, call:expand_container_path, method:test_root_home(self) → None, call:expand_container_path, class:TestGetManifestHomeDir, method:test_with_user_block(self) → None, call:get_manifest_home_dir, method:test_without_user_block(self) → None, call:get_manifest_home_dir, method:test_with_empty_user_name(self) → None, call:get_manifest_home_dir, method:test_with_none_user_name(self) → None, call:get_manifest_home_dir | dep: pytest, src.services.config.config_profile_resolver, src.services.build.manifest_compiler +- test_instance_service.py | Unit tests for home directory expansion in docker-compose file modifications. | exp: class:TestModifyComposeFile, method:test_extra_volumes_expand_home_dir(self, tmp_path), call:compose_path.write_text, call:modify_compose_file, call:str, call:compose_path.read_text, method:test_working_directory_expands_home_dir(self, tmp_path), call:compose_path.write_text, call:modify_compose_file, call:str, call:compose_path.read_text | dep: pytest, src.services.tool.instance_service - test_lifecycle_hooks.py | Unit tests for lifecycle hook helper functions that derive notification titles and determine whether events should trigger notifications. | exp: class:TestDeriveTitle, method:test_known_event_types(self) → None, call:_derive_title, method:test_unknown_event_type(self) → None, call:_derive_title, class:TestShouldNotify, method:test_error_events_are_notified(self) → None, call:_should_notify, method:test_health_changed_running_is_notified(self) → None, call:_should_notify, method:test_created_started_stopped_restarted_deleted_filtered(self) → None, call:_should_notify, method:test_health_changed_non_running_filtered(self) → None, call:_should_notify | dep: pytest, src.services.instance.lifecycle_hooks -- test_manifest_compiler.py | Unit tests for a manifest compiler that generates Dockerfiles with user configuration and home directory setup | exp: func:test_compile_dockerfile_creates_config_dirs_for_user() → None, call:compile_dockerfile, func:test_compile_dockerfile_no_user_does_not_create_home() → None, call:compile_dockerfile | dep: pytest, src.services.build.manifest_compiler +- test_manifest_compiler.py | Unit tests for a manifest compiler that generates Docker configurations (Dockerfile, compose, entrypoint) based on manifest specifications. | exp: class:TestGetManifestHomeDir, method:test_home_directory_in_manifest_wins(self) → None, call:get_manifest_home_dir, method:test_user_name_derives_home(self) → None, call:get_manifest_home_dir, method:test_root_fallback(self) → None, call:get_manifest_home_dir, method:test_empty_home_directory_falls_back(self) → None, call:get_manifest_home_dir, class:TestCompileDockerfileHomeDirectory, method:test_env_home_and_workdir_use_home_directory(self) → None, call:compile_dockerfile, method:test_workspace_symlink_created(self) → None, call:compile_dockerfile, method:test_runtime_working_dir_overrides_home_workdir(self) → None, call:compile_dockerfile, method:test_working_dir_expands_tilde(self) → None, call:compile_dockerfile, class:TestCompileComposeHomeDirectory, method:test_default_repo_mount_synthesized(self) → None, call:compile_compose, method:test_explicit_repo_mount_preserved(self) → None, call:compile_compose, method:test_working_dir_expands_home(self) → None, call:compile_compose, class:TestCompileEntrypoint, method:test_entrypoint_creates_home_and_workspace(self) → None, call:compile_entrypoint, method:test_entrypoint_fixes_mount_owners(self) → None, call:compile_entrypoint, func:test_compile_dockerfile_creates_config_dirs_for_user() → None, call:compile_dockerfile, func:test_compile_dockerfile_no_user_does_not_create_home() → None, call:compile_dockerfile | dep: pytest, src.services.build.manifest_compiler - test_migration_metadata.py | Tests Alembic database migration files for correct table definitions and revision chain metadata | exp: func:test_initial_migration_defines_all_core_tables() → None, call:Path(__file__).resolve, call:spec_from_file_location, call:module_from_spec, call:spec.loader.exec_module, func:test_refresh_tokens_migration_has_expected_revision_chain() → None, call:Path(__file__).resolve, call:spec_from_file_location, call:module_from_spec, call:spec.loader.exec_module | dep: pytest, importlib.util, pathlib, pathlib.Path - test_monitoring_models.py | Unit tests verifying creation, persistence, and querying of monitoring models (InstanceEvent and HealthCheck) with database migration compatibility. | exp: func:test_instance_event_creation(db_session) → None, call:User, call:uuid.uuid4, call:db_session.add, call:db_session.commit, call:ToolInstance, call:InstanceEvent, call:db_session.refresh, call:isinstance, func:test_health_check_creation(db_session) → None, call:User, call:uuid.uuid4, call:db_session.add, call:db_session.commit, call:ToolInstance, call:HealthCheck, call:db_session.refresh, call:isinstance, func:test_instance_event_query_by_instance(db_session) → None, call:User, call:uuid.uuid4, call:db_session.add, call:db_session.commit, call:ToolInstance, call:InstanceEvent, call:db_session.execute, call:select(InstanceEvent).where, call:result.scalar_one | dep: uuid, datetime, pytest, sqlalchemy, src.models.system.health_check, src.models.system.instance_event, src.models.tool.tool_instance, src.models.user.user - test_notification_service.py | Unit tests for NotificationService covering CRUD operations, filtering, sorting, and ownership isolation. | exp: func:notification_service() → NotificationService, call:NotificationService, func:user_a(db_session: AsyncSession) → User, call:User, call:uuid.uuid4, call:db_session.add, call:db_session.commit, func:user_b(db_session: AsyncSession) → User, call:User, call:uuid.uuid4, call:db_session.add, call:db_session.commit, func:test_create_notification(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:uuid.uuid4, func:test_list_notifications_orders_by_created_at_desc(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:datetime.now, call:timedelta, call:db_session.commit, call:db_session.refresh, call:notification_service.list_notifications, func:test_list_notifications_excludes_dismissed(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:notification_service.dismiss, call:notification_service.list_notifications, func:test_list_notifications_unread_only(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:notification_service.mark_read, call:notification_service.list_notifications, func:test_get_unread_count(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:range, call:notification_service.create_notification, call:notification_service.mark_read, call:notification_service.get_unread_count, func:test_mark_read_sets_read_at(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:notification_service.mark_read, func:test_mark_all_read_affects_all_unread(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:range, call:notification_service.create_notification, call:notification_service.mark_all_read, call:notification_service.get_unread_count, func:test_dismiss_sets_dismissed_at(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:notification_service.dismiss, call:db_session.execute, call:select(Notification).where, call:result.scalar_one, func:test_mark_read_wrong_owner_raises(db_session: AsyncSession, notification_service: NotificationService, user_a: User, user_b: User) → None, call:notification_service.create_notification, call:pytest.raises, call:notification_service.mark_read, func:test_dismiss_wrong_owner_raises(db_session: AsyncSession, notification_service: NotificationService, user_a: User, user_b: User) → None, call:notification_service.create_notification, call:pytest.raises, call:notification_service.dismiss, func:test_list_notifications_mute_categories(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:notification_service.list_notifications, func:test_get_unread_count_excludes_dismissed(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:notification_service.create_notification, call:notification_service.dismiss, call:notification_service.get_unread_count, func:test_dismiss_all_affects_all_non_dismissed(db_session: AsyncSession, notification_service: NotificationService, user_a: User) → None, call:range, call:notification_service.create_notification, call:notification_service.dismiss_all, call:notification_service.list_notifications, func:test_dismiss_all_affects_only_caller(db_session: AsyncSession, notification_service: NotificationService, user_a: User, user_b: User) → None, call:range, call:notification_service.create_notification, call:notification_service.dismiss_all, call:notification_service.list_notifications, func:test_mark_all_read_affects_only_caller(db_session: AsyncSession, notification_service: NotificationService, user_a: User, user_b: User) → None, call:range, call:notification_service.create_notification, call:notification_service.mark_all_read, call:notification_service.get_unread_count | dep: uuid, datetime, pytest, sqlalchemy, sqlalchemy.ext.asyncio, src.models.system.notification, src.models.user.user, src.services.shared.notification_service, NotificationService, Notification, User, AsyncSession @@ -28,9 +30,9 @@ Contains comprehensive unit tests for the API backend services covering configur - test_readiness_probe.py | Unit tests for a Docker container readiness probe service that executes commands via docker exec with retry logic. | exp: class:TestExecuteProbe, class:TestIntegrationScenarios | dep: unittest.mock, src.services.shared.readiness_probe, subprocess - test_ssh_keys.py | Unit tests for SSH key preparation functionality including file creation, permissions, ownership, and error handling | exp: class:TestPrepareSshKeyFiles | dep: os, pathlib, unittest.mock, pytest, src.services.shared.ssh_keys ## arch -Standard Python unittest/pytest pattern with heavy mocking of external dependencies (subprocess, docker, filesystem) to test service layer logic in isolation, organized by functional domain with one test module per service component. +pytest-based unit testing with heavy mocking of external dependencies (Docker, Git, subprocess, database) to test components in isolation without live services. ## tags -test, url, call:notification, git, call:, call:db, merge, src +test, url, call:notification, git, home, call:, merge, call:db ## symbols - TestMergeFunctions - TestResolveProfile @@ -42,7 +44,7 @@ test, url, call:notification, git, call:, call:db, merge, src - TestSortVolumesBySpecificity ## workflows - change unit behavior - read: __init__.py, test_config.py, test_config_profile_resolver.py + read: __init__.py, test_alembic_migrations.py, test_config.py - change unit config read: test_config.py, test_config_profile_resolver.py ## dirty diff --git a/apps/api/tests/unit/test_alembic_migrations.py b/apps/api/tests/unit/test_alembic_migrations.py new file mode 100644 index 0000000..cedb34c --- /dev/null +++ b/apps/api/tests/unit/test_alembic_migrations.py @@ -0,0 +1,49 @@ +"""Unit tests for Alembic migration structure/import. + +Actual upgrade/downgrade round-trips require a PostgreSQL database, so these +tests verify that migrations are importable, have the expected identifiers, +and declare the expected dependencies. +""" + +import importlib.util +from pathlib import Path + +import pytest + + +@pytest.mark.unit +def test_home_directory_migration_imports_and_rewrites() -> None: + migration_path = Path(__file__).parent.parent.parent / ( + "alembic/versions/2026_06_14_104415_add_tool_type_home_directory.py" + ) + assert migration_path.exists() + + spec = importlib.util.spec_from_file_location("home_dir_migration", migration_path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + assert module.revision == "2026_06_14_104415" + assert module.down_revision == "f3d2dc90ba3a" + assert callable(module.upgrade) + assert callable(module.downgrade) + assert module.OLD_WORKSPACE == "/workspace" + assert module.NEW_WORKSPACE == "/home/user/{{WORKSPACE_NAME}}" + + +@pytest.mark.unit +def test_merge_migration_resolves_heads() -> None: + migration_path = Path(__file__).parent.parent.parent / ( + "alembic/versions/fc8f1a20cbf6_merge_home_directory_and_pi_agent_mount_.py" + ) + assert migration_path.exists() + + spec = importlib.util.spec_from_file_location("merge_migration", migration_path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + assert module.revision == "fc8f1a20cbf6" + assert "2026_06_14_104415" in module.down_revision + assert "8c6d1dbd4798" in module.down_revision + assert callable(module.upgrade) diff --git a/apps/api/tests/unit/test_instance_service.py b/apps/api/tests/unit/test_instance_service.py new file mode 100644 index 0000000..2699414 --- /dev/null +++ b/apps/api/tests/unit/test_instance_service.py @@ -0,0 +1,44 @@ +"""Unit tests for the tool instance service.""" + +import pytest + +from src.services.tool.instance_service import modify_compose_file + + +@pytest.mark.unit +class TestModifyComposeFile: + """Tests for modify_compose_file home-directory expansion.""" + + def test_extra_volumes_expand_home_dir(self, tmp_path): + compose_path = tmp_path / "docker-compose.yml" + compose_path.write_text( + 'services:\n app:\n image: test:latest\n volumes: []\n' + ) + + modify_compose_file( + str(compose_path), + extra_volumes=[ + {"source": "/host/config", "target": "~/.config", "type": "bind"}, + {"source": "/host/code", "target": "$HOME/code", "type": "bind"}, + ], + home_dir="/home/user", + ) + + content = compose_path.read_text() + assert "/host/config:/home/user/.config" in content + assert "/host/code:/home/user/code" in content + + def test_working_directory_expands_home_dir(self, tmp_path): + compose_path = tmp_path / "docker-compose.yml" + compose_path.write_text( + 'services:\n app:\n image: test:latest\n' + ) + + modify_compose_file( + str(compose_path), + working_directory="~/workspace", + home_dir="/home/user", + ) + + content = compose_path.read_text() + assert "working_dir: /home/user/workspace" in content diff --git a/apps/api/tests/unit/test_manifest_compiler.py b/apps/api/tests/unit/test_manifest_compiler.py index 8d63497..ccf4ef6 100644 --- a/apps/api/tests/unit/test_manifest_compiler.py +++ b/apps/api/tests/unit/test_manifest_compiler.py @@ -2,7 +2,12 @@ import pytest -from src.services.build.manifest_compiler import compile_dockerfile +from src.services.build.manifest_compiler import ( + compile_compose, + compile_dockerfile, + compile_entrypoint, + get_manifest_home_dir, +) @pytest.mark.unit @@ -36,3 +41,185 @@ def test_compile_dockerfile_no_user_does_not_create_home() -> None: assert "useradd" not in dockerfile assert "/home/" not in dockerfile + + +@pytest.mark.unit +class TestGetManifestHomeDir: + """Tests for get_manifest_home_dir precedence.""" + + def test_home_directory_in_manifest_wins(self) -> None: + manifest = { + "home_directory": "/home/custom", + "user": {"name": "dev"}, + } + assert get_manifest_home_dir(manifest) == "/home/custom" + + def test_user_name_derives_home(self) -> None: + manifest = {"user": {"name": "dev", "uid": 1000, "gid": 1000}} + assert get_manifest_home_dir(manifest) == "/home/dev" + + def test_root_fallback(self) -> None: + manifest = {"base_image": "ubuntu:24.04"} + assert get_manifest_home_dir(manifest) == "/root" + + def test_empty_home_directory_falls_back(self) -> None: + manifest = {"home_directory": "", "user": {"name": "dev"}} + assert get_manifest_home_dir(manifest) == "/home/dev" + + +@pytest.mark.unit +class TestCompileDockerfileHomeDirectory: + """Tests that compile_dockerfile honors manifest.home_directory.""" + + def test_env_home_and_workdir_use_home_directory(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + } + dockerfile = compile_dockerfile(manifest) + + assert "ENV HOME=/home/custom" in dockerfile + assert "WORKDIR /home/custom" in dockerfile + + def test_workspace_symlink_created(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + } + dockerfile = compile_dockerfile(manifest) + + assert "ln -sfn /home/custom/{{WORKSPACE_NAME}} /workspace" in dockerfile + + def test_runtime_working_dir_overrides_home_workdir(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + "runtime": {"working_dir": "/app/code"}, + } + dockerfile = compile_dockerfile(manifest) + + assert "WORKDIR /app/code" in dockerfile + assert "WORKDIR /home/custom" not in dockerfile + + def test_working_dir_expands_tilde(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + "runtime": {"working_dir": "~/code"}, + } + dockerfile = compile_dockerfile(manifest) + + assert "WORKDIR /home/custom/code" in dockerfile + + +@pytest.mark.unit +class TestCompileComposeHomeDirectory: + """Tests that compile_compose uses home_directory for volumes and working_dir.""" + + def test_default_repo_mount_synthesized(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + } + variables = { + "IMAGE_TAG": "test:latest", + "INSTANCE_NAME": "test-instance", + "REPO_PATH": "/host/repos/my-app", + "WORKSPACE_NAME": "my-app", + "TOOL_PORT": 0, + "EXTRA_ENV": {}, + "EXTRA_VOLUMES": [], + } + compose = compile_compose(manifest, variables) + + assert "/host/repos/my-app:/home/custom/my-app" in compose + + def test_explicit_repo_mount_preserved(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + "mounts": [ + {"source_type": "repo", "target": "/opt/code", "readonly": True} + ], + } + variables = { + "IMAGE_TAG": "test:latest", + "INSTANCE_NAME": "test-instance", + "REPO_PATH": "/host/repos/my-app", + "WORKSPACE_NAME": "my-app", + "TOOL_PORT": 0, + "EXTRA_ENV": {}, + "EXTRA_VOLUMES": [], + } + compose = compile_compose(manifest, variables) + + assert "/host/repos/my-app:/opt/code:ro" in compose + assert "/home/custom/my-app" not in compose + + def test_working_dir_expands_home(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + "runtime": {"working_dir": "$HOME/code"}, + } + variables = { + "IMAGE_TAG": "test:latest", + "INSTANCE_NAME": "test-instance", + "REPO_PATH": "/host/repos/my-app", + "WORKSPACE_NAME": "my-app", + "TOOL_PORT": 0, + "EXTRA_ENV": {}, + "EXTRA_VOLUMES": [], + } + compose = compile_compose(manifest, variables) + + assert "working_dir: /home/custom/code" in compose + + +@pytest.mark.unit +class TestCompileEntrypoint: + """Tests for the generated permission-fixing entrypoint.""" + + def test_entrypoint_creates_home_and_workspace(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + } + entrypoint = compile_entrypoint(manifest) + + assert 'mkdir -p "$HOME_DIR"' in entrypoint + assert 'mkdir -p "$WORKSPACE_TARGET"' in entrypoint + assert 'ln -sfn "$WORKSPACE_TARGET" /workspace' in entrypoint + + def test_entrypoint_fixes_mount_owners(self) -> None: + manifest = { + "base_image": "ubuntu:24.04", + "interface_type": "terminal", + "home_directory": "/home/custom", + "user": {"name": "dev", "uid": 1000, "gid": 1000}, + "mounts": [ + {"target": "~/.config", "readonly": False}, + {"target": "/opt/readonly", "readonly": True}, + ], + } + entrypoint = compile_entrypoint(manifest) + + assert 'fix_owner "/home/custom/.config"' in entrypoint + assert 'fix_owner "/opt/readonly"' not in entrypoint + diff --git a/docs/.pi-map.index.md b/docs/.pi-map.index.md index ccf5ee4..f9b0028 100644 --- a/docs/.pi-map.index.md +++ b/docs/.pi-map.index.md @@ -2,7 +2,7 @@ dir: docs ## role -Central documentation hub and knowledge base for the Headquarter platform. +Central documentation hub and navigation entry point for the Headquarter platform. ## parent index: ./.pi-map.index.md map: ./.pi-map.md diff --git a/docs/superpowers/.pi-map.index.md b/docs/superpowers/.pi-map.index.md index 3cf2f8e..67762cb 100644 --- a/docs/superpowers/.pi-map.index.md +++ b/docs/superpowers/.pi-map.index.md @@ -2,11 +2,14 @@ dir: docs/superpowers ## role -Documentation for advanced or specialized features and capabilities of the project. +Documentation for specialized capabilities or advanced features of the project. ## parent index: docs/.pi-map.index.md map: docs/.pi-map.md ## children +- docs/superpowers/handoffs + index: docs/superpowers/handoffs/.pi-map.index.md + map: docs/superpowers/handoffs/.pi-map.md - docs/superpowers/plans index: docs/superpowers/plans/.pi-map.index.md map: docs/superpowers/plans/.pi-map.md diff --git a/docs/superpowers/handoffs/.pi-map.index.md b/docs/superpowers/handoffs/.pi-map.index.md new file mode 100644 index 0000000..b6ddb8f --- /dev/null +++ b/docs/superpowers/handoffs/.pi-map.index.md @@ -0,0 +1,19 @@ +# docs/superpowers/handoffs (index) +dir: docs/superpowers/handoffs + +## role +Documents infrastructure migration procedures for development environment tooling. +## parent +index: docs/superpowers/.pi-map.index.md +map: docs/superpowers/.pi-map.md +## children +- +## files +- tool-container-home-directory.md +## links +index: docs/superpowers/handoffs/.pi-map.index.md +map: docs/superpowers/handoffs/.pi-map.md +## workflows +- +## dirty +- diff --git a/docs/superpowers/handoffs/.pi-map.md b/docs/superpowers/handoffs/.pi-map.md new file mode 100644 index 0000000..5826bee --- /dev/null +++ b/docs/superpowers/handoffs/.pi-map.md @@ -0,0 +1,19 @@ +# docs/superpowers/handoffs +dir: docs/superpowers/handoffs + +index: docs/superpowers/handoffs/.pi-map.index.md + +## role +Documents infrastructure migration procedures for development environment tooling. +## files +- tool-container-home-directory.md | Documents a multi-phase implementation plan for changing tool container home directories from `/workspace` to `/home/user` while preserving repo root directory names in mount targets and maintaining backward compatibility. | dep: alembic, docker, python, pydantic/sqlalchemy models, manifest compiler, instance service, config profile resolver, permission fixer +## arch +Technical documentation following phased implementation pattern with backward compatibility considerations. +## tags +home, tool, container, directory, documents, multi, phase, implementation +## symbols +- +## workflows +- +## dirty +- diff --git a/docs/superpowers/handoffs/tool-container-home-directory.md b/docs/superpowers/handoffs/tool-container-home-directory.md new file mode 100644 index 0000000..56a796f --- /dev/null +++ b/docs/superpowers/handoffs/tool-container-home-directory.md @@ -0,0 +1,91 @@ +# Handoff: Tool Container Home Directory + +## Session context + +- Original request: make tool container workspace/home directory default to `/home/user`, keep it user-writable, and preserve repo root directory name in the mount target. +- Completed a Grill Me session that resolved design decisions. +- Phase 1 implementation was started but blocked by persistent Grill Me read-only guard. + +## Current branch + +- `feat/tool-container-home-directory` (created from `dev`) + +## Committed artifacts on `dev` + +- `docs: tool container home directory design, plan, and test plan` +- `chore: update project maps for tool container home directory artifacts` + +## Files already created/edited on `feat/tool-container-home-directory` + +- `apps/api/src/models/tool/tool_type.py` + - Added `home_directory: Mapped[str]` column, default `"/home/user"`. +- `apps/api/alembic/versions/2026_06_14_104415_add_tool_type_home_directory.py` + - Adds column, sets existing rows to `/home/user`. + - Rewrites `/workspace` to `/home/user/{{WORKSPACE_NAME}}` in `compose_template` and `dockerfile_template`. + - Provides reversible downgrade. + +## Design decisions resolved + +| Area | Decision | +|------|----------| +| Config surface | `home_directory` field on ToolType / manifest schema | +| Default | `/home/user` | +| Runtime user | Still `manifest.user.name` | +| Mount target | `{home_directory}/{repo_name}` or `{home_directory}/{workspace_name}` | +| Precedence | Explicit manifest repo mount target wins; `home_directory` fills in default | +| Migration | Alembic data migration for legacy ToolType templates | +| Compatibility | `/workspace` symlink to actual repo/workspace target | +| Permissions | Entrypoint permission fixer at container startup | +| Tests | Unit + integration tests | + +## Pending work + +### Phase 1 verification +Run in `apps/api`: +```bash +python -m alembic current +python -m alembic upgrade head +python -m alembic downgrade -1 +python -m alembic upgrade head +``` + +### Phase 2: Manifest compiler +File: `apps/api/src/services/build/manifest_compiler.py` +- Honor `manifest.home_directory` in `get_manifest_home_dir()`. +- Set `ENV HOME={home_directory}`, `ENV USER={user.name}`, `WORKDIR {home_directory}`. +- Create `/workspace` symlink step. +- Use `{home_directory}/{repo_name}` as default repo mount target when no explicit repo mount exists. + +### Phase 3: Legacy instance generation +File: `apps/api/src/services/tool/instance_service.py` +- Use `tool_type.home_directory` for `dockerfile` tool mounts. +- Add `WORKSPACE_NAME`/`HOME_DIRECTORY` template variables for compose tools. + +### Phase 4: Config-profile / git mount expansion +Files: `apps/api/src/services/tool/instance_service.py`, `apps/api/src/services/config/config_profile_resolver.py` +- Thread resolved `home_dir` through startup. + +### Phase 5: Entrypoint permission fixer +Files: `apps/api/src/services/build/manifest_compiler.py`, `apps/api/src/services/shared/permission_fixer.py`, `tool-images/base.dockerfile` +- Generate startup script that chowns `{home_directory}` and key mounts to container user. + +### Phase 6: Tests +Create/expand: +- `apps/api/tests/unit/test_home_path_expansion.py` +- `apps/api/tests/unit/test_manifest_compiler.py` +- `apps/api/tests/unit/test_instance_service.py` +- `apps/api/tests/unit/test_alembic_migrations.py` +- `apps/api/tests/integration/test_tool_instance_lifecycle.py` + +## Reference docs + +- `openspec/designs/tool-container-home-directory.md` +- `docs/superpowers/plans/tool-container-home-directory.md` +- `docs/superpowers/specs/tool-container-home-directory-test-plan.md` +- `openspec/tasks/home-path-expansion.md` + +## Next action for fresh session + +1. Ensure you are on `feat/tool-container-home-directory`. +2. Verify Phase 1 migration round-trip. +3. Continue with Phase 2 (manifest compiler). diff --git a/openspec/.pi-map.index.md b/openspec/.pi-map.index.md index 02555ee..8ffb692 100644 --- a/openspec/.pi-map.index.md +++ b/openspec/.pi-map.index.md @@ -2,7 +2,7 @@ dir: openspec ## role -Defines and configures the OpenSpec methodology for structured living documentation of requirements, specifications, and task tracking within a Docker-based coding agent management platform. +Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a Docker-based coding agent management platform. ## parent index: ./.pi-map.index.md map: ./.pi-map.md diff --git a/openspec/.pi-map.md b/openspec/.pi-map.md index 2ea06d3..211f8b5 100644 --- a/openspec/.pi-map.md +++ b/openspec/.pi-map.md @@ -4,12 +4,12 @@ dir: openspec index: openspec/.pi-map.index.md ## role -Defines and configures the OpenSpec methodology for structured living documentation of requirements, specifications, and task tracking within a Docker-based coding agent management platform. +Defines the OpenSpec methodology and configuration for managing software requirements, specifications, and task tracking as living documentation within a Docker-based coding agent management platform. ## files - README.md | Documents the OpenSpec methodology for managing software requirements, specifications, and task tracking as living documentation within a project repository. | dep: OpenSpec CLI (@fission-ai/openspec), Docker, SQLAlchemy, Alembic, Authentik, React, TypeScript, Tailwind, Traefik, Jinja2, xterm.js, pytest, mypy, ruff, npm - config.yaml | Configuration file defining project metadata, technology stack, and software development discipline rules for a Docker-based coding agent management platform | dep: FastAPI, React, Vite, PostgreSQL, SQLAlchemy, Redis, Alembic, pytest, Docker Compose, Traefik, Authentik ## arch -Documentation-as-code pattern with YAML-driven configuration separating project metadata, technology stack definitions, and software development discipline rules from implementation code. +Documentation-as-code pattern with YAML-based configuration management, structured metadata definitions, and rule-based discipline enforcement for AI-assisted software development workflows. ## tags software, project, docker, sqlalchemy, alembic, authentik, react, traefik ## symbols diff --git a/openspec/tasks/.pi-map.index.md b/openspec/tasks/.pi-map.index.md index 0011557..e981d51 100644 --- a/openspec/tasks/.pi-map.index.md +++ b/openspec/tasks/.pi-map.index.md @@ -2,7 +2,7 @@ dir: openspec/tasks ## role -Technical specification and task tracking directory for planned engineering improvements across the project's container runtime, workspace management, and developer tooling systems. +Contains technical specification documents tracking planned features and architectural improvements for the project. ## parent index: openspec/.pi-map.index.md map: openspec/.pi-map.md diff --git a/openspec/tasks/.pi-map.md b/openspec/tasks/.pi-map.md index 8accd6c..c770074 100644 --- a/openspec/tasks/.pi-map.md +++ b/openspec/tasks/.pi-map.md @@ -4,17 +4,17 @@ dir: openspec/tasks index: openspec/tasks/.pi-map.index.md ## role -Technical specification and task tracking directory for planned engineering improvements across the project's container runtime, workspace management, and developer tooling systems. +Contains technical specification documents tracking planned features and architectural improvements for the project. ## files - config-profile-multi-repo-mounts.md | Technical specification for refactoring git mount configuration to support multiple repository mappings per mount instead of single source/target pairs | dep: git, docker compose, react/typescript, pytest, npm, fastapi -- home-path-expansion.md | Defines a technical implementation plan for adding configurable home/workspace directory support to tool containers in an API service. | dep: SQLAlchemy/Alembic, Docker, Docker Compose, Jinja2 templating, pytest, npm/frontend tooling +- home-path-expansion.md | Design and implementation plan for adding configurable home/workspace directory support to tool containers in an API application. | dep: SQLAlchemy/Alembic, Docker, Docker Compose, pytest, PostgreSQL, Node.js/npm - mount-specificity-ordering.md | Tracks completion status of a project task for implementing mount specificity ordering in Docker volume handling | dep: docker.py, pytest, tsc - terminal-responsiveness.md | A technical specification document outlining 7 pending tasks to improve web terminal performance through asyncio-native I/O, binary WebSocket frames, WebGL rendering, flow control, and comprehensive benchmarking. | dep: asyncio, WebSocket, xterm-addon-webgl, docker, pytest - tool-definition-manifest.md | A project task specification document defining a multi-PR system for implementing a Tool Definition Manifest System with backend manifest compilation, frontend workshop UI, and legacy migration support. | dep: SQLAlchemy, Alembic, Docker, Docker Compose, FastAPI/REST API, React/TypeScript frontend ## arch -Markdown-based living documentation pattern using standalone specification files that each capture a discrete technical initiative with implementation plans, completion tracking, and cross-referencing between backend API changes, frontend UI work, and infrastructure migrations. +Markdown-based task tracking system using standalone specification documents that each define discrete feature work across backend, frontend, and infrastructure concerns. ## tags -mount, docker, pytest, tool, manifest, technical, specification, support +mount, docker, pytest, tool, manifest, specification, support, docker compose ## symbols - ## workflows diff --git a/openspec/tasks/home-path-expansion.md b/openspec/tasks/home-path-expansion.md index c87f8e7..b04aabf 100644 --- a/openspec/tasks/home-path-expansion.md +++ b/openspec/tasks/home-path-expansion.md @@ -10,17 +10,17 @@ This task extends the earlier `home-path-expansion` work into a full configurabl **File**: `apps/api/src/models/tool/tool_type.py` -- Add `home_directory: Mapped[str]` column. -- Non-nullable with server default `"/home/user"`. +- [x] Add `home_directory: Mapped[str]` column. +- [x] Non-nullable with server default `"/home/user"`. ### T1.2: Alembic migration for legacy tool types **File**: `apps/api/alembic/versions/_add_tool_type_home_directory.py` -- Add the column. -- Set existing rows to `/home/user`. -- Rewrite `compose_template` and `dockerfile_template` to replace `/workspace` with `/home/user/{{WORKSPACE_NAME}}` where applicable. -- Provide downgrade that reverses rewrites and drops the column. +- [x] Add the column. +- [x] Set existing rows to `/home/user`. +- [x] Rewrite `compose_template` and `dockerfile_template` to replace `/workspace` with `/home/user/{{WORKSPACE_NAME}}` where applicable. +- [x] Provide downgrade that reverses rewrites and drops the column. ## T2: Manifest compiler @@ -28,22 +28,22 @@ This task extends the earlier `home-path-expansion` work into a full configurabl **File**: `apps/api/src/services/build/manifest_compiler.py` -- Update `get_manifest_home_dir()` to return `manifest.home_directory` if present, else derive from `user.name`, else `/root`. +- [x] Update `get_manifest_home_dir()` to return `manifest.home_directory` if present, else derive from `user.name`, else `/root`. ### T2.2: Dockerfile generation **File**: `apps/api/src/services/build/manifest_compiler.py` -- Set `ENV HOME={home_directory}` and `ENV USER={user.name}`. -- Set `WORKDIR {home_directory}` unless overridden by `runtime.working_dir`. -- Create `/workspace` symlink pointing to `{home_directory}/{{WORKSPACE_NAME}}` placeholder or create it at runtime. +- [x] Set `ENV HOME={home_directory}` and `ENV USER={user.name}`. +- [x] Set `WORKDIR {home_directory}` unless overridden by `runtime.working_dir`. +- [x] Create `/workspace` symlink pointing to `{home_directory}/{{WORKSPACE_NAME}}` placeholder or create it at runtime. ### T2.3: Compose generation **File**: `apps/api/src/services/build/manifest_compiler.py` -- Use `{home_directory}/{repo_name}` as default repo mount target when the manifest has no explicit repo mount. -- Keep `~`/`$HOME` expansion base equal to `home_directory`. +- [x] Use `{home_directory}/{repo_name}` as default repo mount target when the manifest has no explicit repo mount. +- [x] Keep `~`/`$HOME` expansion base equal to `home_directory`. ## T3: Legacy instance generation @@ -51,16 +51,16 @@ This task extends the earlier `home-path-expansion` work into a full configurabl **File**: `apps/api/src/services/tool/instance_service.py` -- Use `tool_type.home_directory` (default `/home/user`). -- Mount `{repo_path}:{home_directory}/{repo_name}`. -- Ensure `/workspace` symlink exists. +- [x] Use `tool_type.home_directory` (default `/home/user`). +- [x] Mount `{repo_path}:{home_directory}/{repo_name}`. +- [x] Ensure `/workspace` symlink exists. ### T3.2: Compose-based tools **File**: `apps/api/src/services/docker/compose.py`, `apps/api/src/services/tool/instance_service.py` -- Add `WORKSPACE_NAME` and `HOME_DIRECTORY` to template variables. -- Validate migrated templates render correctly. +- [x] Add `WORKSPACE_NAME` and `HOME_DIRECTORY` to template variables. +- [x] Validate migrated templates render correctly. ## T4: Config-profile and git mount expansion @@ -68,14 +68,14 @@ This task extends the earlier `home-path-expansion` work into a full configurabl **File**: `apps/api/src/services/tool/instance_service.py` -- Compute `home_dir` from ToolType/manifest in `start_tool_instance()`. -- Pass to `apply_resolved_profile()` and `resolve_git_mounts()`. +- [x] Compute `home_dir` from ToolType/manifest in `start_tool_instance()`. +- [x] Pass to `apply_resolved_profile()` and `resolve_git_mounts()`. ### T4.2: Verify `~`/`$HOME` expansion **File**: `apps/api/src/services/config/config_profile_resolver.py` -- Ensure `expand_container_path()` is called with the resolved `home_dir`. +- [x] Ensure `expand_container_path()` is called with the resolved `home_dir`. ## T5: Entrypoint permission fixer @@ -83,15 +83,15 @@ This task extends the earlier `home-path-expansion` work into a full configurabl **File**: `apps/api/src/services/build/manifest_compiler.py` -- Generate a startup script that chowns `{home_directory}` and mounted paths to the container user. -- Create `/workspace` symlink at runtime if needed. -- Avoid recursive chown of large repo subtrees. +- [x] Generate a startup script that chowns `{home_directory}` and mounted paths to the container user. +- [x] Create `/workspace` symlink at runtime if needed. +- [x] Avoid recursive chown of large repo subtrees. ### T5.2: Base image support **File**: `tool-images/base.dockerfile` (optional) -- Ensure `sudo` is available and the runtime user can elevate if the fixer runs inside the base image. +- [x] Ensure `sudo` is available and the runtime user can elevate if the fixer runs inside the base image. ## T6: Tests @@ -104,21 +104,21 @@ This task extends the earlier `home-path-expansion` work into a full configurabl - `apps/api/tests/unit/test_alembic_migrations.py` Cover: -- `expand_container_path` edge cases. -- `get_manifest_home_dir` precedence. -- `compile_dockerfile`/`compile_compose` behavior. -- Legacy instance mount target. -- Alembic migration round-trip. +- [x] `expand_container_path` edge cases. +- [x] `get_manifest_home_dir` precedence. +- [x] `compile_dockerfile`/`compile_compose` behavior. +- [x] Legacy instance mount target. +- [x] Alembic migration round-trip (round-trip against Postgres skipped in this environment). ### T6.2: Integration tests **File**: `apps/api/tests/integration/test_tool_instance_lifecycle.py` Cover: -- Container starts with `HOME=/home/user` and repo at `/home/user/{repo_name}`. -- `/workspace` symlink works. -- Config-profile and git mounts under `~` are writable. -- Explicit manifest repo mount target is preserved. +- [ ] Container starts with `HOME=/home/user` and repo at `/home/user/{repo_name}`. +- [ ] `/workspace` symlink works. +- [ ] Config-profile and git mounts under `~` are writable. +- [ ] Explicit manifest repo mount target is preserved. ## T7: Verification @@ -130,6 +130,9 @@ pytest tests/unit/test_home_path_expansion.py tests/unit/test_manifest_compiler. pytest tests/integration/test_tool_instance_lifecycle.py -xvs ``` +- [x] Unit tests passed (32 passed). +- [ ] Integration tests require Docker/Postgres. + ### T7.2: Alembic round-trip ```bash @@ -139,12 +142,18 @@ alembic downgrade -1 alembic upgrade head ``` +- [ ] Alembic round-trip requires PostgreSQL. +- [x] Single head confirmed (`fc8f1a20cbf6`). +- [x] Migration imports verified. + ### T7.3: Frontend typecheck ```bash cd apps/web && npm run typecheck ``` +- [x] No frontend changes made. + ## Estimation | Task | Effort | Files | diff --git a/tool-images/.pi-map.index.md b/tool-images/.pi-map.index.md index 6302bf0..d0d6588 100644 --- a/tool-images/.pi-map.index.md +++ b/tool-images/.pi-map.index.md @@ -2,7 +2,7 @@ dir: tool-images ## role -Provides Docker image templates for various development environment types used in the Headquarter platform. +Provides containerized development environment templates for AI coding agents and IDE tools used in the Headquarter platform. ## parent index: ./.pi-map.index.md map: ./.pi-map.md diff --git a/tool-images/.pi-map.md b/tool-images/.pi-map.md index ac433bf..1eb34df 100644 --- a/tool-images/.pi-map.md +++ b/tool-images/.pi-map.md @@ -4,16 +4,16 @@ dir: tool-images index: tool-images/.pi-map.index.md ## role -Provides Docker image templates for various development environment types used in the Headquarter platform. +Provides containerized development environment templates for AI coding agents and IDE tools used in the Headquarter platform. ## files - README.md | Documents a directory of Dockerfile templates for base tool types used in the Headquarter platform | dep: Docker, git, neovim, ranger, tmux, htop, tree, jq, Node.js 20, Pi Coding Agent -- base.dockerfile | Defines a base Docker image for development environments with common CLI tools, Node.js, and a non-root user setup. | dep: ubuntu:24.04, curl, wget, git, neovim, ranger, tmux, htop, tree, jq, ca-certificates, nodejs +- base.dockerfile | Defines a base Docker development image with common CLI tools, Node.js, and a non-root user with sudo privileges for bind-mounted directory permission fixes. | dep: ubuntu:24.04, curl, wget, git, neovim, ranger, tmux, htop, tree, jq, ca-certificates, sudo, nodejs - code-server.dockerfile | Builds a Docker image for browser-based VS Code with additional development tools and Node.js | dep: linuxserver/code-server, git, neovim, ranger, tmux, htop, tree, jq, Node.js 20 - jupyter.dockerfile | Builds a Docker image for Jupyter Notebook/Lab with additional CLI tools and git configuration | dep: jupyter/scipy-notebook, apt, git, neovim, ranger, tmux, htop, tree, jq - opencode.dockerfile | Builds a Docker container for running the OpenCode AI agent as a server | dep: ubuntu:24.04, nodejs, npm, python3, git, opencode - pi-agent.dockerfile | Builds a Docker container providing a terminal-based development environment with the Pi coding agent and supporting tools (neovim, tmux, ranger, etc.) | dep: ubuntu:24.04, nodejs, npm, @earendil-works/pi-coding-agent, git, tmux, neovim, ranger, python3, build-essential ## arch -Template-based container architecture with layered Dockerfiles extending a common base image pattern for specialized IDE, notebook, and AI agent environments. +Multi-flavored Dockerfile pattern with shared base layer (common CLI tools, Node.js, non-root user) and specialized downstream images for distinct tool types (VS Code, Jupyter, OpenCode, Pi agent). ## tags git, docker, neovim, ranger, tmux, agent, base, htop ## symbols diff --git a/tool-images/base.dockerfile b/tool-images/base.dockerfile index 00fc74b..2459699 100644 --- a/tool-images/base.dockerfile +++ b/tool-images/base.dockerfile @@ -4,7 +4,8 @@ FROM ubuntu:24.04 # Prevent interactive prompts during apt install ENV DEBIAN_FRONTEND=noninteractive -# Install base tools +# Install base tools plus sudo so the entrypoint permission fixer can +# chown runtime mounts staged by the API before dropping to the user. RUN apt-get update && apt-get install -y \ curl \ wget \ @@ -16,10 +17,14 @@ RUN apt-get update && apt-get install -y \ tree \ jq \ ca-certificates \ + sudo \ && rm -rf /var/lib/apt/lists/* -# Create a non-root user -RUN useradd -m -s /bin/bash user +# Create a non-root user and allow passwordless sudo so the startup +# permission fixer can adjust ownership of bind-mounted directories. +RUN useradd -m -s /bin/bash user \ + && echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/user \ + && chmod 0440 /etc/sudoers.d/user WORKDIR /home/user # Install Node.js (needed for pi and many dev tools)