headquarter/openspec/explorations/streamline-tool-container-definitions.md

SDD Exploration: Streamline Tool Container Definitions

Status

Phase: explore
Date: 2026-05-28
Owner: el Gentleman (parent session)
Scope: Tool container definition, build, and mount system

---

Executive Summary

The current tool container system works for the happy path (pi-agent on Ubuntu) but has deep structural inflexibility:

1. Monolithic Dockerfile strings in the database — impossible to review, version, or compose
2. Ad-hoc compose generation — string formatting with hardcoded fields (stdin_open, tty, working_dir missing for dockerfile types)
3. Hardcoded mount paths — /workspace and /root/.ssh don't adapt to the container's runtime user
4. No package/base-image modularity — every tool type carries a full Dockerfile copy
5. Permission mismatch — bind mounts come in as root-owned; non-root container users can't write
6. Config overlap — tool configs, config profiles, and compose templates fight for control of the same fields

This exploration proposes a layered, declarative container definition system where tool types compose from reusable base images, mount schemas, and permission policies.

---

Current Architecture Map

Data Model

ToolType (DB table)
  ├── name, display_name, description, category
  ├── interface_type: "web" | "terminal"
  ├── definition_type: "dockerfile" | "compose"
  ├── dockerfile_template: TEXT (giant Dockerfile string)
  ├── compose_template: TEXT (Jinja-like {{VAR}} string)
  ├── build_context: JSON {path: content}
  ├── required_variables: JSON ["REPO_PATH", ...]
  ├── default_port: int
  └── readiness_probe: JSON

ToolInstance (DB table)
  ├── name, display_name, status
  ├── tool_type_id → ToolType
  ├── repository_id → GitRepository
  ├── compose_path: str
  ├── container_id, container_name
  ├── port, url, public_url, tunnel_id
  ├── clone_mode: "mount" | "clone"
  ├── branch, new_branch
  └── selected_config_profile_id → ConfigProfile

ToolConfig (DB table, per-user per-tool-type)
  ├── config_type: "env" | "file"
  ├── key, value, file_path
  ├── port_override, start_command, working_directory
  ├── environment_variables: JSON
  └── volumes: JSON [{source, target, type}]

ConfigProfile (DB table)
  ├── name, description
  ├── user_id, project_id, tool_type_id
  ├── environment_variables: JSON
  ├── files: JSON {path: content}
  ├── mounts: JSON [{source, target, type}]
  ├── git_mounts: JSON [{remote_url, source_path, target_path, branch}]
  └── parent_profile_id → ConfigProfile (hierarchy)

Creation Flow (create_instance)

POST /projects/{id}/repositories/{id}/instances
  → validate tool_type, repo, config_profile
  → generate instance_name = "{tool_type}-{repo}-{uuid8}"
  → ensure_instance_directory(instance_name)
  → find_free_port()
  → determine repo_path (mount = repo.path; clone = clone_repository())
  → IF tool_type.definition_type == "dockerfile":
       build_image(instance_dir, dockerfile_template, tag, build_context)
       generate compose_content (HARDCODED STRING FORMATTING)
     ELSE:
       render_compose_template(tool_type.compose_template, variables)
  → write_compose_file()
  → create ToolInstance DB record (status="pending")

Startup Flow (start_instance)

POST /instances/{id}/start
  → fetch ToolConfigs (env, files, port_override, start_command, working_dir, volumes)
  → IF selected_config_profile:
       resolve_profile() → env, files, mounts, git_mounts, hints
  → write .env file, config files
  → IF clone_mode: mount SSH keys at /root/.ssh (HARDCODED)
  → _modify_compose_file(port, command, working_dir, extra_volumes)
  → _sanitize_compose_file()
  → execute_compose_command("up")
  → get_container_id(instance.name)  ← CASE-SENSITIVE BUG (fixed)
  → get_container_name(instance.name)
  → connect_container_to_network("backend")
  → wait_for_container_running()
  → IF web: start_cloudflared_tunnel()
  → instance.status = "running"

Key Files

File	Responsibility
apps/api/src/api/tool_instances.py	create_instance, start_instance, stop_instance, restart_instance, proxy, logs
apps/api/src/api/tool_types.py	CRUD for ToolType (DB strings)
apps/api/src/services/docker.py	compose execution, container queries, tunnel management
apps/api/src/services/docker_build.py	docker build wrapper
apps/api/src/services/terminal_session.py	PTY-based terminal over docker exec
apps/api/src/services/terminal_manager.py	WebSocket ↔ terminal session lifecycle
apps/api/src/models/tool_type.py	SQLAlchemy model

---

Pain Points (Detailed)

1. Monolithic Dockerfile Templates

The pi-agent Dockerfile template is a 40-line string stored in the DB migration:

INSERT INTO tool_types (... dockerfile_template ...)
VALUES ('...# Pi Coding Agent - Terminal-based coding harness\nFROM ubuntu:24.04\n...')

Problems:

• No syntax highlighting, linting, or docker build validation at edit time
• Every tool type copies the entire Dockerfile; no reuse of common layers
• Changes require a DB migration
• No way for users to customize packages without forking the whole template

2. Ad-Hoc Compose Generation

For dockerfile type tools, the compose is generated by Python f-string:

compose_content = f"""version: "3.8"
services:
  app:
    image: {image_tag}
    container_name: {instance_name.lower()}
{ports_section}    volumes:
      - {repo_path}:/workspace
    restart: unless-stopped
"""

Problems:

• Missing stdin_open: true and tty: true (essential for terminal tools)
• Missing working_dir: /workspace
• No way to add labels, networks, healthchecks, or extra services
• Port section is conditionally included with awkward string concatenation

3. Hardcoded Mount Paths

Mount	Current Target	Problem
Repository	/workspace	Always root-owned; no permission fix for non-root users
SSH keys (clone mode)	/root/.ssh	Invisible to containers running as user
Git-mount configs	/tmp/.pi	May be root-owned; conflicts with user's .pi
Config profile files	Instance-relative paths	No validation against container filesystem

4. No Base Image / Layer Composition

Every tool type must specify a complete Dockerfile from FROM to CMD. There's no way to say:

base: ubuntu-24.04-dev  # pre-built with curl, git, build-essential
layers:
  - nodejs-20
  - pi-coding-agent
  - custom-packages: [neovim, ranger, tmux]

5. Permission Mismatch (Non-Root Users)

The pi-agent Dockerfile creates a user account and uses USER user. Bind mounts from the host come in as root-owned. The API has no automatic permission fix — this caused the workspace-unwritable bug.

Workarounds considered:

• Post-start docker exec --user root chown (current fix)
• Dockerfile entrypoint script that chowns before dropping privileges
• Matching container UID to host UID

None of these are systematic or configurable.

6. Config Overlap and Precedence Confusion

Three systems control the same container aspects:

System	Controls	Stored
ToolConfig	env vars, files, port, command, working_dir, volumes	DB (per-user per-tool)
ConfigProfile	env vars, files, mounts, git_mounts, hints	DB (hierarchical)
Compose template / generation	volumes, ports, command, working_dir	DB string / Python f-string

Precedence is unclear:

• ToolConfig working_directory vs ConfigProfile hint working_directory vs compose working_dir
• ToolConfig volumes vs ConfigProfile mounts vs compose volumes
• start_command from ToolConfig vs ConfigProfile vs Dockerfile CMD

7. Build Context Limitations

build_context is a JSON dictionary of {relative_path: file_content}. This is stored in the DB as text.

Problems:

• Binary files (images, tarballs) can't be stored
• Large files bloat the DB
• No versioning or external reference (e.g., "use file from git repo")

---

Extensibility Gaps

Want	Current State	Gap
Add a new language runtime (e.g., Go, Rust)	Copy entire Dockerfile, edit	No modular package/layer system
Use a custom base image (e.g., my-registry/dev-base:v2)	Edit full Dockerfile	No base-image reference field
Mount a second repo or a secrets file	Write ConfigProfile or ToolConfig JSON	No declarative mount schema
Run as root instead of user	Edit full Dockerfile	No runtime-user field
Add a sidecar (e.g., postgres for integration tests)	Edit compose_template string	No multi-service compose support
Pre-install VS Code server	Edit full Dockerfile	No "feature" or "extension" mechanism
Custom entrypoint script	Edit full Dockerfile	No entrypoint field

---

Design Directions (Pre-Proposal)

Direction A: Declarative Tool Manifests

Replace the monolithic dockerfile_template with a structured manifest:

# Example: tool manifest for pi-agent
name: pi-agent
base_image: ubuntu:24.04
user:
  name: user
  uid: 1000
  home: /home/user
packages:
  apt: [curl, wget, git, neovim, ranger, tmux, htop, tree, jq, python3, python3-pip, build-essential]
  npm_global: [@earendil-works/pi-coding-agent]
  node_version: "20"
env:
  DEBIAN_FRONTEND: noninteractive
config_files:
  /home/user/.tmux.conf: "set -g mouse on\n..."
  /home/user/.config/ranger/rc.conf: "set preview_files true\n..."
working_directory: /workspace
command: ["/bin/bash"]
ports: []
mounts:
  repo: {target: /workspace, writable: true}
  ssh: {target: /home/user/.ssh, mode: "0600"}
  tmp_state: {target: /tmp/.pi, writable: true}

Pros: Structured, reviewable, composable
Cons: Requires a manifest-to-Dockerfile compiler; migration complexity

Direction B: Base Image Registry + Layers

Maintain a registry of pre-built base images:

headquarter/base/ubuntu-24.04-dev
headquarter/base/nodejs-20
headquarter/base/python-3.11

Tool types reference a base image and a list of layers:

base_image: headquarter/base/ubuntu-24.04-dev
layers:
  - type: npm_install
    package: @earendil-works/pi-coding-agent
  - type: config_file
    path: /home/user/.tmux.conf
    content: "..."

Pros: Fast builds (base images cached), reusable, versioned
Cons: Requires image registry management, layer ordering complexity

Direction C: Compose-First with Dockerfile Overrides

Treat compose_template as the primary definition. For simple cases, use a pre-built image. For custom cases, allow an inline Dockerfile or a build section in the compose:

services:
  app:
    build:
      context: .
      dockerfile_inline: |
        FROM ubuntu:24.04
        ...
    stdin_open: true
    tty: true
    working_dir: /workspace
    volumes:
      - ${REPO_PATH}:/workspace
      - ${SSH_PATH}:/home/user/.ssh:ro
    user: "${CONTAINER_USER:-user}"

Pros: Leverages Docker Compose native features, familiar to users
Cons: Still string-based; inline Dockerfiles are hard to edit

Direction D: Permission-Aware Mount Schema

Decouple mount declaration from mount implementation:

class MountPolicy:
    source: str          # host path
    target: str          # container path
    owner: str | None    # container user to own the mount
    permissions: str     # chmod string
    readonly: bool

At startup, the API runs a post-start "permission fixer" that applies all policies:

docker exec --user root <container> chown -R <owner> <target>
docker exec --user root <container> chmod <permissions> <target>

Pros: Systematic, works with any base image, configurable per mount
Cons: Adds startup latency, requires root to exist in container

---

Recommended Next Steps

1. Proposal phase: Evaluate Direction A (Declarative Manifests) vs Direction C (Compose-First) for the primary architecture
2. Design phase: Detail the manifest schema or compose enhancement, migration path, and API changes
3. Consider Direction D as a cross-cutting concern regardless of primary direction

---

Risks

• Migration risk: Existing dockerfile_template and compose_template columns need backward-compatible migration
• Build cache invalidation: Changing the build system may invalidate Docker layer caches
• User confusion: Adding a manifest layer on top of Dockerfiles may feel like "yet another abstraction"
• Scope creep: This touches tool types, tool configs, config profiles, compose generation, and the startup flow — high cross-cutting surface

---

Artifacts

• openspec/config.yaml — SDD configuration
• openspec/explorations/streamline-tool-container-definitions.md — This document