headquarter/openspec/proposals/streamline-tool-container-definitions.md

SDD Proposal: Declarative Tool Container Compiler

Status

Phase: proposal
Date: 2026-05-28
Owner: el Gentleman
Based on: Exploration streamline-tool-container-definitions

---

User Story

As a platform operator, I want to define a tool container by specifying:

• A base image (e.g. ubuntu:24.04 or a pre-built headquarter/base:dev-ubuntu)
• A list of packages to install (apt, npm, pip, etc.)
• Setup scripts that run at build-time or container-startup
• Mount policies that automatically fix permissions for the runtime user

I do not want to write Dockerfiles or Compose files by hand.

The system should compile these declarations into Dockerfiles and Compose files automatically, while remaining fully compatible with the existing ConfigProfile mount system.

---

Core Concept: The Tool Definition Manifest

Replace the monolithic dockerfile_template and compose_template strings with a single structured Tool Definition Manifest.

# Tool Definition Manifest (stored as JSON in DB)
name: pi-agent
display_name: "Pi Agent"
description: "Terminal-based coding harness"
category: development
interface_type: terminal  # web | terminal

# ── Base Image ───────────────────────────────────────────────
base_image: ubuntu:24.04
# OR reference a pre-built base definition:
# base_definition_id: "base-ubuntu-24.04-dev"

# ── Packages ─────────────────────────────────────────────────
packages:
  apt:
    - curl
    - wget
    - git
    - neovim
    - ranger
    - tmux
    - htop
    - tree
    - jq
    - ca-certificates
    - python3
    - python3-pip
    - build-essential
  node:
    version: "20"           # triggers nodesource setup
  npm_global:
    - "@earendil-works/pi-coding-agent"
  # pip:
  #   - requests
  #   - httpx

# ── Runtime User ─────────────────────────────────────────────
user:
  name: user
  uid: 1001
  gid: 1001
  create_home: true
  shell: /bin/bash

# ── Environment ──────────────────────────────────────────────
env:
  DEBIAN_FRONTEND: noninteractive

# ── Setup Scripts ────────────────────────────────────────────
scripts:
  # build: runs during `docker build` → becomes RUN commands
  build:
    - |
      git config --global init.defaultBranch main
      git config --global user.email "dev@headquarter.local"
      git config --global user.name "Developer"
    - |
      mkdir -p /home/user/.config/ranger
      echo 'set preview_files true' > /home/user/.config/ranger/rc.conf

  # startup: runs when container starts → becomes entrypoint script
  startup:
    - |
      # Ensure workspace is owned by runtime user
      if [ -d /workspace ]; then
        sudo chown -R user:user /workspace 2>/dev/null || true
      fi

# ── Mount Schema ─────────────────────────────────────────────
mounts:
  - name: workspace
    target: /workspace
    source_type: repo        # resolved from repository path at instance creation
    writable: true
    owner: user              # post-start: chown -R user:user /workspace

  - name: ssh_keys
    target: /home/user/.ssh
    source_type: ssh_key     # resolved from repository's SSH key
    mode: "0700"             # post-start: chmod 0700 /home/user/.ssh
    file_mode: "0600"        # post-start: chmod 0600 files inside
    readonly: true

  - name: pi_state
    target: /tmp/.pi/agents
    source_type: instance    # resolved to {instance_dir}/mounts/tmp_.pi_agents
    writable: true

  - name: pi_config
    target: /home/user/.pi
    source_type: git_mount   # resolved from config profile git_mounts
    git_mount_ref: dotfiles   # references a named git mount in the config profile
    writable: true
    owner: user

# ── Runtime ──────────────────────────────────────────────────
runtime:
  command: ["/bin/bash"]
  stdin_open: true
  tty: true
  working_dir: /workspace
  # ports are auto-derived from interface_type:
  #   web: expose default_port
  #   terminal: no port mapping

---

How It Compiles

1. Dockerfile Generation

The manifest compiler transforms the spec into a Dockerfile:

# Generated Dockerfile — do not edit manually
FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive

# ── System Packages ──
RUN apt-get update && apt-get install -y \
    curl wget git neovim ranger tmux htop tree jq \
    ca-certificates python3 python3-pip build-essential \
    && rm -rf /var/lib/apt/lists/*

# ── Node.js ──
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
    && apt-get install -y nodejs \
    && rm -rf /var/lib/apt/lists/*

# ── NPM Packages ──
RUN npm install -g --ignore-scripts @earendil-works/pi-coding-agent

# ── Runtime User ──
RUN groupadd -g 1001 user && \
    useradd -u 1001 -g 1001 -m -s /bin/bash user

# ── Build Scripts ──
RUN git config --global init.defaultBranch main && \
    git config --global user.email "dev@headquarter.local" && \
    git config --global user.name "Developer"

RUN mkdir -p /home/user/.config/ranger && \
    echo 'set preview_files true' > /home/user/.config/ranger/rc.conf

# ── Environment ──
ENV DEBIAN_FRONTEND=noninteractive

# ── Setup Directories ──
RUN mkdir -p /workspace /tmp/.pi/agents /home/user/.pi && \
    chown -R user:user /workspace /tmp/.pi /home/user/.pi

# ── Entrypoint for Startup Scripts ──
COPY .headquarter/entrypoint.sh /usr/local/bin/headquarter-entrypoint
RUN chmod +x /usr/local/bin/headquarter-entrypoint

USER user
WORKDIR /home/user

ENTRYPOINT ["/usr/local/bin/headquarter-entrypoint"]
CMD ["/bin/bash"]

The generated entrypoint.sh:

#!/bin/bash
set -e

# Run startup scripts
echo "Ensure workspace is owned by runtime user"
if [ -d /workspace ]; then
  sudo chown -R user:user /workspace 2>/dev/null || true
fi

# Pass through to the main command
exec "$@"

2. Compose File Generation

The manifest compiler also generates the Compose file:

services:
  app:
    image: ${IMAGE_TAG}
    container_name: ${INSTANCE_NAME}
    stdin_open: true
    tty: true
    working_dir: /workspace
    user: "1001:1001"  # from manifest user.uid/gid
    volumes:
      - ${REPO_PATH}:/workspace
      - ${SSH_PATH}:/home/user/.ssh:ro
      - ${INSTANCE_DIR}/mounts/tmp_.pi_agents:/tmp/.pi/agents
      - ${GIT_MOUNT_dotfiles}:/home/user/.pi
    environment:
      DEBIAN_FRONTEND: noninteractive
    restart: unless-stopped

Variables are resolved at instance creation time:

• ${REPO_PATH} → the repository working directory
• ${SSH_PATH} → prepared SSH key directory
• ${INSTANCE_DIR} → the instance working directory
• ${GIT_MOUNT_dotfiles} → resolved from config profile git_mounts

3. Permission Fixer (Post-Start)

After docker compose up, the API iterates over the mount schema and applies permission policies:

for mount in manifest.mounts:
    if mount.owner:
        docker_exec(f"chown -R {mount.owner}:{mount.owner} {mount.target}")
    if mount.mode:
        docker_exec(f"chmod {mount.mode} {mount.target}")
    if mount.file_mode:
        docker_exec(f"find {mount.target} -type f -exec chmod {mount.file_mode} {{}} +")

This is systematic and configurable — not hardcoded to /workspace.

---

Config Profile Compatibility

The existing ConfigProfile system provides:

• environment_variables → merged into compose environment
• files → written to instance dir, mounted via volumes
• mounts → appended to compose volumes
• git_mounts → resolved to host paths, appended to compose volumes
• hints.start_command → overrides runtime.command
• hints.working_directory → overrides runtime.working_dir

With the manifest system, ConfigProfiles extend the default mount schema:

1. Tool manifest defines the default mount schema (workspace, ssh, state)
2. ConfigProfile can add additional mounts or override runtime hints
3. Both are merged at instance-start time into the final compose file

The merge precedence:

1. Tool manifest (defaults)
2. ToolConfig overrides (per-user per-tool settings)
3. ConfigProfile overrides (hierarchical, can inherit from parent)
4. User-provided start options (e.g. branch selection)

---

Base Image Definitions

A base definition is itself a manifest with no runtime section:

# Base Definition: "ubuntu-24.04-dev"
name: ubuntu-24.04-dev
description: "Ubuntu 24.04 with build tools"
base_image: ubuntu:24.04
packages:
  apt:
    - curl
    - wget
    - git
    - build-essential
    - ca-certificates
user:
  name: user
  uid: 1000
  gid: 1000
  create_home: true

A tool definition can reference it:

base_definition_id: "ubuntu-24.04-dev"
packages:
  apt:
    - neovim
    - ranger
    - tmux
  node:
    version: "20"

The compiler merges the base definition with the tool-specific overrides:

• Packages are unioned (base apt + tool apt)
• Scripts are appended (base build scripts, then tool build scripts)
• User/env are overridden (tool wins)

This enables a family of tool types to share a common base.

---

Database Schema (Proposed)

New Table: tool_definition_manifests

Column	Type	Description
id	UUID	PK
name	str	Unique identifier
display_name	str	Human-readable
description	str
category	str	development, data-science, etc.
interface_type	enum	web, terminal
base_image	str	e.g. ubuntu:24.04
base_definition_id	UUID?	FK to another manifest
manifest	JSONB	The full manifest JSON
dockerfile_cache	TEXT	Last generated Dockerfile (for inspection)
created_at	datetime
updated_at	datetime

Migration: tool_types table

Add a nullable manifest_id column to tool_types.

For backward compatibility:

• If manifest_id is set → use the new manifest system
• If manifest_id is null → fall back to dockerfile_template / compose_template

A data migration converts existing pi-agent to the new manifest format.

---

API Changes

New Endpoints

GET    /tool-definitions              → list all base/tool definitions
GET    /tool-definitions/{id}         → get a definition
POST   /tool-definitions              → create a new definition
PUT    /tool-definitions/{id}         → update a definition
DELETE /tool-definitions/{id}         → delete (if not in use)
POST   /tool-definitions/{id}/compile → preview generated Dockerfile + compose

Modified Endpoints

POST   /tool-types                    → can now accept manifest_id instead of templates
GET    /tool-types/{id}               → includes manifest if available

Frontend Changes

New UI page: Tool Workshop

• Base image selector (dropdown of existing bases or custom FROM)
• Package manager tabs (apt, npm, pip, etc.)
• Script editor (build vs startup)
• Mount schema designer (drag-drop or form)
• Live preview of generated Dockerfile
• Test build button (builds image and reports success/failure)

---

Open Questions

1. Should we support multi-stage builds?

   • Pros: smaller images, separation of build deps from runtime
   • Cons: more complexity in the manifest schema

2. Should base definitions be versioned?

   • Pros: reproducible builds, safe updates
   • Cons: more DB complexity

3. How do we handle binary build context files?

   • Current: JSON text in DB
   • Option A: Store in filesystem, reference by path
   • Option B: Upload to object storage (S3/minio)

4. Should generated images be cached/pushed to a registry?

   • Currently: built locally per instance
   • Option: push to headquarter/tools/{tool-name}:{hash} for reuse

---

Risks

Risk	Severity	Mitigation
Migration complexity	Medium	Keep old fields nullable; gradual adoption
Build cache invalidation	Medium	Use deterministic Dockerfile generation; hash manifest for image tag
User confusion ("yet another abstraction")	Low	Provide live preview + "view generated Dockerfile" button
Scope creep into full CI/CD	High	Keep scope to container definition only; no pipeline/orchestration
Binary files in manifests	Medium	Limit build context to text; document workaround for binaries

---

Effort Estimate

Phase	Files	Lines (est)	Complexity
DB migration + models	3	200	Low
Manifest compiler (Dockerfile)	2	400	Medium
Manifest compiler (Compose)	2	300	Medium
Permission fixer refactor	2	200	Low
API endpoints	3	400	Medium
Frontend Tool Workshop	8	1200	High
Tests	4	600	Medium
Total	24	~3300	High

Review workload forecast: ~3300 lines is well above the 400-line budget. This should be split into chained PRs:

1. Backend: manifest schema, compiler, API (PR 1)
2. Frontend: Tool Workshop UI (PR 2)
3. Migration + data conversion (PR 3)

---

Next Recommended Phase

Design — Detail the manifest JSON schema, compiler internals, and migration plan.

Should I proceed to design?