headquarter/openspec/proposals/streamline-tool-container-definitions.md

# SDD Proposal: Declarative Tool Container Compiler

## Status
**Phase:** proposal  
**Date:** 2026-05-28  
**Owner:** el Gentleman  
**Based on:** Exploration `streamline-tool-container-definitions`

---

## User Story

As a platform operator, I want to define a tool container by specifying:
- A **base image** (e.g. `ubuntu:24.04` or a pre-built `headquarter/base:dev-ubuntu`)
- A **list of packages** to install (apt, npm, pip, etc.)
- **Setup scripts** that run at build-time or container-startup
- **Mount policies** that automatically fix permissions for the runtime user

I do **not** want to write Dockerfiles or Compose files by hand.

The system should compile these declarations into Dockerfiles and Compose files automatically, while remaining fully compatible with the existing ConfigProfile mount system.

---

## Core Concept: The Tool Definition Manifest

Replace the monolithic `dockerfile_template` and `compose_template` strings with a single structured **Tool Definition Manifest**.

```yaml
# Tool Definition Manifest (stored as JSON in DB)
name: pi-agent
display_name: "Pi Agent"
description: "Terminal-based coding harness"
category: development
interface_type: terminal  # web | terminal

# ── Base Image ───────────────────────────────────────────────
base_image: ubuntu:24.04
# OR reference a pre-built base definition:
# base_definition_id: "base-ubuntu-24.04-dev"

# ── Packages ─────────────────────────────────────────────────
packages:
  apt:
    - curl
    - wget
    - git
    - neovim
    - ranger
    - tmux
    - htop
    - tree
    - jq
    - ca-certificates
    - python3
    - python3-pip
    - build-essential
  node:
    version: "20"           # triggers nodesource setup
  npm_global:
    - "@earendil-works/pi-coding-agent"
  # pip:
  #   - requests
  #   - httpx

# ── Runtime User ─────────────────────────────────────────────
user:
  name: user
  uid: 1001
  gid: 1001
  create_home: true
  shell: /bin/bash

# ── Environment ──────────────────────────────────────────────
env:
  DEBIAN_FRONTEND: noninteractive

# ── Setup Scripts ────────────────────────────────────────────
scripts:
  # build: runs during `docker build` → becomes RUN commands
  build:
    - |
      git config --global init.defaultBranch main
      git config --global user.email "dev@headquarter.local"
      git config --global user.name "Developer"
    - |
      mkdir -p /home/user/.config/ranger
      echo 'set preview_files true' > /home/user/.config/ranger/rc.conf

  # startup: runs when container starts → becomes entrypoint script
  startup:
    - |
      # Ensure workspace is owned by runtime user
      if [ -d /workspace ]; then
        sudo chown -R user:user /workspace 2>/dev/null || true
      fi

# ── Mount Schema ─────────────────────────────────────────────
mounts:
  - name: workspace
    target: /workspace
    source_type: repo        # resolved from repository path at instance creation
    writable: true
    owner: user              # post-start: chown -R user:user /workspace

  - name: ssh_keys
    target: /home/user/.ssh
    source_type: ssh_key     # resolved from repository's SSH key
    mode: "0700"             # post-start: chmod 0700 /home/user/.ssh
    file_mode: "0600"        # post-start: chmod 0600 files inside
    readonly: true

  - name: pi_state
    target: /tmp/.pi/agents
    source_type: instance    # resolved to {instance_dir}/mounts/tmp_.pi_agents
    writable: true

  - name: pi_config
    target: /home/user/.pi
    source_type: git_mount   # resolved from config profile git_mounts
    git_mount_ref: dotfiles   # references a named git mount in the config profile
    writable: true
    owner: user

# ── Runtime ──────────────────────────────────────────────────
runtime:
  command: ["/bin/bash"]
  stdin_open: true
  tty: true
  working_dir: /workspace
  # ports are auto-derived from interface_type:
  #   web: expose default_port
  #   terminal: no port mapping
```

---

## How It Compiles

### 1. Dockerfile Generation

The manifest compiler transforms the spec into a Dockerfile:

```dockerfile
# Generated Dockerfile — do not edit manually
FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive

# ── System Packages ──
RUN apt-get update && apt-get install -y \
    curl wget git neovim ranger tmux htop tree jq \
    ca-certificates python3 python3-pip build-essential \
    && rm -rf /var/lib/apt/lists/*

# ── Node.js ──
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
    && apt-get install -y nodejs \
    && rm -rf /var/lib/apt/lists/*

# ── NPM Packages ──
RUN npm install -g --ignore-scripts @earendil-works/pi-coding-agent

# ── Runtime User ──
RUN groupadd -g 1001 user && \
    useradd -u 1001 -g 1001 -m -s /bin/bash user

# ── Build Scripts ──
RUN git config --global init.defaultBranch main && \
    git config --global user.email "dev@headquarter.local" && \
    git config --global user.name "Developer"

RUN mkdir -p /home/user/.config/ranger && \
    echo 'set preview_files true' > /home/user/.config/ranger/rc.conf

# ── Environment ──
ENV DEBIAN_FRONTEND=noninteractive

# ── Setup Directories ──
RUN mkdir -p /workspace /tmp/.pi/agents /home/user/.pi && \
    chown -R user:user /workspace /tmp/.pi /home/user/.pi

# ── Entrypoint for Startup Scripts ──
COPY .headquarter/entrypoint.sh /usr/local/bin/headquarter-entrypoint
RUN chmod +x /usr/local/bin/headquarter-entrypoint

USER user
WORKDIR /home/user

ENTRYPOINT ["/usr/local/bin/headquarter-entrypoint"]
CMD ["/bin/bash"]
```

The generated `entrypoint.sh`:

```bash
#!/bin/bash
set -e

# Run startup scripts
echo "Ensure workspace is owned by runtime user"
if [ -d /workspace ]; then
  sudo chown -R user:user /workspace 2>/dev/null || true
fi

# Pass through to the main command
exec "$@"
```

### 2. Compose File Generation

The manifest compiler also generates the Compose file:

```yaml
services:
  app:
    image: ${IMAGE_TAG}
    container_name: ${INSTANCE_NAME}
    stdin_open: true
    tty: true
    working_dir: /workspace
    user: "1001:1001"  # from manifest user.uid/gid
    volumes:
      - ${REPO_PATH}:/workspace
      - ${SSH_PATH}:/home/user/.ssh:ro
      - ${INSTANCE_DIR}/mounts/tmp_.pi_agents:/tmp/.pi/agents
      - ${GIT_MOUNT_dotfiles}:/home/user/.pi
    environment:
      DEBIAN_FRONTEND: noninteractive
    restart: unless-stopped
```

Variables are resolved at instance creation time:
- `${REPO_PATH}` → the repository working directory
- `${SSH_PATH}` → prepared SSH key directory
- `${INSTANCE_DIR}` → the instance working directory
- `${GIT_MOUNT_dotfiles}` → resolved from config profile git_mounts

### 3. Permission Fixer (Post-Start)

After `docker compose up`, the API iterates over the mount schema and applies permission policies:

```python
for mount in manifest.mounts:
    if mount.owner:
        docker_exec(f"chown -R {mount.owner}:{mount.owner} {mount.target}")
    if mount.mode:
        docker_exec(f"chmod {mount.mode} {mount.target}")
    if mount.file_mode:
        docker_exec(f"find {mount.target} -type f -exec chmod {mount.file_mode} {{}} +")
```

This is **systematic and configurable** — not hardcoded to `/workspace`.

---

## Config Profile Compatibility

The existing ConfigProfile system provides:
- `environment_variables` → merged into compose `environment`
- `files` → written to instance dir, mounted via `volumes`
- `mounts` → appended to compose `volumes`
- `git_mounts` → resolved to host paths, appended to compose `volumes`
- `hints.start_command` → overrides `runtime.command`
- `hints.working_directory` → overrides `runtime.working_dir`

With the manifest system, ConfigProfiles **extend** the default mount schema:

1. Tool manifest defines the **default mount schema** (workspace, ssh, state)
2. ConfigProfile can add **additional mounts** or **override runtime hints**
3. Both are merged at instance-start time into the final compose file

The merge precedence:
1. Tool manifest (defaults)
2. ToolConfig overrides (per-user per-tool settings)
3. ConfigProfile overrides (hierarchical, can inherit from parent)
4. User-provided start options (e.g. branch selection)

---

## Base Image Definitions

A base definition is itself a manifest with no `runtime` section:

```yaml
# Base Definition: "ubuntu-24.04-dev"
name: ubuntu-24.04-dev
description: "Ubuntu 24.04 with build tools"
base_image: ubuntu:24.04
packages:
  apt:
    - curl
    - wget
    - git
    - build-essential
    - ca-certificates
user:
  name: user
  uid: 1000
  gid: 1000
  create_home: true
```

A tool definition can reference it:

```yaml
base_definition_id: "ubuntu-24.04-dev"
packages:
  apt:
    - neovim
    - ranger
    - tmux
  node:
    version: "20"
```

The compiler **merges** the base definition with the tool-specific overrides:
- Packages are **unioned** (base apt + tool apt)
- Scripts are **appended** (base build scripts, then tool build scripts)
- User/env are **overridden** (tool wins)

This enables a family of tool types to share a common base.

---

## Database Schema (Proposed)

### New Table: `tool_definition_manifests`

| Column | Type | Description |
|--------|------|-------------|
| `id` | UUID | PK |
| `name` | str | Unique identifier |
| `display_name` | str | Human-readable |
| `description` | str | |
| `category` | str | development, data-science, etc. |
| `interface_type` | enum | web, terminal |
| `base_image` | str | e.g. `ubuntu:24.04` |
| `base_definition_id` | UUID? | FK to another manifest |
| `manifest` | JSONB | The full manifest JSON |
| `dockerfile_cache` | TEXT | Last generated Dockerfile (for inspection) |
| `created_at` | datetime | |
| `updated_at` | datetime | |

### Migration: `tool_types` table

Add a nullable `manifest_id` column to `tool_types`.

For backward compatibility:
- If `manifest_id` is set → use the new manifest system
- If `manifest_id` is null → fall back to `dockerfile_template` / `compose_template`

A data migration converts existing pi-agent to the new manifest format.

---

## API Changes

### New Endpoints

```
GET    /tool-definitions              → list all base/tool definitions
GET    /tool-definitions/{id}         → get a definition
POST   /tool-definitions              → create a new definition
PUT    /tool-definitions/{id}         → update a definition
DELETE /tool-definitions/{id}         → delete (if not in use)
POST   /tool-definitions/{id}/compile → preview generated Dockerfile + compose
```

### Modified Endpoints

```
POST   /tool-types                    → can now accept manifest_id instead of templates
GET    /tool-types/{id}               → includes manifest if available
```

### Frontend Changes

New UI page: **Tool Workshop**

- Base image selector (dropdown of existing bases or custom FROM)
- Package manager tabs (apt, npm, pip, etc.)
- Script editor (build vs startup)
- Mount schema designer (drag-drop or form)
- Live preview of generated Dockerfile
- Test build button (builds image and reports success/failure)

---

## Open Questions

1. **Should we support multi-stage builds?**
   - Pros: smaller images, separation of build deps from runtime
   - Cons: more complexity in the manifest schema

2. **Should base definitions be versioned?**
   - Pros: reproducible builds, safe updates
   - Cons: more DB complexity

3. **How do we handle binary build context files?**
   - Current: JSON text in DB
   - Option A: Store in filesystem, reference by path
   - Option B: Upload to object storage (S3/minio)

4. **Should generated images be cached/pushed to a registry?**
   - Currently: built locally per instance
   - Option: push to `headquarter/tools/{tool-name}:{hash}` for reuse

---

## Risks

| Risk | Severity | Mitigation |
|------|----------|------------|
| Migration complexity | Medium | Keep old fields nullable; gradual adoption |
| Build cache invalidation | Medium | Use deterministic Dockerfile generation; hash manifest for image tag |
| User confusion ("yet another abstraction") | Low | Provide live preview + "view generated Dockerfile" button |
| Scope creep into full CI/CD | High | Keep scope to container definition only; no pipeline/orchestration |
| Binary files in manifests | Medium | Limit build context to text; document workaround for binaries |

---

## Effort Estimate

| Phase | Files | Lines (est) | Complexity |
|-------|-------|-------------|------------|
| DB migration + models | 3 | 200 | Low |
| Manifest compiler (Dockerfile) | 2 | 400 | Medium |
| Manifest compiler (Compose) | 2 | 300 | Medium |
| Permission fixer refactor | 2 | 200 | Low |
| API endpoints | 3 | 400 | Medium |
| Frontend Tool Workshop | 8 | 1200 | High |
| Tests | 4 | 600 | Medium |
| **Total** | **24** | **~3300** | **High** |

**Review workload forecast:** ~3300 lines is well above the 400-line budget. This should be split into **chained PRs**:
1. Backend: manifest schema, compiler, API (PR 1)
2. Frontend: Tool Workshop UI (PR 2)
3. Migration + data conversion (PR 3)

---

## Next Recommended Phase

**Design** — Detail the manifest JSON schema, compiler internals, and migration plan.

Should I proceed to design?