feat: tool definition manifest system (PR 1)

- Add ToolDefinitionManifest model with base image versioning - Add manifest compiler: Dockerfile + Compose generation from JSON manifests - Add permission fixer: post-start chown/chmod for mount policies - Add tool definition CRUD API with live compile preview endpoint - Integrate manifest-based startup flow in start_instance - Add Alembic migration with data conversion for pi-agent - Add 48 unit tests for manifest compiler, permission fixer, docker service - Keep backward compatibility with legacy dockerfile_template/compose_template Migration: applied successfully. Pi-agent converted to manifest. Quality gates: pytest (146 passed, 4 pre-existing unrelated failures)
2026-05-28 13:26:54 +02:00
parent 314ba3aee4
commit 5deee8c65c
20 changed files with 4211 additions and 69 deletions
@@ -0,0 +1,447 @@
+# SDD Proposal: Declarative Tool Container Compiler
+
+## Status
+**Phase:** proposal  
+**Date:** 2026-05-28  
+**Owner:** el Gentleman  
+**Based on:** Exploration `streamline-tool-container-definitions`
+
+---
+
+## User Story
+
+As a platform operator, I want to define a tool container by specifying:
+- A **base image** (e.g. `ubuntu:24.04` or a pre-built `headquarter/base:dev-ubuntu`)
+- A **list of packages** to install (apt, npm, pip, etc.)
+- **Setup scripts** that run at build-time or container-startup
+- **Mount policies** that automatically fix permissions for the runtime user
+
+I do **not** want to write Dockerfiles or Compose files by hand.
+
+The system should compile these declarations into Dockerfiles and Compose files automatically, while remaining fully compatible with the existing ConfigProfile mount system.
+
+---
+
+## Core Concept: The Tool Definition Manifest
+
+Replace the monolithic `dockerfile_template` and `compose_template` strings with a single structured **Tool Definition Manifest**.
+
+```yaml
+# Tool Definition Manifest (stored as JSON in DB)
+name: pi-agent
+display_name: "Pi Agent"
+description: "Terminal-based coding harness"
+category: development
+interface_type: terminal  # web | terminal
+
+# ── Base Image ───────────────────────────────────────────────
+base_image: ubuntu:24.04
+# OR reference a pre-built base definition:
+# base_definition_id: "base-ubuntu-24.04-dev"
+
+# ── Packages ─────────────────────────────────────────────────
+packages:
+  apt:
+    - curl
+    - wget
+    - git
+    - neovim
+    - ranger
+    - tmux
+    - htop
+    - tree
+    - jq
+    - ca-certificates
+    - python3
+    - python3-pip
+    - build-essential
+  node:
+    version: "20"           # triggers nodesource setup
+  npm_global:
+    - "@earendil-works/pi-coding-agent"
+  # pip:
+  #   - requests
+  #   - httpx
+
+# ── Runtime User ─────────────────────────────────────────────
+user:
+  name: user
+  uid: 1001
+  gid: 1001
+  create_home: true
+  shell: /bin/bash
+
+# ── Environment ──────────────────────────────────────────────
+env:
+  DEBIAN_FRONTEND: noninteractive
+
+# ── Setup Scripts ────────────────────────────────────────────
+scripts:
+  # build: runs during `docker build` → becomes RUN commands
+  build:
+    - |
+      git config --global init.defaultBranch main
+      git config --global user.email "dev@headquarter.local"
+      git config --global user.name "Developer"
+    - |
+      mkdir -p /home/user/.config/ranger
+      echo 'set preview_files true' > /home/user/.config/ranger/rc.conf
+
+  # startup: runs when container starts → becomes entrypoint script
+  startup:
+    - |
+      # Ensure workspace is owned by runtime user
+      if [ -d /workspace ]; then
+        sudo chown -R user:user /workspace 2>/dev/null || true
+      fi
+
+# ── Mount Schema ─────────────────────────────────────────────
+mounts:
+  - name: workspace
+    target: /workspace
+    source_type: repo        # resolved from repository path at instance creation
+    writable: true
+    owner: user              # post-start: chown -R user:user /workspace
+
+  - name: ssh_keys
+    target: /home/user/.ssh
+    source_type: ssh_key     # resolved from repository's SSH key
+    mode: "0700"             # post-start: chmod 0700 /home/user/.ssh
+    file_mode: "0600"        # post-start: chmod 0600 files inside
+    readonly: true
+
+  - name: pi_state
+    target: /tmp/.pi/agents
+    source_type: instance    # resolved to {instance_dir}/mounts/tmp_.pi_agents
+    writable: true
+
+  - name: pi_config
+    target: /home/user/.pi
+    source_type: git_mount   # resolved from config profile git_mounts
+    git_mount_ref: dotfiles   # references a named git mount in the config profile
+    writable: true
+    owner: user
+
+# ── Runtime ──────────────────────────────────────────────────
+runtime:
+  command: ["/bin/bash"]
+  stdin_open: true
+  tty: true
+  working_dir: /workspace
+  # ports are auto-derived from interface_type:
+  #   web: expose default_port
+  #   terminal: no port mapping
+```
+
+---
+
+## How It Compiles
+
+### 1. Dockerfile Generation
+
+The manifest compiler transforms the spec into a Dockerfile:
+
+```dockerfile
+# Generated Dockerfile — do not edit manually
+FROM ubuntu:24.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# ── System Packages ──
+RUN apt-get update && apt-get install -y \
+    curl wget git neovim ranger tmux htop tree jq \
+    ca-certificates python3 python3-pip build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# ── Node.js ──
+RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
+    && apt-get install -y nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+# ── NPM Packages ──
+RUN npm install -g --ignore-scripts @earendil-works/pi-coding-agent
+
+# ── Runtime User ──
+RUN groupadd -g 1001 user && \
+    useradd -u 1001 -g 1001 -m -s /bin/bash user
+
+# ── Build Scripts ──
+RUN git config --global init.defaultBranch main && \
+    git config --global user.email "dev@headquarter.local" && \
+    git config --global user.name "Developer"
+
+RUN mkdir -p /home/user/.config/ranger && \
+    echo 'set preview_files true' > /home/user/.config/ranger/rc.conf
+
+# ── Environment ──
+ENV DEBIAN_FRONTEND=noninteractive
+
+# ── Setup Directories ──
+RUN mkdir -p /workspace /tmp/.pi/agents /home/user/.pi && \
+    chown -R user:user /workspace /tmp/.pi /home/user/.pi
+
+# ── Entrypoint for Startup Scripts ──
+COPY .headquarter/entrypoint.sh /usr/local/bin/headquarter-entrypoint
+RUN chmod +x /usr/local/bin/headquarter-entrypoint
+
+USER user
+WORKDIR /home/user
+
+ENTRYPOINT ["/usr/local/bin/headquarter-entrypoint"]
+CMD ["/bin/bash"]
+```
+
+The generated `entrypoint.sh`:
+
+```bash
+#!/bin/bash
+set -e
+
+# Run startup scripts
+echo "Ensure workspace is owned by runtime user"
+if [ -d /workspace ]; then
+  sudo chown -R user:user /workspace 2>/dev/null || true
+fi
+
+# Pass through to the main command
+exec "$@"
+```
+
+### 2. Compose File Generation
+
+The manifest compiler also generates the Compose file:
+
+```yaml
+services:
+  app:
+    image: ${IMAGE_TAG}
+    container_name: ${INSTANCE_NAME}
+    stdin_open: true
+    tty: true
+    working_dir: /workspace
+    user: "1001:1001"  # from manifest user.uid/gid
+    volumes:
+      - ${REPO_PATH}:/workspace
+      - ${SSH_PATH}:/home/user/.ssh:ro
+      - ${INSTANCE_DIR}/mounts/tmp_.pi_agents:/tmp/.pi/agents
+      - ${GIT_MOUNT_dotfiles}:/home/user/.pi
+    environment:
+      DEBIAN_FRONTEND: noninteractive
+    restart: unless-stopped
+```
+
+Variables are resolved at instance creation time:
+- `${REPO_PATH}` → the repository working directory
+- `${SSH_PATH}` → prepared SSH key directory
+- `${INSTANCE_DIR}` → the instance working directory
+- `${GIT_MOUNT_dotfiles}` → resolved from config profile git_mounts
+
+### 3. Permission Fixer (Post-Start)
+
+After `docker compose up`, the API iterates over the mount schema and applies permission policies:
+
+```python
+for mount in manifest.mounts:
+    if mount.owner:
+        docker_exec(f"chown -R {mount.owner}:{mount.owner} {mount.target}")
+    if mount.mode:
+        docker_exec(f"chmod {mount.mode} {mount.target}")
+    if mount.file_mode:
+        docker_exec(f"find {mount.target} -type f -exec chmod {mount.file_mode} {{}} +")
+```
+
+This is **systematic and configurable** — not hardcoded to `/workspace`.
+
+---
+
+## Config Profile Compatibility
+
+The existing ConfigProfile system provides:
+- `environment_variables` → merged into compose `environment`
+- `files` → written to instance dir, mounted via `volumes`
+- `mounts` → appended to compose `volumes`
+- `git_mounts` → resolved to host paths, appended to compose `volumes`
+- `hints.start_command` → overrides `runtime.command`
+- `hints.working_directory` → overrides `runtime.working_dir`
+
+With the manifest system, ConfigProfiles **extend** the default mount schema:
+
+1. Tool manifest defines the **default mount schema** (workspace, ssh, state)
+2. ConfigProfile can add **additional mounts** or **override runtime hints**
+3. Both are merged at instance-start time into the final compose file
+
+The merge precedence:
+1. Tool manifest (defaults)
+2. ToolConfig overrides (per-user per-tool settings)
+3. ConfigProfile overrides (hierarchical, can inherit from parent)
+4. User-provided start options (e.g. branch selection)
+
+---
+
+## Base Image Definitions
+
+A base definition is itself a manifest with no `runtime` section:
+
+```yaml
+# Base Definition: "ubuntu-24.04-dev"
+name: ubuntu-24.04-dev
+description: "Ubuntu 24.04 with build tools"
+base_image: ubuntu:24.04
+packages:
+  apt:
+    - curl
+    - wget
+    - git
+    - build-essential
+    - ca-certificates
+user:
+  name: user
+  uid: 1000
+  gid: 1000
+  create_home: true
+```
+
+A tool definition can reference it:
+
+```yaml
+base_definition_id: "ubuntu-24.04-dev"
+packages:
+  apt:
+    - neovim
+    - ranger
+    - tmux
+  node:
+    version: "20"
+```
+
+The compiler **merges** the base definition with the tool-specific overrides:
+- Packages are **unioned** (base apt + tool apt)
+- Scripts are **appended** (base build scripts, then tool build scripts)
+- User/env are **overridden** (tool wins)
+
+This enables a family of tool types to share a common base.
+
+---
+
+## Database Schema (Proposed)
+
+### New Table: `tool_definition_manifests`
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `id` | UUID | PK |
+| `name` | str | Unique identifier |
+| `display_name` | str | Human-readable |
+| `description` | str | |
+| `category` | str | development, data-science, etc. |
+| `interface_type` | enum | web, terminal |
+| `base_image` | str | e.g. `ubuntu:24.04` |
+| `base_definition_id` | UUID? | FK to another manifest |
+| `manifest` | JSONB | The full manifest JSON |
+| `dockerfile_cache` | TEXT | Last generated Dockerfile (for inspection) |
+| `created_at` | datetime | |
+| `updated_at` | datetime | |
+
+### Migration: `tool_types` table
+
+Add a nullable `manifest_id` column to `tool_types`.
+
+For backward compatibility:
+- If `manifest_id` is set → use the new manifest system
+- If `manifest_id` is null → fall back to `dockerfile_template` / `compose_template`
+
+A data migration converts existing pi-agent to the new manifest format.
+
+---
+
+## API Changes
+
+### New Endpoints
+
+```
+GET    /tool-definitions              → list all base/tool definitions
+GET    /tool-definitions/{id}         → get a definition
+POST   /tool-definitions              → create a new definition
+PUT    /tool-definitions/{id}         → update a definition
+DELETE /tool-definitions/{id}         → delete (if not in use)
+POST   /tool-definitions/{id}/compile → preview generated Dockerfile + compose
+```
+
+### Modified Endpoints
+
+```
+POST   /tool-types                    → can now accept manifest_id instead of templates
+GET    /tool-types/{id}               → includes manifest if available
+```
+
+### Frontend Changes
+
+New UI page: **Tool Workshop**
+
+- Base image selector (dropdown of existing bases or custom FROM)
+- Package manager tabs (apt, npm, pip, etc.)
+- Script editor (build vs startup)
+- Mount schema designer (drag-drop or form)
+- Live preview of generated Dockerfile
+- Test build button (builds image and reports success/failure)
+
+---
+
+## Open Questions
+
+1. **Should we support multi-stage builds?**
+   - Pros: smaller images, separation of build deps from runtime
+   - Cons: more complexity in the manifest schema
+
+2. **Should base definitions be versioned?**
+   - Pros: reproducible builds, safe updates
+   - Cons: more DB complexity
+
+3. **How do we handle binary build context files?**
+   - Current: JSON text in DB
+   - Option A: Store in filesystem, reference by path
+   - Option B: Upload to object storage (S3/minio)
+
+4. **Should generated images be cached/pushed to a registry?**
+   - Currently: built locally per instance
+   - Option: push to `headquarter/tools/{tool-name}:{hash}` for reuse
+
+---
+
+## Risks
+
+| Risk | Severity | Mitigation |
+|------|----------|------------|
+| Migration complexity | Medium | Keep old fields nullable; gradual adoption |
+| Build cache invalidation | Medium | Use deterministic Dockerfile generation; hash manifest for image tag |
+| User confusion ("yet another abstraction") | Low | Provide live preview + "view generated Dockerfile" button |
+| Scope creep into full CI/CD | High | Keep scope to container definition only; no pipeline/orchestration |
+| Binary files in manifests | Medium | Limit build context to text; document workaround for binaries |
+
+---
+
+## Effort Estimate
+
+| Phase | Files | Lines (est) | Complexity |
+|-------|-------|-------------|------------|
+| DB migration + models | 3 | 200 | Low |
+| Manifest compiler (Dockerfile) | 2 | 400 | Medium |
+| Manifest compiler (Compose) | 2 | 300 | Medium |
+| Permission fixer refactor | 2 | 200 | Low |
+| API endpoints | 3 | 400 | Medium |
+| Frontend Tool Workshop | 8 | 1200 | High |
+| Tests | 4 | 600 | Medium |
+| **Total** | **24** | **~3300** | **High** |
+
+**Review workload forecast:** ~3300 lines is well above the 400-line budget. This should be split into **chained PRs**:
+1. Backend: manifest schema, compiler, API (PR 1)
+2. Frontend: Tool Workshop UI (PR 2)
+3. Migration + data conversion (PR 3)
+
+---
+
+## Next Recommended Phase
+
+**Design** — Detail the manifest JSON schema, compiler internals, and migration plan.
+
+Should I proceed to design?