feat: tool definition manifest system (PR 1)

- Add ToolDefinitionManifest model with base image versioning - Add manifest compiler: Dockerfile + Compose generation from JSON manifests - Add permission fixer: post-start chown/chmod for mount policies - Add tool definition CRUD API with live compile preview endpoint - Integrate manifest-based startup flow in start_instance - Add Alembic migration with data conversion for pi-agent - Add 48 unit tests for manifest compiler, permission fixer, docker service - Keep backward compatibility with legacy dockerfile_template/compose_template Migration: applied successfully. Pi-agent converted to manifest. Quality gates: pytest (146 passed, 4 pre-existing unrelated failures)
2026-05-28 13:26:54 +02:00
parent 314ba3aee4
commit 5deee8c65c
20 changed files with 4211 additions and 69 deletions
@@ -0,0 +1,362 @@
+# SDD Exploration: Streamline Tool Container Definitions
+
+## Status
+**Phase:** explore  
+**Date:** 2026-05-28  
+**Owner:** el Gentleman (parent session)  
+**Scope:** Tool container definition, build, and mount system
+
+---
+
+## Executive Summary
+
+The current tool container system works for the happy path (pi-agent on Ubuntu) but has deep structural inflexibility:
+
+1. **Monolithic Dockerfile strings** in the database — impossible to review, version, or compose
+2. **Ad-hoc compose generation** — string formatting with hardcoded fields (`stdin_open`, `tty`, `working_dir` missing for dockerfile types)
+3. **Hardcoded mount paths** — `/workspace` and `/root/.ssh` don't adapt to the container's runtime user
+4. **No package/base-image modularity** — every tool type carries a full Dockerfile copy
+5. **Permission mismatch** — bind mounts come in as root-owned; non-root container users can't write
+6. **Config overlap** — tool configs, config profiles, and compose templates fight for control of the same fields
+
+This exploration proposes a **layered, declarative container definition system** where tool types compose from reusable base images, mount schemas, and permission policies.
+
+---
+
+## Current Architecture Map
+
+### Data Model
+
+```
+ToolType (DB table)
+  ├── name, display_name, description, category
+  ├── interface_type: "web" | "terminal"
+  ├── definition_type: "dockerfile" | "compose"
+  ├── dockerfile_template: TEXT (giant Dockerfile string)
+  ├── compose_template: TEXT (Jinja-like {{VAR}} string)
+  ├── build_context: JSON {path: content}
+  ├── required_variables: JSON ["REPO_PATH", ...]
+  ├── default_port: int
+  └── readiness_probe: JSON
+
+ToolInstance (DB table)
+  ├── name, display_name, status
+  ├── tool_type_id → ToolType
+  ├── repository_id → GitRepository
+  ├── compose_path: str
+  ├── container_id, container_name
+  ├── port, url, public_url, tunnel_id
+  ├── clone_mode: "mount" | "clone"
+  ├── branch, new_branch
+  └── selected_config_profile_id → ConfigProfile
+
+ToolConfig (DB table, per-user per-tool-type)
+  ├── config_type: "env" | "file"
+  ├── key, value, file_path
+  ├── port_override, start_command, working_directory
+  ├── environment_variables: JSON
+  └── volumes: JSON [{source, target, type}]
+
+ConfigProfile (DB table)
+  ├── name, description
+  ├── user_id, project_id, tool_type_id
+  ├── environment_variables: JSON
+  ├── files: JSON {path: content}
+  ├── mounts: JSON [{source, target, type}]
+  ├── git_mounts: JSON [{remote_url, source_path, target_path, branch}]
+  └── parent_profile_id → ConfigProfile (hierarchy)
+```
+
+### Creation Flow (`create_instance`)
+
+```
+POST /projects/{id}/repositories/{id}/instances
+  → validate tool_type, repo, config_profile
+  → generate instance_name = "{tool_type}-{repo}-{uuid8}"
+  → ensure_instance_directory(instance_name)
+  → find_free_port()
+  → determine repo_path (mount = repo.path; clone = clone_repository())
+  → IF tool_type.definition_type == "dockerfile":
+       build_image(instance_dir, dockerfile_template, tag, build_context)
+       generate compose_content (HARDCODED STRING FORMATTING)
+     ELSE:
+       render_compose_template(tool_type.compose_template, variables)
+  → write_compose_file()
+  → create ToolInstance DB record (status="pending")
+```
+
+### Startup Flow (`start_instance`)
+
+```
+POST /instances/{id}/start
+  → fetch ToolConfigs (env, files, port_override, start_command, working_dir, volumes)
+  → IF selected_config_profile:
+       resolve_profile() → env, files, mounts, git_mounts, hints
+  → write .env file, config files
+  → IF clone_mode: mount SSH keys at /root/.ssh (HARDCODED)
+  → _modify_compose_file(port, command, working_dir, extra_volumes)
+  → _sanitize_compose_file()
+  → execute_compose_command("up")
+  → get_container_id(instance.name)  ← CASE-SENSITIVE BUG (fixed)
+  → get_container_name(instance.name)
+  → connect_container_to_network("backend")
+  → wait_for_container_running()
+  → IF web: start_cloudflared_tunnel()
+  → instance.status = "running"
+```
+
+### Key Files
+
+| File | Responsibility |
+|------|---------------|
+| `apps/api/src/api/tool_instances.py` | create_instance, start_instance, stop_instance, restart_instance, proxy, logs |
+| `apps/api/src/api/tool_types.py` | CRUD for ToolType (DB strings) |
+| `apps/api/src/services/docker.py` | compose execution, container queries, tunnel management |
+| `apps/api/src/services/docker_build.py` | `docker build` wrapper |
+| `apps/api/src/services/terminal_session.py` | PTY-based terminal over `docker exec` |
+| `apps/api/src/services/terminal_manager.py` | WebSocket ↔ terminal session lifecycle |
+| `apps/api/src/models/tool_type.py` | SQLAlchemy model |
+
+---
+
+## Pain Points (Detailed)
+
+### 1. Monolithic Dockerfile Templates
+
+The pi-agent Dockerfile template is a 40-line string stored in the DB migration:
+
+```sql
+INSERT INTO tool_types (... dockerfile_template ...)
+VALUES ('...# Pi Coding Agent - Terminal-based coding harness\nFROM ubuntu:24.04\n...')
+```
+
+**Problems:**
+- No syntax highlighting, linting, or `docker build` validation at edit time
+- Every tool type copies the entire Dockerfile; no reuse of common layers
+- Changes require a DB migration
+- No way for users to customize packages without forking the whole template
+
+### 2. Ad-Hoc Compose Generation
+
+For `dockerfile` type tools, the compose is generated by Python f-string:
+
+```python
+compose_content = f"""version: "3.8"
+services:
+  app:
+    image: {image_tag}
+    container_name: {instance_name.lower()}
+{ports_section}    volumes:
+      - {repo_path}:/workspace
+    restart: unless-stopped
+"""
+```
+
+**Problems:**
+- Missing `stdin_open: true` and `tty: true` (essential for terminal tools)
+- Missing `working_dir: /workspace`
+- No way to add labels, networks, healthchecks, or extra services
+- Port section is conditionally included with awkward string concatenation
+
+### 3. Hardcoded Mount Paths
+
+| Mount | Current Target | Problem |
+|-------|---------------|---------|
+| Repository | `/workspace` | Always root-owned; no permission fix for non-root users |
+| SSH keys (clone mode) | `/root/.ssh` | Invisible to containers running as `user` |
+| Git-mount configs | `/tmp/.pi` | May be root-owned; conflicts with user's `.pi` |
+| Config profile files | Instance-relative paths | No validation against container filesystem |
+
+### 4. No Base Image / Layer Composition
+
+Every tool type must specify a complete Dockerfile from `FROM` to `CMD`. There's no way to say:
+
+```yaml
+base: ubuntu-24.04-dev  # pre-built with curl, git, build-essential
+layers:
+  - nodejs-20
+  - pi-coding-agent
+  - custom-packages: [neovim, ranger, tmux]
+```
+
+### 5. Permission Mismatch (Non-Root Users)
+
+The pi-agent Dockerfile creates a `user` account and uses `USER user`. Bind mounts from the host come in as root-owned. The API has **no automatic permission fix** — this caused the workspace-unwritable bug.
+
+Workarounds considered:
+- Post-start `docker exec --user root chown` (current fix)
+- Dockerfile entrypoint script that chowns before dropping privileges
+- Matching container UID to host UID
+
+None of these are systematic or configurable.
+
+### 6. Config Overlap and Precedence Confusion
+
+Three systems control the same container aspects:
+
+| System | Controls | Stored |
+|--------|----------|--------|
+| ToolConfig | env vars, files, port, command, working_dir, volumes | DB (per-user per-tool) |
+| ConfigProfile | env vars, files, mounts, git_mounts, hints | DB (hierarchical) |
+| Compose template / generation | volumes, ports, command, working_dir | DB string / Python f-string |
+
+**Precedence is unclear:**
+- ToolConfig `working_directory` vs ConfigProfile hint `working_directory` vs compose `working_dir`
+- ToolConfig `volumes` vs ConfigProfile `mounts` vs compose `volumes`
+- `start_command` from ToolConfig vs ConfigProfile vs Dockerfile `CMD`
+
+### 7. Build Context Limitations
+
+`build_context` is a JSON dictionary of `{relative_path: file_content}`. This is stored in the DB as text.
+
+**Problems:**
+- Binary files (images, tarballs) can't be stored
+- Large files bloat the DB
+- No versioning or external reference (e.g., "use file from git repo")
+
+---
+
+## Extensibility Gaps
+
+| Want | Current State | Gap |
+|------|--------------|-----|
+| Add a new language runtime (e.g., Go, Rust) | Copy entire Dockerfile, edit | No modular package/layer system |
+| Use a custom base image (e.g., `my-registry/dev-base:v2`) | Edit full Dockerfile | No base-image reference field |
+| Mount a second repo or a secrets file | Write ConfigProfile or ToolConfig JSON | No declarative mount schema |
+| Run as root instead of `user` | Edit full Dockerfile | No runtime-user field |
+| Add a sidecar (e.g., postgres for integration tests) | Edit compose_template string | No multi-service compose support |
+| Pre-install VS Code server | Edit full Dockerfile | No "feature" or "extension" mechanism |
+| Custom entrypoint script | Edit full Dockerfile | No entrypoint field |
+
+---
+
+## Design Directions (Pre-Proposal)
+
+### Direction A: Declarative Tool Manifests
+
+Replace the monolithic `dockerfile_template` with a structured manifest:
+
+```yaml
+# Example: tool manifest for pi-agent
+name: pi-agent
+base_image: ubuntu:24.04
+user:
+  name: user
+  uid: 1000
+  home: /home/user
+packages:
+  apt: [curl, wget, git, neovim, ranger, tmux, htop, tree, jq, python3, python3-pip, build-essential]
+  npm_global: [@earendil-works/pi-coding-agent]
+  node_version: "20"
+env:
+  DEBIAN_FRONTEND: noninteractive
+config_files:
+  /home/user/.tmux.conf: "set -g mouse on\n..."
+  /home/user/.config/ranger/rc.conf: "set preview_files true\n..."
+working_directory: /workspace
+command: ["/bin/bash"]
+ports: []
+mounts:
+  repo: {target: /workspace, writable: true}
+  ssh: {target: /home/user/.ssh, mode: "0600"}
+  tmp_state: {target: /tmp/.pi, writable: true}
+```
+
+**Pros:** Structured, reviewable, composable  
+**Cons:** Requires a manifest-to-Dockerfile compiler; migration complexity
+
+### Direction B: Base Image Registry + Layers
+
+Maintain a registry of pre-built base images:
+
+```
+headquarter/base/ubuntu-24.04-dev
+headquarter/base/nodejs-20
+headquarter/base/python-3.11
+```
+
+Tool types reference a base image and a list of layers:
+
+```yaml
+base_image: headquarter/base/ubuntu-24.04-dev
+layers:
+  - type: npm_install
+    package: @earendil-works/pi-coding-agent
+  - type: config_file
+    path: /home/user/.tmux.conf
+    content: "..."
+```
+
+**Pros:** Fast builds (base images cached), reusable, versioned  
+**Cons:** Requires image registry management, layer ordering complexity
+
+### Direction C: Compose-First with Dockerfile Overrides
+
+Treat `compose_template` as the primary definition. For simple cases, use a pre-built image. For custom cases, allow an inline Dockerfile or a `build` section in the compose:
+
+```yaml
+services:
+  app:
+    build:
+      context: .
+      dockerfile_inline: |
+        FROM ubuntu:24.04
+        ...
+    stdin_open: true
+    tty: true
+    working_dir: /workspace
+    volumes:
+      - ${REPO_PATH}:/workspace
+      - ${SSH_PATH}:/home/user/.ssh:ro
+    user: "${CONTAINER_USER:-user}"
+```
+
+**Pros:** Leverages Docker Compose native features, familiar to users  
+**Cons:** Still string-based; inline Dockerfiles are hard to edit
+
+### Direction D: Permission-Aware Mount Schema
+
+Decouple mount declaration from mount implementation:
+
+```python
+class MountPolicy:
+    source: str          # host path
+    target: str          # container path
+    owner: str | None    # container user to own the mount
+    permissions: str     # chmod string
+    readonly: bool
+```
+
+At startup, the API runs a post-start "permission fixer" that applies all policies:
+
+```bash
+docker exec --user root <container> chown -R <owner> <target>
+docker exec --user root <container> chmod <permissions> <target>
+```
+
+**Pros:** Systematic, works with any base image, configurable per mount  
+**Cons:** Adds startup latency, requires root to exist in container
+
+---
+
+## Recommended Next Steps
+
+1. **Proposal phase:** Evaluate Direction A (Declarative Manifests) vs Direction C (Compose-First) for the primary architecture
+2. **Design phase:** Detail the manifest schema or compose enhancement, migration path, and API changes
+3. **Consider Direction D** as a cross-cutting concern regardless of primary direction
+
+---
+
+## Risks
+
+- **Migration risk:** Existing `dockerfile_template` and `compose_template` columns need backward-compatible migration
+- **Build cache invalidation:** Changing the build system may invalidate Docker layer caches
+- **User confusion:** Adding a manifest layer on top of Dockerfiles may feel like "yet another abstraction"
+- **Scope creep:** This touches tool types, tool configs, config profiles, compose generation, and the startup flow — high cross-cutting surface
+
+---
+
+## Artifacts
+
+- `openspec/config.yaml` — SDD configuration
+- `openspec/explorations/streamline-tool-container-definitions.md` — This document