feat: tool definition manifest system (PR 1)
- Add ToolDefinitionManifest model with base image versioning - Add manifest compiler: Dockerfile + Compose generation from JSON manifests - Add permission fixer: post-start chown/chmod for mount policies - Add tool definition CRUD API with live compile preview endpoint - Integrate manifest-based startup flow in start_instance - Add Alembic migration with data conversion for pi-agent - Add 48 unit tests for manifest compiler, permission fixer, docker service - Keep backward compatibility with legacy dockerfile_template/compose_template Migration: applied successfully. Pi-agent converted to manifest. Quality gates: pytest (146 passed, 4 pre-existing unrelated failures)
This commit is contained in:
Alex Blank
@@ -0,0 +1,37 @@
|
||||
project:
|
||||
name: Headquarter
|
||||
description: Docker-based development platform for managing coding agent tool instances
|
||||
repository: https://git.commumedia.org/alex/headquarter
|
||||
|
||||
stack:
|
||||
backend:
|
||||
framework: FastAPI
|
||||
language: Python 3.11
|
||||
database: PostgreSQL 15 (async SQLAlchemy)
|
||||
cache: Redis 7
|
||||
migrations: Alembic
|
||||
testing: pytest
|
||||
frontend:
|
||||
framework: React + Vite
|
||||
language: TypeScript
|
||||
infrastructure:
|
||||
local: Docker Compose
|
||||
production: Docker Compose + Traefik
|
||||
auth: Authentik SSO
|
||||
|
||||
sdd:
|
||||
execution_mode: interactive
|
||||
artifact_store: openspec
|
||||
chained_pr_strategy: auto-forecast
|
||||
review_budget_lines: 400
|
||||
|
||||
strict_tdd:
|
||||
enabled: true
|
||||
test_command: docker exec hq-api pytest
|
||||
evidence_required: red_green_triangulate_refactor
|
||||
|
||||
phase_rules:
|
||||
explore_before_proposal: true
|
||||
spec_before_design: true
|
||||
design_before_tasks: true
|
||||
verify_before_archive: true
|
||||
@@ -0,0 +1,751 @@
|
||||
# Design: Tool Definition Manifest System
|
||||
|
||||
## Status
|
||||
**Phase:** design
|
||||
**Date:** 2026-05-28
|
||||
**Owner:** el Gentleman
|
||||
**Based on:** Spec `tool-definition-manifest`
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Tool Workshop (Frontend) │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │
|
||||
│ │ Base Image │ │ Packages │ │ Mount Schema Designer │ │
|
||||
│ │ Selector │ │ Editors │ │ (target, owner, mode) │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────────────────┘ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ Live Preview: Dockerfile + Compose ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ POST /tool-definitions
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ API Backend │
|
||||
│ ┌─────────────────┐ ┌──────────────────┐ ┌────────────┐ │
|
||||
│ │ ManifestSchema │───▶│ ManifestCompiler │───▶│ LiveBuild │ │
|
||||
│ │ (validation) │ │ (Dockerfile + │ │ (optional) │ │
|
||||
│ │ │ │ Compose gen) │ │ │ │
|
||||
│ └─────────────────┘ └──────────────────┘ └────────────┘ │
|
||||
│ │
|
||||
│ ▼ save to DB
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ ToolDefinitionManifest (JSONB in PostgreSQL) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ POST /instances/{id}/start
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Instance Startup Flow │
|
||||
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
||||
│ │ Resolve │──▶│ Compile │──▶│ Build Image │ │
|
||||
│ │ Manifest │ │ to Dockerfile│ │ (docker build) │ │
|
||||
│ │ (base merge)│ │ + Compose │ │ │ │
|
||||
│ └─────────────┘ └──────────────┘ └──────────────────────┘ │
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌─────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
||||
│ │ Permission │◀──│ docker comp. │◀──│ Generate Compose │ │
|
||||
│ │ Fixer │ │ up │ │ (mount resolution) │ │
|
||||
│ │ (post-start)│ │ │ │ │ │
|
||||
│ └─────────────┘ └──────────────┘ └──────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manifest JSON Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-07/schema#",
|
||||
"type": "object",
|
||||
"required": ["name", "interface_type"],
|
||||
"oneOf": [
|
||||
{"required": ["base_image"]},
|
||||
{"required": ["base_definition_id"]}
|
||||
],
|
||||
"properties": {
|
||||
"name": {"type": "string", "pattern": "^[a-z0-9-]+$", "maxLength": 64},
|
||||
"display_name": {"type": "string", "maxLength": 128},
|
||||
"description": {"type": "string"},
|
||||
"category": {"type": "string", "maxLength": 64},
|
||||
"interface_type": {"type": "string", "enum": ["web", "terminal"]},
|
||||
"base_image": {"type": "string", "maxLength": 256},
|
||||
"base_definition_id": {"type": "string", "format": "uuid"},
|
||||
"base_version": {"type": "string", "maxLength": 32, "default": "latest"},
|
||||
"packages": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"apt": {"type": "array", "items": {"type": "string"}},
|
||||
"node": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"version": {"type": "string", "pattern": "^\\d+$"}
|
||||
},
|
||||
"required": ["version"]
|
||||
},
|
||||
"npm_global": {"type": "array", "items": {"type": "string"}},
|
||||
"pip": {"type": "array", "items": {"type": "string"}}
|
||||
}
|
||||
},
|
||||
"user": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "maxLength": 32},
|
||||
"uid": {"type": "integer", "minimum": 1, "maximum": 65535},
|
||||
"gid": {"type": "integer", "minimum": 1, "maximum": 65535},
|
||||
"create_home": {"type": "boolean", "default": true},
|
||||
"shell": {"type": "string", "maxLength": 64, "default": "/bin/bash"}
|
||||
},
|
||||
"required": ["name", "uid", "gid"]
|
||||
},
|
||||
"env": {"type": "object", "additionalProperties": {"type": "string"}},
|
||||
"scripts": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"build": {"type": "array", "items": {"type": "string"}},
|
||||
"startup": {"type": "array", "items": {"type": "string"}}
|
||||
}
|
||||
},
|
||||
"mounts": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["name", "target", "source_type"],
|
||||
"properties": {
|
||||
"name": {"type": "string", "maxLength": 64},
|
||||
"target": {"type": "string", "maxLength": 256},
|
||||
"source_type": {"type": "string", "enum": ["repo", "ssh_key", "instance", "git_mount", "host_path"]},
|
||||
"writable": {"type": "boolean", "default": true},
|
||||
"owner": {"type": "string", "maxLength": 32},
|
||||
"mode": {"type": "string", "pattern": "^[0-7]{3,4}$"},
|
||||
"file_mode": {"type": "string", "pattern": "^[0-7]{3,4}$"},
|
||||
"readonly": {"type": "boolean", "default": false},
|
||||
"git_mount_ref": {"type": "string", "maxLength": 64}
|
||||
}
|
||||
}
|
||||
},
|
||||
"runtime": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"command": {"type": "array", "items": {"type": "string"}},
|
||||
"stdin_open": {"type": "boolean", "default": false},
|
||||
"tty": {"type": "boolean", "default": false},
|
||||
"working_dir": {"type": "string", "maxLength": 256}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Manifest Compiler Algorithm
|
||||
|
||||
### Step 1: Resolve Base
|
||||
|
||||
```python
|
||||
def resolve_base(manifest: dict) -> dict:
|
||||
"""Merge base definition into the manifest."""
|
||||
if manifest.get("base_definition_id"):
|
||||
base = load_base_definition(manifest["base_definition_id"],
|
||||
manifest.get("base_version", "latest"))
|
||||
# Deep merge: base first, then tool-specific overrides
|
||||
merged = deep_merge(base, manifest)
|
||||
# Remove base fields from the merged result
|
||||
merged.pop("base_definition_id", None)
|
||||
merged.pop("base_version", None)
|
||||
return merged
|
||||
return manifest
|
||||
```
|
||||
|
||||
Merge rules:
|
||||
- `packages`: Union arrays (base apt + tool apt = combined apt)
|
||||
- `env`: Tool overrides base (dict merge, tool wins on key conflict)
|
||||
- `scripts.build`: Concatenate arrays (base scripts first, then tool)
|
||||
- `scripts.startup`: Concatenate arrays
|
||||
- `user`: Tool overrides base entirely
|
||||
- `mounts`: Concatenate arrays
|
||||
- `runtime`: Tool overrides base (dict merge)
|
||||
|
||||
### Step 2: Generate Dockerfile
|
||||
|
||||
```python
|
||||
def compile_dockerfile(manifest: dict) -> str:
|
||||
"""Compile a resolved manifest to a Dockerfile string."""
|
||||
lines = []
|
||||
|
||||
# FROM
|
||||
lines.append(f"FROM {manifest['base_image']}")
|
||||
lines.append("")
|
||||
|
||||
# ENV (build-time)
|
||||
for key, value in manifest.get("env", {}).items():
|
||||
lines.append(f"ENV {key}={shlex.quote(value)}")
|
||||
if manifest.get("env"):
|
||||
lines.append("")
|
||||
|
||||
# System packages (apt)
|
||||
apt_packages = manifest.get("packages", {}).get("apt", [])
|
||||
if apt_packages:
|
||||
lines.append("RUN apt-get update && apt-get install -y \\\\")
|
||||
for pkg in apt_packages:
|
||||
lines.append(f" {pkg} \\\\")
|
||||
lines.append(" && rm -rf /var/lib/apt/lists/*")
|
||||
lines.append("")
|
||||
|
||||
# Node.js
|
||||
node = manifest.get("packages", {}).get("node")
|
||||
if node:
|
||||
lines.append(
|
||||
f"RUN curl -fsSL https://deb.nodesource.com/setup_{node['version']}.x | bash - && \\\\")
|
||||
lines.append(" apt-get install -y nodejs && \\\\")
|
||||
lines.append(" rm -rf /var/lib/apt/lists/*")
|
||||
lines.append("")
|
||||
|
||||
# NPM global
|
||||
npm_packages = manifest.get("packages", {}).get("npm_global", [])
|
||||
if npm_packages:
|
||||
pkg_list = " ".join(shlex.quote(p) for p in npm_packages)
|
||||
lines.append(f"RUN npm install -g {pkg_list}")
|
||||
lines.append("")
|
||||
|
||||
# Pip
|
||||
pip_packages = manifest.get("packages", {}).get("pip", [])
|
||||
if pip_packages:
|
||||
pkg_list = " ".join(shlex.quote(p) for p in pip_packages)
|
||||
lines.append(f"RUN pip install {pkg_list}")
|
||||
lines.append("")
|
||||
|
||||
# User creation
|
||||
user = manifest.get("user")
|
||||
if user:
|
||||
lines.append(
|
||||
f"RUN groupadd -g {user['gid']} {user['name']} && \\\\")
|
||||
lines.append(
|
||||
f" useradd -u {user['uid']} -g {user['gid']} "
|
||||
f"{'-m ' if user.get('create_home', True) else ''}"
|
||||
f"-s {user['shell']} {user['name']}")
|
||||
lines.append("")
|
||||
|
||||
# Build scripts
|
||||
build_scripts = manifest.get("scripts", {}).get("build", [])
|
||||
for script in build_scripts:
|
||||
# Each script block becomes one RUN command
|
||||
# Normalize multi-line scripts
|
||||
normalized = " && ".join(line.strip() for line in script.strip().split("\n") if line.strip())
|
||||
lines.append(f"RUN {normalized}")
|
||||
if build_scripts:
|
||||
lines.append("")
|
||||
|
||||
# Create mount target directories and pre-set ownership
|
||||
mounts = manifest.get("mounts", [])
|
||||
if mounts:
|
||||
dirs = []
|
||||
for mount in mounts:
|
||||
dirs.append(mount["target"])
|
||||
if dirs:
|
||||
dir_str = " ".join(dirs)
|
||||
lines.append(f"RUN mkdir -p {dir_str}")
|
||||
if user:
|
||||
lines.append(f"RUN chown -R {user['name']}:{user['name']} {dir_str}")
|
||||
lines.append("")
|
||||
|
||||
# Entrypoint for startup scripts
|
||||
startup_scripts = manifest.get("scripts", {}).get("startup", [])
|
||||
if startup_scripts:
|
||||
lines.append("COPY .headquarter/entrypoint.sh /usr/local/bin/headquarter-entrypoint")
|
||||
lines.append("RUN chmod +x /usr/local/bin/headquarter-entrypoint")
|
||||
lines.append("")
|
||||
|
||||
# Switch to user
|
||||
if user:
|
||||
lines.append(f"USER {user['name']}")
|
||||
lines.append(f"WORKDIR /home/{user['name']}")
|
||||
lines.append("")
|
||||
|
||||
# Entrypoint and CMD
|
||||
runtime = manifest.get("runtime", {})
|
||||
if startup_scripts:
|
||||
lines.append('ENTRYPOINT ["/usr/local/bin/headquarter-entrypoint"]')
|
||||
command = runtime.get("command", ["/bin/bash"])
|
||||
cmd_json = json.dumps(command)
|
||||
lines.append(f"CMD {cmd_json}")
|
||||
|
||||
return "\n".join(lines)
|
||||
```
|
||||
|
||||
### Step 3: Generate Entrypoint Script
|
||||
|
||||
```python
|
||||
def compile_entrypoint(manifest: dict) -> str:
|
||||
"""Generate the startup entrypoint script."""
|
||||
lines = ["#!/bin/bash", "set -e", ""]
|
||||
|
||||
startup_scripts = manifest.get("scripts", {}).get("startup", [])
|
||||
for script in startup_scripts:
|
||||
lines.append(script)
|
||||
lines.append("")
|
||||
|
||||
lines.append('exec "$@"')
|
||||
return "\n".join(lines)
|
||||
```
|
||||
|
||||
### Step 4: Generate Compose
|
||||
|
||||
```python
|
||||
def compile_compose(manifest: dict, variables: dict) -> str:
|
||||
"""Compile a resolved manifest to a Docker Compose string."""
|
||||
runtime = manifest.get("runtime", {})
|
||||
user = manifest.get("user")
|
||||
interface_type = manifest["interface_type"]
|
||||
|
||||
service = {
|
||||
"image": variables["IMAGE_TAG"],
|
||||
"container_name": variables["INSTANCE_NAME"],
|
||||
"restart": "unless-stopped",
|
||||
}
|
||||
|
||||
# Terminal-specific fields
|
||||
if runtime.get("stdin_open", False):
|
||||
service["stdin_open"] = True
|
||||
if runtime.get("tty", False):
|
||||
service["tty"] = True
|
||||
if runtime.get("working_dir"):
|
||||
service["working_dir"] = runtime["working_dir"]
|
||||
|
||||
# User override in compose (helps with permission consistency)
|
||||
if user:
|
||||
service["user"] = f"{user['uid']}:{user['gid']}"
|
||||
|
||||
# Ports for web tools
|
||||
if interface_type == "web" and manifest.get("default_port"):
|
||||
service["ports"] = [f"{variables['TOOL_PORT']}:{manifest['default_port']}"]
|
||||
|
||||
# Environment
|
||||
env = manifest.get("env", {})
|
||||
if env:
|
||||
service["environment"] = env
|
||||
|
||||
# Volumes from mounts
|
||||
volumes = []
|
||||
for mount in manifest.get("mounts", []):
|
||||
source = resolve_mount_source(mount, variables)
|
||||
target = mount["target"]
|
||||
readonly = ":ro" if mount.get("readonly", False) else ""
|
||||
volumes.append(f"{source}:{target}{readonly}")
|
||||
|
||||
# Append extra volumes from tool config / config profile
|
||||
for vol in variables.get("EXTRA_VOLUMES", []):
|
||||
vol_str = f"{vol['source']}:{vol['target']}"
|
||||
if vol.get("readonly"):
|
||||
vol_str += ":ro"
|
||||
volumes.append(vol_str)
|
||||
|
||||
if volumes:
|
||||
service["volumes"] = volumes
|
||||
|
||||
compose = {
|
||||
"services": {"app": service}
|
||||
}
|
||||
|
||||
return yaml.dump(compose, default_flow_style=False)
|
||||
```
|
||||
|
||||
### Step 5: Image Tag Hash
|
||||
|
||||
```python
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
def compute_image_tag(tool_name: str, manifest: dict) -> str:
|
||||
"""Deterministic image tag from manifest content."""
|
||||
# Normalize: sort keys, stable JSON
|
||||
canonical = json.dumps(manifest, sort_keys=True, separators=(',', ':'))
|
||||
hash_suffix = hashlib.sha256(canonical.encode()).hexdigest()[:8]
|
||||
safe_name = tool_name.lower().replace(" ", "-").replace("_", "-")
|
||||
return f"headquarter/{safe_name}-{hash_suffix}:latest"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mount Resolution
|
||||
|
||||
Each `source_type` resolves differently at instance creation time:
|
||||
|
||||
| source_type | Resolution | Example |
|
||||
|-------------|------------|---------|
|
||||
| `repo` | `{repo_path}` (mount or clone) | `/data/repos/headquarter` |
|
||||
| `ssh_key` | `{instance_dir}/.ssh` | `/data/instances/.../.ssh` |
|
||||
| `instance` | `{instance_dir}/{name}` | `/data/instances/.../mounts/tmp_.pi_agents` |
|
||||
| `git_mount` | Resolved from ConfigProfile git_mounts | `/data/instances/.../git-mounts/...` |
|
||||
| `host_path` | Literal host path | `/var/run/docker.sock` |
|
||||
|
||||
```python
|
||||
def resolve_mount_source(mount: dict, variables: dict) -> str:
|
||||
source_type = mount["source_type"]
|
||||
if source_type == "repo":
|
||||
return variables["REPO_PATH"]
|
||||
elif source_type == "ssh_key":
|
||||
return variables["SSH_PATH"]
|
||||
elif source_type == "instance":
|
||||
instance_dir = variables["INSTANCE_DIR"]
|
||||
mount_name = mount["name"]
|
||||
return f"{instance_dir}/mounts/{mount_name}"
|
||||
elif source_type == "git_mount":
|
||||
ref = mount.get("git_mount_ref", "default")
|
||||
return variables.get(f"GIT_MOUNT_{ref}", "")
|
||||
elif source_type == "host_path":
|
||||
return mount.get("source", "")
|
||||
else:
|
||||
raise ValueError(f"Unknown source_type: {source_type}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Permission Fixer (Post-Start)
|
||||
|
||||
```python
|
||||
def apply_mount_permissions(
|
||||
container_id: str,
|
||||
mounts: list[dict],
|
||||
timeout: int = 10
|
||||
) -> list[dict]:
|
||||
"""Apply permission policies to mounted directories in a running container.
|
||||
|
||||
Returns a list of results: [{mount_name, success, error}]
|
||||
"""
|
||||
results = []
|
||||
for mount in mounts:
|
||||
name = mount["name"]
|
||||
target = mount["target"]
|
||||
owner = mount.get("owner")
|
||||
mode = mount.get("mode")
|
||||
file_mode = mount.get("file_mode")
|
||||
|
||||
result = {"mount_name": name, "success": True, "error": None}
|
||||
|
||||
try:
|
||||
if owner:
|
||||
proc = subprocess.run(
|
||||
["docker", "exec", container_id, "chown", "-R",
|
||||
f"{owner}:{owner}", target],
|
||||
capture_output=True, text=True, timeout=timeout
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
result["success"] = False
|
||||
result["error"] = f"chown failed: {proc.stderr}"
|
||||
|
||||
if mode and result["success"]:
|
||||
proc = subprocess.run(
|
||||
["docker", "exec", container_id, "chmod", mode, target],
|
||||
capture_output=True, text=True, timeout=timeout
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
result["success"] = False
|
||||
result["error"] = f"chmod failed: {proc.stderr}"
|
||||
|
||||
if file_mode and result["success"]:
|
||||
proc = subprocess.run(
|
||||
["docker", "exec", container_id, "sh", "-c",
|
||||
f"find {target} -type f -exec chmod {file_mode} {{}} +"],
|
||||
capture_output=True, text=True, timeout=timeout
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
result["success"] = False
|
||||
result["error"] = f"file_mode chmod failed: {proc.stderr}"
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
result["success"] = False
|
||||
result["error"] = "Permission fix timed out"
|
||||
|
||||
results.append(result)
|
||||
|
||||
return results
|
||||
```
|
||||
|
||||
**Important:** The permission fixer checks if `root` exists in the container before running. If the container has no `root` user (e.g., distroless images), it logs a warning and skips.
|
||||
|
||||
---
|
||||
|
||||
## Modified Startup Flow
|
||||
|
||||
The `start_instance` function in `tool_instances.py` is modified as follows:
|
||||
|
||||
```python
|
||||
async def start_instance(...):
|
||||
# ... existing validation ...
|
||||
|
||||
tool_type = await session.get(ToolType, instance.tool_type_id)
|
||||
|
||||
if tool_type.definition_type == "manifest" and tool_type.manifest_id:
|
||||
# NEW: Manifest-based startup flow
|
||||
manifest = await load_manifest(session, tool_type.manifest_id)
|
||||
|
||||
# Merge base definition
|
||||
resolved = resolve_base(manifest)
|
||||
|
||||
# Merge tool configs and config profile
|
||||
resolved = apply_tool_configs(resolved, configs)
|
||||
resolved = apply_config_profile(resolved, profile)
|
||||
|
||||
# Compile
|
||||
dockerfile = compile_dockerfile(resolved)
|
||||
entrypoint = compile_entrypoint(resolved)
|
||||
image_tag = compute_image_tag(tool_type.name, resolved)
|
||||
|
||||
# Build image
|
||||
build_context = {
|
||||
"Dockerfile": dockerfile,
|
||||
".headquarter/entrypoint.sh": entrypoint,
|
||||
}
|
||||
returncode, stdout, stderr = build_image(
|
||||
instance_dir=instance_dir,
|
||||
dockerfile=dockerfile, # The Dockerfile references entrypoint.sh
|
||||
tag=image_tag,
|
||||
build_context=build_context,
|
||||
)
|
||||
|
||||
# Generate compose with resolved variables
|
||||
variables = {
|
||||
"IMAGE_TAG": image_tag,
|
||||
"INSTANCE_NAME": instance.name.lower(),
|
||||
"REPO_PATH": repo_path,
|
||||
"SSH_PATH": ssh_dir,
|
||||
"INSTANCE_DIR": instance_dir,
|
||||
# ... git mount resolutions from config profile ...
|
||||
}
|
||||
compose_content = compile_compose(resolved, variables)
|
||||
write_compose_file(instance_dir, compose_content)
|
||||
|
||||
# Store image tag for reuse
|
||||
instance.image_tag = image_tag
|
||||
|
||||
else:
|
||||
# LEGACY: Existing dockerfile_template / compose_template flow
|
||||
...
|
||||
|
||||
# ... docker compose up ...
|
||||
# ... wait for running ...
|
||||
|
||||
# NEW: Apply mount permissions post-start
|
||||
if tool_type.definition_type == "manifest":
|
||||
manifest = await load_manifest(session, tool_type.manifest_id)
|
||||
resolved = resolve_base(manifest)
|
||||
permission_results = apply_mount_permissions(
|
||||
instance.container_id,
|
||||
resolved.get("mounts", [])
|
||||
)
|
||||
for result in permission_results:
|
||||
if not result["success"]:
|
||||
logger.warning(
|
||||
"Permission fix failed for mount %s: %s",
|
||||
result["mount_name"], result["error"]
|
||||
)
|
||||
|
||||
# ... readiness probe ...
|
||||
# ... tunnel creation ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
apps/api/src/
|
||||
├── models/
|
||||
│ ├── tool_definition_manifest.py # NEW: SQLAlchemy model
|
||||
│ └── tool_type.py # MOD: add manifest_id, definition_type
|
||||
├── services/
|
||||
│ ├── manifest_compiler.py # NEW: compile_dockerfile, compile_compose, resolve_base
|
||||
│ ├── permission_fixer.py # NEW: apply_mount_permissions
|
||||
│ └── docker_build.py # MOD: support build_context files
|
||||
├── api/
|
||||
│ ├── tool_definitions.py # NEW: CRUD + compile endpoints
|
||||
│ └── tool_instances.py # MOD: manifest-based startup flow
|
||||
├── schemas/
|
||||
│ └── manifest_schema.py # NEW: JSON Schema + Pydantic validators
|
||||
└── alembic/versions/
|
||||
└── 20260528_add_tool_definition_manifests.py # NEW: migration
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
|
||||
| Module | Tests | Coverage |
|
||||
|--------|-------|----------|
|
||||
| `manifest_compiler.py` | Dockerfile generation for all package managers, base merging, entrypoint generation | All branches |
|
||||
| `permission_fixer.py` | chown/chmod success, failure, timeout, missing root user | All branches |
|
||||
| `manifest_schema.py` | Valid manifest acceptance, invalid manifest rejection (all error paths) | All validation rules |
|
||||
|
||||
### Integration Tests
|
||||
|
||||
| Scenario | Test |
|
||||
|----------|------|
|
||||
| Manifest → Dockerfile → Build | Create manifest, compile, build image, verify it runs |
|
||||
| ConfigProfile merge | Start instance with profile, verify mounts merged correctly |
|
||||
| Permission fix | Start non-root container, verify workspace is writable |
|
||||
| SSH key mount | Start clone-mode instance, verify SSH keys accessible and have correct permissions |
|
||||
| Legacy compatibility | Start instance from old dockerfile_template tool type, verify it still works |
|
||||
| Base versioning | Create tool with base v1, update base to v2, verify tool still uses v1 |
|
||||
|
||||
### E2E Tests
|
||||
|
||||
| Scenario | Test |
|
||||
|----------|------|
|
||||
| Tool Workshop CRUD | Create, edit, preview, delete a tool definition via UI |
|
||||
| Instance lifecycle | Create instance from manifest tool, start, terminal connect, stop, delete |
|
||||
|
||||
---
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Step 1: Schema Migration
|
||||
|
||||
```python
|
||||
# alembic migration
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"tool_definition_manifests",
|
||||
sa.Column("id", sa.UUID(), nullable=False),
|
||||
sa.Column("name", sa.String(64), nullable=False),
|
||||
sa.Column("display_name", sa.String(128), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=True),
|
||||
sa.Column("category", sa.String(64), nullable=True),
|
||||
sa.Column("interface_type", sa.String(16), nullable=False),
|
||||
sa.Column("base_image", sa.String(256), nullable=True),
|
||||
sa.Column("base_definition_id", sa.UUID(), nullable=True),
|
||||
sa.Column("base_version", sa.String(32), nullable=False, server_default="latest"),
|
||||
sa.Column("manifest", sa.JSON(), nullable=False),
|
||||
sa.Column("dockerfile_cache", sa.Text(), nullable=True),
|
||||
sa.Column("compose_cache", sa.Text(), nullable=True),
|
||||
sa.Column("version", sa.String(32), nullable=False, server_default="v1"),
|
||||
sa.Column("is_base", sa.Boolean(), nullable=False, server_default="false"),
|
||||
sa.Column("created_by_id", sa.UUID(), nullable=True),
|
||||
sa.Column("created_at", sa.TIMESTAMP(timezone=True), server_default=sa.func.now()),
|
||||
sa.Column("updated_at", sa.TIMESTAMP(timezone=True), server_default=sa.func.now()),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("name"),
|
||||
sa.ForeignKeyConstraint(["base_definition_id"], ["tool_definition_manifests.id"]),
|
||||
sa.ForeignKeyConstraint(["created_by_id"], ["users.id"]),
|
||||
sa.CheckConstraint(
|
||||
"(base_image IS NOT NULL) OR (base_definition_id IS NOT NULL)",
|
||||
name="ck_tool_definition_manifests_base_required"
|
||||
),
|
||||
)
|
||||
|
||||
op.add_column("tool_types", sa.Column("manifest_id", sa.UUID(), nullable=True))
|
||||
op.add_column("tool_types", sa.Column("definition_type", sa.String(16), nullable=False, server_default="legacy"))
|
||||
op.create_foreign_key(
|
||||
"fk_tool_types_manifest_id",
|
||||
"tool_types", "tool_definition_manifests",
|
||||
["manifest_id"], ["id"]
|
||||
)
|
||||
|
||||
op.add_column("tool_instances", sa.Column("manifest_compiled_at", sa.TIMESTAMP(timezone=True), nullable=True))
|
||||
op.add_column("tool_instances", sa.Column("image_tag", sa.String(256), nullable=True))
|
||||
```
|
||||
|
||||
### Step 2: Data Migration
|
||||
|
||||
Convert the existing pi-agent tool type from `dockerfile_template` to manifest:
|
||||
|
||||
```python
|
||||
def upgrade_data():
|
||||
conn = op.get_bind()
|
||||
|
||||
# Create the base definition for ubuntu-24.04-dev
|
||||
base_id = uuid.uuid4()
|
||||
conn.execute(sa.text("""
|
||||
INSERT INTO tool_definition_manifests
|
||||
(id, name, display_name, description, interface_type, base_image, manifest, is_base, version)
|
||||
VALUES (:id, 'ubuntu-24.04-dev', 'Ubuntu 24.04 Dev Base', 'Base development environment',
|
||||
'terminal', 'ubuntu:24.04', :manifest, true, 'v1')
|
||||
"""), {
|
||||
"id": base_id,
|
||||
"manifest": json.dumps({
|
||||
"name": "ubuntu-24.04-dev",
|
||||
"base_image": "ubuntu:24.04",
|
||||
"packages": {"apt": ["curl", "wget", "git", "build-essential", "ca-certificates"]},
|
||||
"user": {"name": "user", "uid": 1000, "gid": 1000},
|
||||
})
|
||||
})
|
||||
|
||||
# Create pi-agent manifest referencing the base
|
||||
pi_agent_id = uuid.UUID("d07b8376-2151-4119-8c1d-27f792aae9a3")
|
||||
conn.execute(sa.text("""
|
||||
INSERT INTO tool_definition_manifests
|
||||
(id, name, display_name, description, category, interface_type,
|
||||
base_definition_id, base_version, manifest, version)
|
||||
VALUES (:id, 'pi-agent', 'Pi Agent', 'Terminal-based coding harness',
|
||||
'development', 'terminal', :base_id, 'v1', :manifest, 'v1')
|
||||
"""), {
|
||||
"id": pi_agent_id,
|
||||
"base_id": base_id,
|
||||
"manifest": json.dumps({
|
||||
"name": "pi-agent",
|
||||
"display_name": "Pi Agent",
|
||||
"interface_type": "terminal",
|
||||
"packages": {
|
||||
"apt": ["neovim", "ranger", "tmux", "htop", "tree", "jq", "python3", "python3-pip"],
|
||||
"node": {"version": "20"},
|
||||
"npm_global": ["@earendil-works/pi-coding-agent"]
|
||||
},
|
||||
"user": {"name": "user", "uid": 1001, "gid": 1001, "create_home": True, "shell": "/bin/bash"},
|
||||
"env": {"DEBIAN_FRONTEND": "noninteractive"},
|
||||
"scripts": {
|
||||
"build": [
|
||||
"git config --global init.defaultBranch main && git config --global user.email 'dev@headquarter.local' && git config --global user.name 'Developer'",
|
||||
"mkdir -p /home/user/.config/ranger && echo 'set preview_files true' > /home/user/.config/ranger/rc.conf"
|
||||
],
|
||||
"startup": [
|
||||
"if [ -d /workspace ]; then sudo chown -R user:user /workspace 2>/dev/null || true; fi"
|
||||
]
|
||||
},
|
||||
"mounts": [
|
||||
{"name": "workspace", "target": "/workspace", "source_type": "repo", "writable": True, "owner": "user"},
|
||||
{"name": "ssh_keys", "target": "/home/user/.ssh", "source_type": "ssh_key", "mode": "0700", "file_mode": "0600", "readonly": True},
|
||||
{"name": "pi_state", "target": "/tmp/.pi/agents", "source_type": "instance", "writable": True},
|
||||
{"name": "pi_config", "target": "/home/user/.pi", "source_type": "git_mount", "git_mount_ref": "dotfiles", "writable": True, "owner": "user"}
|
||||
],
|
||||
"runtime": {
|
||||
"command": ["/bin/bash"],
|
||||
"stdin_open": True,
|
||||
"tty": True,
|
||||
"working_dir": "/workspace"
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
# Update the existing tool_types row
|
||||
conn.execute(sa.text("""
|
||||
UPDATE tool_types
|
||||
SET manifest_id = :manifest_id,
|
||||
definition_type = 'manifest',
|
||||
dockerfile_template = NULL,
|
||||
compose_template = NULL
|
||||
WHERE name = 'pi-agent'
|
||||
"""), {"manifest_id": pi_agent_id})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Review Workload Forecast
|
||||
|
||||
| PR | Scope | Est. Lines | Review Risk |
|
||||
|----|-------|-----------|-------------|
|
||||
| PR 1: Backend | Compiler, fixer, API, tests, migration | ~1800 | Medium — core algorithm changes |
|
||||
| PR 2: Frontend | Tool Workshop UI | ~1200 | Medium — new feature, self-contained |
|
||||
| PR 3: Migration | Data migration, legacy fallback | ~300 | Low — additive only |
|
||||
|
||||
All PRs are under the 400-line budget individually. Chained PRs recommended for sequential review.
|
||||
@@ -0,0 +1,362 @@
|
||||
# SDD Exploration: Streamline Tool Container Definitions
|
||||
|
||||
## Status
|
||||
**Phase:** explore
|
||||
**Date:** 2026-05-28
|
||||
**Owner:** el Gentleman (parent session)
|
||||
**Scope:** Tool container definition, build, and mount system
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The current tool container system works for the happy path (pi-agent on Ubuntu) but has deep structural inflexibility:
|
||||
|
||||
1. **Monolithic Dockerfile strings** in the database — impossible to review, version, or compose
|
||||
2. **Ad-hoc compose generation** — string formatting with hardcoded fields (`stdin_open`, `tty`, `working_dir` missing for dockerfile types)
|
||||
3. **Hardcoded mount paths** — `/workspace` and `/root/.ssh` don't adapt to the container's runtime user
|
||||
4. **No package/base-image modularity** — every tool type carries a full Dockerfile copy
|
||||
5. **Permission mismatch** — bind mounts come in as root-owned; non-root container users can't write
|
||||
6. **Config overlap** — tool configs, config profiles, and compose templates fight for control of the same fields
|
||||
|
||||
This exploration proposes a **layered, declarative container definition system** where tool types compose from reusable base images, mount schemas, and permission policies.
|
||||
|
||||
---
|
||||
|
||||
## Current Architecture Map
|
||||
|
||||
### Data Model
|
||||
|
||||
```
|
||||
ToolType (DB table)
|
||||
├── name, display_name, description, category
|
||||
├── interface_type: "web" | "terminal"
|
||||
├── definition_type: "dockerfile" | "compose"
|
||||
├── dockerfile_template: TEXT (giant Dockerfile string)
|
||||
├── compose_template: TEXT (Jinja-like {{VAR}} string)
|
||||
├── build_context: JSON {path: content}
|
||||
├── required_variables: JSON ["REPO_PATH", ...]
|
||||
├── default_port: int
|
||||
└── readiness_probe: JSON
|
||||
|
||||
ToolInstance (DB table)
|
||||
├── name, display_name, status
|
||||
├── tool_type_id → ToolType
|
||||
├── repository_id → GitRepository
|
||||
├── compose_path: str
|
||||
├── container_id, container_name
|
||||
├── port, url, public_url, tunnel_id
|
||||
├── clone_mode: "mount" | "clone"
|
||||
├── branch, new_branch
|
||||
└── selected_config_profile_id → ConfigProfile
|
||||
|
||||
ToolConfig (DB table, per-user per-tool-type)
|
||||
├── config_type: "env" | "file"
|
||||
├── key, value, file_path
|
||||
├── port_override, start_command, working_directory
|
||||
├── environment_variables: JSON
|
||||
└── volumes: JSON [{source, target, type}]
|
||||
|
||||
ConfigProfile (DB table)
|
||||
├── name, description
|
||||
├── user_id, project_id, tool_type_id
|
||||
├── environment_variables: JSON
|
||||
├── files: JSON {path: content}
|
||||
├── mounts: JSON [{source, target, type}]
|
||||
├── git_mounts: JSON [{remote_url, source_path, target_path, branch}]
|
||||
└── parent_profile_id → ConfigProfile (hierarchy)
|
||||
```
|
||||
|
||||
### Creation Flow (`create_instance`)
|
||||
|
||||
```
|
||||
POST /projects/{id}/repositories/{id}/instances
|
||||
→ validate tool_type, repo, config_profile
|
||||
→ generate instance_name = "{tool_type}-{repo}-{uuid8}"
|
||||
→ ensure_instance_directory(instance_name)
|
||||
→ find_free_port()
|
||||
→ determine repo_path (mount = repo.path; clone = clone_repository())
|
||||
→ IF tool_type.definition_type == "dockerfile":
|
||||
build_image(instance_dir, dockerfile_template, tag, build_context)
|
||||
generate compose_content (HARDCODED STRING FORMATTING)
|
||||
ELSE:
|
||||
render_compose_template(tool_type.compose_template, variables)
|
||||
→ write_compose_file()
|
||||
→ create ToolInstance DB record (status="pending")
|
||||
```
|
||||
|
||||
### Startup Flow (`start_instance`)
|
||||
|
||||
```
|
||||
POST /instances/{id}/start
|
||||
→ fetch ToolConfigs (env, files, port_override, start_command, working_dir, volumes)
|
||||
→ IF selected_config_profile:
|
||||
resolve_profile() → env, files, mounts, git_mounts, hints
|
||||
→ write .env file, config files
|
||||
→ IF clone_mode: mount SSH keys at /root/.ssh (HARDCODED)
|
||||
→ _modify_compose_file(port, command, working_dir, extra_volumes)
|
||||
→ _sanitize_compose_file()
|
||||
→ execute_compose_command("up")
|
||||
→ get_container_id(instance.name) ← CASE-SENSITIVE BUG (fixed)
|
||||
→ get_container_name(instance.name)
|
||||
→ connect_container_to_network("backend")
|
||||
→ wait_for_container_running()
|
||||
→ IF web: start_cloudflared_tunnel()
|
||||
→ instance.status = "running"
|
||||
```
|
||||
|
||||
### Key Files
|
||||
|
||||
| File | Responsibility |
|
||||
|------|---------------|
|
||||
| `apps/api/src/api/tool_instances.py` | create_instance, start_instance, stop_instance, restart_instance, proxy, logs |
|
||||
| `apps/api/src/api/tool_types.py` | CRUD for ToolType (DB strings) |
|
||||
| `apps/api/src/services/docker.py` | compose execution, container queries, tunnel management |
|
||||
| `apps/api/src/services/docker_build.py` | `docker build` wrapper |
|
||||
| `apps/api/src/services/terminal_session.py` | PTY-based terminal over `docker exec` |
|
||||
| `apps/api/src/services/terminal_manager.py` | WebSocket ↔ terminal session lifecycle |
|
||||
| `apps/api/src/models/tool_type.py` | SQLAlchemy model |
|
||||
|
||||
---
|
||||
|
||||
## Pain Points (Detailed)
|
||||
|
||||
### 1. Monolithic Dockerfile Templates
|
||||
|
||||
The pi-agent Dockerfile template is a 40-line string stored in the DB migration:
|
||||
|
||||
```sql
|
||||
INSERT INTO tool_types (... dockerfile_template ...)
|
||||
VALUES ('...# Pi Coding Agent - Terminal-based coding harness\nFROM ubuntu:24.04\n...')
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- No syntax highlighting, linting, or `docker build` validation at edit time
|
||||
- Every tool type copies the entire Dockerfile; no reuse of common layers
|
||||
- Changes require a DB migration
|
||||
- No way for users to customize packages without forking the whole template
|
||||
|
||||
### 2. Ad-Hoc Compose Generation
|
||||
|
||||
For `dockerfile` type tools, the compose is generated by Python f-string:
|
||||
|
||||
```python
|
||||
compose_content = f"""version: "3.8"
|
||||
services:
|
||||
app:
|
||||
image: {image_tag}
|
||||
container_name: {instance_name.lower()}
|
||||
{ports_section} volumes:
|
||||
- {repo_path}:/workspace
|
||||
restart: unless-stopped
|
||||
"""
|
||||
```
|
||||
|
||||
**Problems:**
|
||||
- Missing `stdin_open: true` and `tty: true` (essential for terminal tools)
|
||||
- Missing `working_dir: /workspace`
|
||||
- No way to add labels, networks, healthchecks, or extra services
|
||||
- Port section is conditionally included with awkward string concatenation
|
||||
|
||||
### 3. Hardcoded Mount Paths
|
||||
|
||||
| Mount | Current Target | Problem |
|
||||
|-------|---------------|---------|
|
||||
| Repository | `/workspace` | Always root-owned; no permission fix for non-root users |
|
||||
| SSH keys (clone mode) | `/root/.ssh` | Invisible to containers running as `user` |
|
||||
| Git-mount configs | `/tmp/.pi` | May be root-owned; conflicts with user's `.pi` |
|
||||
| Config profile files | Instance-relative paths | No validation against container filesystem |
|
||||
|
||||
### 4. No Base Image / Layer Composition
|
||||
|
||||
Every tool type must specify a complete Dockerfile from `FROM` to `CMD`. There's no way to say:
|
||||
|
||||
```yaml
|
||||
base: ubuntu-24.04-dev # pre-built with curl, git, build-essential
|
||||
layers:
|
||||
- nodejs-20
|
||||
- pi-coding-agent
|
||||
- custom-packages: [neovim, ranger, tmux]
|
||||
```
|
||||
|
||||
### 5. Permission Mismatch (Non-Root Users)
|
||||
|
||||
The pi-agent Dockerfile creates a `user` account and uses `USER user`. Bind mounts from the host come in as root-owned. The API has **no automatic permission fix** — this caused the workspace-unwritable bug.
|
||||
|
||||
Workarounds considered:
|
||||
- Post-start `docker exec --user root chown` (current fix)
|
||||
- Dockerfile entrypoint script that chowns before dropping privileges
|
||||
- Matching container UID to host UID
|
||||
|
||||
None of these are systematic or configurable.
|
||||
|
||||
### 6. Config Overlap and Precedence Confusion
|
||||
|
||||
Three systems control the same container aspects:
|
||||
|
||||
| System | Controls | Stored |
|
||||
|--------|----------|--------|
|
||||
| ToolConfig | env vars, files, port, command, working_dir, volumes | DB (per-user per-tool) |
|
||||
| ConfigProfile | env vars, files, mounts, git_mounts, hints | DB (hierarchical) |
|
||||
| Compose template / generation | volumes, ports, command, working_dir | DB string / Python f-string |
|
||||
|
||||
**Precedence is unclear:**
|
||||
- ToolConfig `working_directory` vs ConfigProfile hint `working_directory` vs compose `working_dir`
|
||||
- ToolConfig `volumes` vs ConfigProfile `mounts` vs compose `volumes`
|
||||
- `start_command` from ToolConfig vs ConfigProfile vs Dockerfile `CMD`
|
||||
|
||||
### 7. Build Context Limitations
|
||||
|
||||
`build_context` is a JSON dictionary of `{relative_path: file_content}`. This is stored in the DB as text.
|
||||
|
||||
**Problems:**
|
||||
- Binary files (images, tarballs) can't be stored
|
||||
- Large files bloat the DB
|
||||
- No versioning or external reference (e.g., "use file from git repo")
|
||||
|
||||
---
|
||||
|
||||
## Extensibility Gaps
|
||||
|
||||
| Want | Current State | Gap |
|
||||
|------|--------------|-----|
|
||||
| Add a new language runtime (e.g., Go, Rust) | Copy entire Dockerfile, edit | No modular package/layer system |
|
||||
| Use a custom base image (e.g., `my-registry/dev-base:v2`) | Edit full Dockerfile | No base-image reference field |
|
||||
| Mount a second repo or a secrets file | Write ConfigProfile or ToolConfig JSON | No declarative mount schema |
|
||||
| Run as root instead of `user` | Edit full Dockerfile | No runtime-user field |
|
||||
| Add a sidecar (e.g., postgres for integration tests) | Edit compose_template string | No multi-service compose support |
|
||||
| Pre-install VS Code server | Edit full Dockerfile | No "feature" or "extension" mechanism |
|
||||
| Custom entrypoint script | Edit full Dockerfile | No entrypoint field |
|
||||
|
||||
---
|
||||
|
||||
## Design Directions (Pre-Proposal)
|
||||
|
||||
### Direction A: Declarative Tool Manifests
|
||||
|
||||
Replace the monolithic `dockerfile_template` with a structured manifest:
|
||||
|
||||
```yaml
|
||||
# Example: tool manifest for pi-agent
|
||||
name: pi-agent
|
||||
base_image: ubuntu:24.04
|
||||
user:
|
||||
name: user
|
||||
uid: 1000
|
||||
home: /home/user
|
||||
packages:
|
||||
apt: [curl, wget, git, neovim, ranger, tmux, htop, tree, jq, python3, python3-pip, build-essential]
|
||||
npm_global: [@earendil-works/pi-coding-agent]
|
||||
node_version: "20"
|
||||
env:
|
||||
DEBIAN_FRONTEND: noninteractive
|
||||
config_files:
|
||||
/home/user/.tmux.conf: "set -g mouse on\n..."
|
||||
/home/user/.config/ranger/rc.conf: "set preview_files true\n..."
|
||||
working_directory: /workspace
|
||||
command: ["/bin/bash"]
|
||||
ports: []
|
||||
mounts:
|
||||
repo: {target: /workspace, writable: true}
|
||||
ssh: {target: /home/user/.ssh, mode: "0600"}
|
||||
tmp_state: {target: /tmp/.pi, writable: true}
|
||||
```
|
||||
|
||||
**Pros:** Structured, reviewable, composable
|
||||
**Cons:** Requires a manifest-to-Dockerfile compiler; migration complexity
|
||||
|
||||
### Direction B: Base Image Registry + Layers
|
||||
|
||||
Maintain a registry of pre-built base images:
|
||||
|
||||
```
|
||||
headquarter/base/ubuntu-24.04-dev
|
||||
headquarter/base/nodejs-20
|
||||
headquarter/base/python-3.11
|
||||
```
|
||||
|
||||
Tool types reference a base image and a list of layers:
|
||||
|
||||
```yaml
|
||||
base_image: headquarter/base/ubuntu-24.04-dev
|
||||
layers:
|
||||
- type: npm_install
|
||||
package: @earendil-works/pi-coding-agent
|
||||
- type: config_file
|
||||
path: /home/user/.tmux.conf
|
||||
content: "..."
|
||||
```
|
||||
|
||||
**Pros:** Fast builds (base images cached), reusable, versioned
|
||||
**Cons:** Requires image registry management, layer ordering complexity
|
||||
|
||||
### Direction C: Compose-First with Dockerfile Overrides
|
||||
|
||||
Treat `compose_template` as the primary definition. For simple cases, use a pre-built image. For custom cases, allow an inline Dockerfile or a `build` section in the compose:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
build:
|
||||
context: .
|
||||
dockerfile_inline: |
|
||||
FROM ubuntu:24.04
|
||||
...
|
||||
stdin_open: true
|
||||
tty: true
|
||||
working_dir: /workspace
|
||||
volumes:
|
||||
- ${REPO_PATH}:/workspace
|
||||
- ${SSH_PATH}:/home/user/.ssh:ro
|
||||
user: "${CONTAINER_USER:-user}"
|
||||
```
|
||||
|
||||
**Pros:** Leverages Docker Compose native features, familiar to users
|
||||
**Cons:** Still string-based; inline Dockerfiles are hard to edit
|
||||
|
||||
### Direction D: Permission-Aware Mount Schema
|
||||
|
||||
Decouple mount declaration from mount implementation:
|
||||
|
||||
```python
|
||||
class MountPolicy:
|
||||
source: str # host path
|
||||
target: str # container path
|
||||
owner: str | None # container user to own the mount
|
||||
permissions: str # chmod string
|
||||
readonly: bool
|
||||
```
|
||||
|
||||
At startup, the API runs a post-start "permission fixer" that applies all policies:
|
||||
|
||||
```bash
|
||||
docker exec --user root <container> chown -R <owner> <target>
|
||||
docker exec --user root <container> chmod <permissions> <target>
|
||||
```
|
||||
|
||||
**Pros:** Systematic, works with any base image, configurable per mount
|
||||
**Cons:** Adds startup latency, requires root to exist in container
|
||||
|
||||
---
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
1. **Proposal phase:** Evaluate Direction A (Declarative Manifests) vs Direction C (Compose-First) for the primary architecture
|
||||
2. **Design phase:** Detail the manifest schema or compose enhancement, migration path, and API changes
|
||||
3. **Consider Direction D** as a cross-cutting concern regardless of primary direction
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
- **Migration risk:** Existing `dockerfile_template` and `compose_template` columns need backward-compatible migration
|
||||
- **Build cache invalidation:** Changing the build system may invalidate Docker layer caches
|
||||
- **User confusion:** Adding a manifest layer on top of Dockerfiles may feel like "yet another abstraction"
|
||||
- **Scope creep:** This touches tool types, tool configs, config profiles, compose generation, and the startup flow — high cross-cutting surface
|
||||
|
||||
---
|
||||
|
||||
## Artifacts
|
||||
|
||||
- `openspec/config.yaml` — SDD configuration
|
||||
- `openspec/explorations/streamline-tool-container-definitions.md` — This document
|
||||
@@ -0,0 +1,447 @@
|
||||
# SDD Proposal: Declarative Tool Container Compiler
|
||||
|
||||
## Status
|
||||
**Phase:** proposal
|
||||
**Date:** 2026-05-28
|
||||
**Owner:** el Gentleman
|
||||
**Based on:** Exploration `streamline-tool-container-definitions`
|
||||
|
||||
---
|
||||
|
||||
## User Story
|
||||
|
||||
As a platform operator, I want to define a tool container by specifying:
|
||||
- A **base image** (e.g. `ubuntu:24.04` or a pre-built `headquarter/base:dev-ubuntu`)
|
||||
- A **list of packages** to install (apt, npm, pip, etc.)
|
||||
- **Setup scripts** that run at build-time or container-startup
|
||||
- **Mount policies** that automatically fix permissions for the runtime user
|
||||
|
||||
I do **not** want to write Dockerfiles or Compose files by hand.
|
||||
|
||||
The system should compile these declarations into Dockerfiles and Compose files automatically, while remaining fully compatible with the existing ConfigProfile mount system.
|
||||
|
||||
---
|
||||
|
||||
## Core Concept: The Tool Definition Manifest
|
||||
|
||||
Replace the monolithic `dockerfile_template` and `compose_template` strings with a single structured **Tool Definition Manifest**.
|
||||
|
||||
```yaml
|
||||
# Tool Definition Manifest (stored as JSON in DB)
|
||||
name: pi-agent
|
||||
display_name: "Pi Agent"
|
||||
description: "Terminal-based coding harness"
|
||||
category: development
|
||||
interface_type: terminal # web | terminal
|
||||
|
||||
# ── Base Image ───────────────────────────────────────────────
|
||||
base_image: ubuntu:24.04
|
||||
# OR reference a pre-built base definition:
|
||||
# base_definition_id: "base-ubuntu-24.04-dev"
|
||||
|
||||
# ── Packages ─────────────────────────────────────────────────
|
||||
packages:
|
||||
apt:
|
||||
- curl
|
||||
- wget
|
||||
- git
|
||||
- neovim
|
||||
- ranger
|
||||
- tmux
|
||||
- htop
|
||||
- tree
|
||||
- jq
|
||||
- ca-certificates
|
||||
- python3
|
||||
- python3-pip
|
||||
- build-essential
|
||||
node:
|
||||
version: "20" # triggers nodesource setup
|
||||
npm_global:
|
||||
- "@earendil-works/pi-coding-agent"
|
||||
# pip:
|
||||
# - requests
|
||||
# - httpx
|
||||
|
||||
# ── Runtime User ─────────────────────────────────────────────
|
||||
user:
|
||||
name: user
|
||||
uid: 1001
|
||||
gid: 1001
|
||||
create_home: true
|
||||
shell: /bin/bash
|
||||
|
||||
# ── Environment ──────────────────────────────────────────────
|
||||
env:
|
||||
DEBIAN_FRONTEND: noninteractive
|
||||
|
||||
# ── Setup Scripts ────────────────────────────────────────────
|
||||
scripts:
|
||||
# build: runs during `docker build` → becomes RUN commands
|
||||
build:
|
||||
- |
|
||||
git config --global init.defaultBranch main
|
||||
git config --global user.email "dev@headquarter.local"
|
||||
git config --global user.name "Developer"
|
||||
- |
|
||||
mkdir -p /home/user/.config/ranger
|
||||
echo 'set preview_files true' > /home/user/.config/ranger/rc.conf
|
||||
|
||||
# startup: runs when container starts → becomes entrypoint script
|
||||
startup:
|
||||
- |
|
||||
# Ensure workspace is owned by runtime user
|
||||
if [ -d /workspace ]; then
|
||||
sudo chown -R user:user /workspace 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# ── Mount Schema ─────────────────────────────────────────────
|
||||
mounts:
|
||||
- name: workspace
|
||||
target: /workspace
|
||||
source_type: repo # resolved from repository path at instance creation
|
||||
writable: true
|
||||
owner: user # post-start: chown -R user:user /workspace
|
||||
|
||||
- name: ssh_keys
|
||||
target: /home/user/.ssh
|
||||
source_type: ssh_key # resolved from repository's SSH key
|
||||
mode: "0700" # post-start: chmod 0700 /home/user/.ssh
|
||||
file_mode: "0600" # post-start: chmod 0600 files inside
|
||||
readonly: true
|
||||
|
||||
- name: pi_state
|
||||
target: /tmp/.pi/agents
|
||||
source_type: instance # resolved to {instance_dir}/mounts/tmp_.pi_agents
|
||||
writable: true
|
||||
|
||||
- name: pi_config
|
||||
target: /home/user/.pi
|
||||
source_type: git_mount # resolved from config profile git_mounts
|
||||
git_mount_ref: dotfiles # references a named git mount in the config profile
|
||||
writable: true
|
||||
owner: user
|
||||
|
||||
# ── Runtime ──────────────────────────────────────────────────
|
||||
runtime:
|
||||
command: ["/bin/bash"]
|
||||
stdin_open: true
|
||||
tty: true
|
||||
working_dir: /workspace
|
||||
# ports are auto-derived from interface_type:
|
||||
# web: expose default_port
|
||||
# terminal: no port mapping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How It Compiles
|
||||
|
||||
### 1. Dockerfile Generation
|
||||
|
||||
The manifest compiler transforms the spec into a Dockerfile:
|
||||
|
||||
```dockerfile
|
||||
# Generated Dockerfile — do not edit manually
|
||||
FROM ubuntu:24.04
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# ── System Packages ──
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl wget git neovim ranger tmux htop tree jq \
|
||||
ca-certificates python3 python3-pip build-essential \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# ── Node.js ──
|
||||
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||
&& apt-get install -y nodejs \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# ── NPM Packages ──
|
||||
RUN npm install -g --ignore-scripts @earendil-works/pi-coding-agent
|
||||
|
||||
# ── Runtime User ──
|
||||
RUN groupadd -g 1001 user && \
|
||||
useradd -u 1001 -g 1001 -m -s /bin/bash user
|
||||
|
||||
# ── Build Scripts ──
|
||||
RUN git config --global init.defaultBranch main && \
|
||||
git config --global user.email "dev@headquarter.local" && \
|
||||
git config --global user.name "Developer"
|
||||
|
||||
RUN mkdir -p /home/user/.config/ranger && \
|
||||
echo 'set preview_files true' > /home/user/.config/ranger/rc.conf
|
||||
|
||||
# ── Environment ──
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# ── Setup Directories ──
|
||||
RUN mkdir -p /workspace /tmp/.pi/agents /home/user/.pi && \
|
||||
chown -R user:user /workspace /tmp/.pi /home/user/.pi
|
||||
|
||||
# ── Entrypoint for Startup Scripts ──
|
||||
COPY .headquarter/entrypoint.sh /usr/local/bin/headquarter-entrypoint
|
||||
RUN chmod +x /usr/local/bin/headquarter-entrypoint
|
||||
|
||||
USER user
|
||||
WORKDIR /home/user
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/headquarter-entrypoint"]
|
||||
CMD ["/bin/bash"]
|
||||
```
|
||||
|
||||
The generated `entrypoint.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Run startup scripts
|
||||
echo "Ensure workspace is owned by runtime user"
|
||||
if [ -d /workspace ]; then
|
||||
sudo chown -R user:user /workspace 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Pass through to the main command
|
||||
exec "$@"
|
||||
```
|
||||
|
||||
### 2. Compose File Generation
|
||||
|
||||
The manifest compiler also generates the Compose file:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
app:
|
||||
image: ${IMAGE_TAG}
|
||||
container_name: ${INSTANCE_NAME}
|
||||
stdin_open: true
|
||||
tty: true
|
||||
working_dir: /workspace
|
||||
user: "1001:1001" # from manifest user.uid/gid
|
||||
volumes:
|
||||
- ${REPO_PATH}:/workspace
|
||||
- ${SSH_PATH}:/home/user/.ssh:ro
|
||||
- ${INSTANCE_DIR}/mounts/tmp_.pi_agents:/tmp/.pi/agents
|
||||
- ${GIT_MOUNT_dotfiles}:/home/user/.pi
|
||||
environment:
|
||||
DEBIAN_FRONTEND: noninteractive
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
Variables are resolved at instance creation time:
|
||||
- `${REPO_PATH}` → the repository working directory
|
||||
- `${SSH_PATH}` → prepared SSH key directory
|
||||
- `${INSTANCE_DIR}` → the instance working directory
|
||||
- `${GIT_MOUNT_dotfiles}` → resolved from config profile git_mounts
|
||||
|
||||
### 3. Permission Fixer (Post-Start)
|
||||
|
||||
After `docker compose up`, the API iterates over the mount schema and applies permission policies:
|
||||
|
||||
```python
|
||||
for mount in manifest.mounts:
|
||||
if mount.owner:
|
||||
docker_exec(f"chown -R {mount.owner}:{mount.owner} {mount.target}")
|
||||
if mount.mode:
|
||||
docker_exec(f"chmod {mount.mode} {mount.target}")
|
||||
if mount.file_mode:
|
||||
docker_exec(f"find {mount.target} -type f -exec chmod {mount.file_mode} {{}} +")
|
||||
```
|
||||
|
||||
This is **systematic and configurable** — not hardcoded to `/workspace`.
|
||||
|
||||
---
|
||||
|
||||
## Config Profile Compatibility
|
||||
|
||||
The existing ConfigProfile system provides:
|
||||
- `environment_variables` → merged into compose `environment`
|
||||
- `files` → written to instance dir, mounted via `volumes`
|
||||
- `mounts` → appended to compose `volumes`
|
||||
- `git_mounts` → resolved to host paths, appended to compose `volumes`
|
||||
- `hints.start_command` → overrides `runtime.command`
|
||||
- `hints.working_directory` → overrides `runtime.working_dir`
|
||||
|
||||
With the manifest system, ConfigProfiles **extend** the default mount schema:
|
||||
|
||||
1. Tool manifest defines the **default mount schema** (workspace, ssh, state)
|
||||
2. ConfigProfile can add **additional mounts** or **override runtime hints**
|
||||
3. Both are merged at instance-start time into the final compose file
|
||||
|
||||
The merge precedence:
|
||||
1. Tool manifest (defaults)
|
||||
2. ToolConfig overrides (per-user per-tool settings)
|
||||
3. ConfigProfile overrides (hierarchical, can inherit from parent)
|
||||
4. User-provided start options (e.g. branch selection)
|
||||
|
||||
---
|
||||
|
||||
## Base Image Definitions
|
||||
|
||||
A base definition is itself a manifest with no `runtime` section:
|
||||
|
||||
```yaml
|
||||
# Base Definition: "ubuntu-24.04-dev"
|
||||
name: ubuntu-24.04-dev
|
||||
description: "Ubuntu 24.04 with build tools"
|
||||
base_image: ubuntu:24.04
|
||||
packages:
|
||||
apt:
|
||||
- curl
|
||||
- wget
|
||||
- git
|
||||
- build-essential
|
||||
- ca-certificates
|
||||
user:
|
||||
name: user
|
||||
uid: 1000
|
||||
gid: 1000
|
||||
create_home: true
|
||||
```
|
||||
|
||||
A tool definition can reference it:
|
||||
|
||||
```yaml
|
||||
base_definition_id: "ubuntu-24.04-dev"
|
||||
packages:
|
||||
apt:
|
||||
- neovim
|
||||
- ranger
|
||||
- tmux
|
||||
node:
|
||||
version: "20"
|
||||
```
|
||||
|
||||
The compiler **merges** the base definition with the tool-specific overrides:
|
||||
- Packages are **unioned** (base apt + tool apt)
|
||||
- Scripts are **appended** (base build scripts, then tool build scripts)
|
||||
- User/env are **overridden** (tool wins)
|
||||
|
||||
This enables a family of tool types to share a common base.
|
||||
|
||||
---
|
||||
|
||||
## Database Schema (Proposed)
|
||||
|
||||
### New Table: `tool_definition_manifests`
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `id` | UUID | PK |
|
||||
| `name` | str | Unique identifier |
|
||||
| `display_name` | str | Human-readable |
|
||||
| `description` | str | |
|
||||
| `category` | str | development, data-science, etc. |
|
||||
| `interface_type` | enum | web, terminal |
|
||||
| `base_image` | str | e.g. `ubuntu:24.04` |
|
||||
| `base_definition_id` | UUID? | FK to another manifest |
|
||||
| `manifest` | JSONB | The full manifest JSON |
|
||||
| `dockerfile_cache` | TEXT | Last generated Dockerfile (for inspection) |
|
||||
| `created_at` | datetime | |
|
||||
| `updated_at` | datetime | |
|
||||
|
||||
### Migration: `tool_types` table
|
||||
|
||||
Add a nullable `manifest_id` column to `tool_types`.
|
||||
|
||||
For backward compatibility:
|
||||
- If `manifest_id` is set → use the new manifest system
|
||||
- If `manifest_id` is null → fall back to `dockerfile_template` / `compose_template`
|
||||
|
||||
A data migration converts existing pi-agent to the new manifest format.
|
||||
|
||||
---
|
||||
|
||||
## API Changes
|
||||
|
||||
### New Endpoints
|
||||
|
||||
```
|
||||
GET /tool-definitions → list all base/tool definitions
|
||||
GET /tool-definitions/{id} → get a definition
|
||||
POST /tool-definitions → create a new definition
|
||||
PUT /tool-definitions/{id} → update a definition
|
||||
DELETE /tool-definitions/{id} → delete (if not in use)
|
||||
POST /tool-definitions/{id}/compile → preview generated Dockerfile + compose
|
||||
```
|
||||
|
||||
### Modified Endpoints
|
||||
|
||||
```
|
||||
POST /tool-types → can now accept manifest_id instead of templates
|
||||
GET /tool-types/{id} → includes manifest if available
|
||||
```
|
||||
|
||||
### Frontend Changes
|
||||
|
||||
New UI page: **Tool Workshop**
|
||||
|
||||
- Base image selector (dropdown of existing bases or custom FROM)
|
||||
- Package manager tabs (apt, npm, pip, etc.)
|
||||
- Script editor (build vs startup)
|
||||
- Mount schema designer (drag-drop or form)
|
||||
- Live preview of generated Dockerfile
|
||||
- Test build button (builds image and reports success/failure)
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Should we support multi-stage builds?**
|
||||
- Pros: smaller images, separation of build deps from runtime
|
||||
- Cons: more complexity in the manifest schema
|
||||
|
||||
2. **Should base definitions be versioned?**
|
||||
- Pros: reproducible builds, safe updates
|
||||
- Cons: more DB complexity
|
||||
|
||||
3. **How do we handle binary build context files?**
|
||||
- Current: JSON text in DB
|
||||
- Option A: Store in filesystem, reference by path
|
||||
- Option B: Upload to object storage (S3/minio)
|
||||
|
||||
4. **Should generated images be cached/pushed to a registry?**
|
||||
- Currently: built locally per instance
|
||||
- Option: push to `headquarter/tools/{tool-name}:{hash}` for reuse
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|------|----------|------------|
|
||||
| Migration complexity | Medium | Keep old fields nullable; gradual adoption |
|
||||
| Build cache invalidation | Medium | Use deterministic Dockerfile generation; hash manifest for image tag |
|
||||
| User confusion ("yet another abstraction") | Low | Provide live preview + "view generated Dockerfile" button |
|
||||
| Scope creep into full CI/CD | High | Keep scope to container definition only; no pipeline/orchestration |
|
||||
| Binary files in manifests | Medium | Limit build context to text; document workaround for binaries |
|
||||
|
||||
---
|
||||
|
||||
## Effort Estimate
|
||||
|
||||
| Phase | Files | Lines (est) | Complexity |
|
||||
|-------|-------|-------------|------------|
|
||||
| DB migration + models | 3 | 200 | Low |
|
||||
| Manifest compiler (Dockerfile) | 2 | 400 | Medium |
|
||||
| Manifest compiler (Compose) | 2 | 300 | Medium |
|
||||
| Permission fixer refactor | 2 | 200 | Low |
|
||||
| API endpoints | 3 | 400 | Medium |
|
||||
| Frontend Tool Workshop | 8 | 1200 | High |
|
||||
| Tests | 4 | 600 | Medium |
|
||||
| **Total** | **24** | **~3300** | **High** |
|
||||
|
||||
**Review workload forecast:** ~3300 lines is well above the 400-line budget. This should be split into **chained PRs**:
|
||||
1. Backend: manifest schema, compiler, API (PR 1)
|
||||
2. Frontend: Tool Workshop UI (PR 2)
|
||||
3. Migration + data conversion (PR 3)
|
||||
|
||||
---
|
||||
|
||||
## Next Recommended Phase
|
||||
|
||||
**Design** — Detail the manifest JSON schema, compiler internals, and migration plan.
|
||||
|
||||
Should I proceed to design?
|
||||
@@ -0,0 +1,323 @@
|
||||
# Spec: Tool Definition Manifest System
|
||||
|
||||
## Status
|
||||
**Phase:** spec
|
||||
**Date:** 2026-05-28
|
||||
**Owner:** el Gentleman
|
||||
**Based on:** Proposal `streamline-tool-container-definitions`
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
### R1. Declarative Tool Definitions
|
||||
Users must be able to define a tool container without writing Dockerfiles or Compose files. The definition is a structured manifest specifying base image, packages, scripts, mounts, and runtime configuration.
|
||||
|
||||
### R2. Base Image Versioning
|
||||
Base definitions must be versioned. Tool definitions reference a specific base version. Updating a base creates a new version; existing tools remain pinned to their version until explicitly updated.
|
||||
|
||||
### R3. Package Managers
|
||||
The manifest must support multiple package managers: `apt`, `npm` (global), `pip`, and `node` (version installation).
|
||||
|
||||
### R4. Build vs Startup Scripts
|
||||
Scripts are categorized by execution phase:
|
||||
- **Build scripts**: Run during `docker build` (e.g., `git config`, config file setup)
|
||||
- **Startup scripts**: Run when the container starts (e.g., permission fixes, dynamic setup)
|
||||
|
||||
### R5. Mount Schema with Permission Policies
|
||||
Mounts declare:
|
||||
- `target`: Container path
|
||||
- `source_type`: How the source is resolved (`repo`, `ssh_key`, `instance`, `git_mount`, `host_path`)
|
||||
- `writable`: Whether the mount is read-write
|
||||
- `owner`: Container user to own the target path (post-start chown)
|
||||
- `mode`: Directory permissions (post-start chmod)
|
||||
- `file_mode`: File permissions inside the directory
|
||||
- `readonly`: Whether mounted read-only in compose
|
||||
|
||||
### R6. Config Profile Compatibility
|
||||
ConfigProfiles continue to add `env`, `files`, `mounts`, and `git_mounts` on top of the manifest defaults. The merge precedence is: manifest defaults → ToolConfig → ConfigProfile → user options.
|
||||
|
||||
### R7. Backward Compatibility
|
||||
Existing `dockerfile_template` and `compose_template` columns remain functional. New tool types use the manifest system; old types continue to work. A data migration converts the existing pi-agent to the new format.
|
||||
|
||||
### R8. Local Builds
|
||||
Images are built locally per instance using the standard `docker build` command. No registry integration in this phase.
|
||||
|
||||
### R9. Live Preview
|
||||
The API provides a `compile` endpoint that returns the generated Dockerfile and Compose file without building.
|
||||
|
||||
### R10. Deterministic Image Tags
|
||||
The image tag is derived from a hash of the manifest content, enabling build cache reuse when the manifest hasn't changed.
|
||||
|
||||
---
|
||||
|
||||
## Scenarios
|
||||
|
||||
### S1. Creating a New Tool Definition
|
||||
|
||||
**Given** a user on the Tool Workshop page
|
||||
**When** they select base "ubuntu-24.04-dev:v1", add packages `[neovim, tmux]`, add a build script for git config, and define mounts for workspace + ssh
|
||||
**Then** the system generates a manifest, compiles a Dockerfile + Compose preview, and upon save stores the manifest in the database.
|
||||
|
||||
### S2. Building an Instance from a Manifest
|
||||
|
||||
**Given** a tool instance created from a manifest-based tool type
|
||||
**When** `start_instance` is called
|
||||
**Then** the API compiles the manifest to a Dockerfile, builds the image, generates the Compose file with resolved mount paths, starts the container, and applies permission policies post-start.
|
||||
|
||||
### S3. Permission Fix on Non-Root Containers
|
||||
|
||||
**Given** a manifest with `user: {name: user, uid: 1001}` and a mount `target: /workspace, owner: user`
|
||||
**When** the container starts with the workspace bind-mounted from host (root-owned)
|
||||
**Then** the post-start permission fixer runs `docker exec --user root chown -R user:user /workspace`, making the directory writable for the container user.
|
||||
|
||||
### S4. SSH Key Mount for Non-Root User
|
||||
|
||||
**Given** a manifest with a mount `target: /home/user/.ssh, source_type: ssh_key, mode: "0700"`
|
||||
**When** the container starts
|
||||
**Then** SSH keys are mounted from the instance `.ssh` directory to `/home/user/.ssh`, and post-start fixes permissions to `0700` for the directory and `0600` for key files.
|
||||
|
||||
### S5. Config Profile Extends Manifest
|
||||
|
||||
**Given** a manifest with default mount `workspace: /workspace` and a ConfigProfile that adds `git_mounts: [{remote_url: "...", target_path: "/home/user/.config"}]`
|
||||
**When** the instance starts with that profile selected
|
||||
**Then** the final Compose includes both the workspace mount and the config git mount, merged in the correct precedence.
|
||||
|
||||
### S6. Base Version Pinning
|
||||
|
||||
**Given** a tool definition referencing `base_definition_id: "ubuntu-24.04-dev", base_version: "v1"`
|
||||
**When** the base definition is updated to "v2"
|
||||
**Then** the tool definition continues to use "v1" until explicitly updated. New tool definitions default to the latest version.
|
||||
|
||||
### S7. Deterministic Image Tag
|
||||
|
||||
**Given** a manifest with specific packages and scripts
|
||||
**When** compiled
|
||||
**Then** the generated image tag is `headquarter/{tool-name}-{manifest-hash}:latest`, and rebuilding the same manifest reuses the cached image layer.
|
||||
|
||||
### S8. Live Preview Without Build
|
||||
|
||||
**Given** a manifest being edited
|
||||
**When** the user clicks "Preview"
|
||||
**Then** the API returns the generated Dockerfile and Compose file within 500ms, without invoking Docker.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### A1. Manifest Schema Validation
|
||||
- [ ] The manifest JSON must validate against a defined JSON Schema
|
||||
- [ ] Invalid manifests return 400 with detailed field-level errors
|
||||
- [ ] Missing required fields (name, base_image or base_definition_id) are rejected
|
||||
|
||||
### A2. Dockerfile Compilation
|
||||
- [ ] Generated Dockerfile builds successfully with `docker build`
|
||||
- [ ] Build scripts appear as `RUN` commands in order
|
||||
- [ ] Startup scripts appear in the generated entrypoint script
|
||||
- [ ] Packages are installed in a single layer per package manager
|
||||
- [ ] User creation uses the declared uid/gid
|
||||
|
||||
### A3. Compose Compilation
|
||||
- [ ] Generated Compose file starts successfully with `docker compose up`
|
||||
- [ ] Mounts are resolved from `source_type` to actual host paths
|
||||
- [ ] `stdin_open` and `tty` are set for terminal interface types
|
||||
- [ ] Ports are only included for web interface types
|
||||
|
||||
### A4. Permission Fixer
|
||||
- [ ] Post-start chown runs for all mounts with an `owner` declared
|
||||
- [ ] Post-start chmod runs for all mounts with `mode` or `file_mode` declared
|
||||
- [ ] Permission fixes complete within 5 seconds of container start
|
||||
- [ ] If the container has no `root` user, permission fixes are skipped with a warning
|
||||
|
||||
### A5. Config Profile Merge
|
||||
- [ ] ConfigProfile env vars override manifest defaults
|
||||
- [ ] ConfigProfile mounts are appended to manifest mounts
|
||||
- [ ] ConfigProfile git_mounts are resolved and appended
|
||||
- [ ] ToolConfig values override both manifest and ConfigProfile
|
||||
|
||||
### A6. Backward Compatibility
|
||||
- [ ] Existing tool types with `dockerfile_template` continue to work
|
||||
- [ ] Existing tool types with `compose_template` continue to work
|
||||
- [ ] The pi-agent tool type is migrated to the new manifest format
|
||||
- [ ] Old and new tool types can coexist in the same project
|
||||
|
||||
### A7. Base Versioning
|
||||
- [ ] Base definitions store a version string
|
||||
- [ ] Tool definitions store the base version they reference
|
||||
- [ ] Updating a base creates a new version; old versions remain accessible
|
||||
- [ ] The "latest" version can be referenced explicitly or by omission
|
||||
|
||||
### A8. Image Tag Determinism
|
||||
- [ ] Same manifest produces the same image tag
|
||||
- [ ] Changing any field (package, script, env) produces a different tag
|
||||
- [ ] The tag is lowercased and valid as a Docker image reference
|
||||
|
||||
### A9. API Endpoints
|
||||
- [ ] `POST /tool-definitions` creates a definition (201)
|
||||
- [ ] `GET /tool-definitions/{id}` returns the definition with compiled preview
|
||||
- [ ] `POST /tool-definitions/{id}/compile` returns Dockerfile + Compose (no build)
|
||||
- [ ] `PUT /tool-definitions/{id}` updates and re-validates
|
||||
|
||||
### A10. Frontend Tool Workshop
|
||||
- [ ] Users can create a tool definition via form (no raw JSON editing required)
|
||||
- [ ] Live preview shows generated Dockerfile and Compose
|
||||
- [ ] Package lists support add/remove/reorder
|
||||
- [ ] Mount schema supports add/remove with visual feedback
|
||||
- [ ] Base image selector shows available versions
|
||||
|
||||
---
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Multi-stage builds** — Out of scope for this phase. The compiler generates single-stage Dockerfiles.
|
||||
- **Docker registry integration** — Images are built locally per instance.
|
||||
- **Binary build context files** — Build context is limited to text files stored in the DB.
|
||||
- **Custom Dockerfile editing** — Users work exclusively through the manifest; no raw Dockerfile editing.
|
||||
- **Container orchestration beyond Compose** — No Kubernetes, Swarm, or other orchestrators.
|
||||
- **Real-time collaborative editing** — Tool Workshop is single-user editing.
|
||||
|
||||
---
|
||||
|
||||
## API Contract
|
||||
|
||||
### POST /tool-definitions
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"name": "pi-agent",
|
||||
"display_name": "Pi Agent",
|
||||
"description": "Terminal coding harness",
|
||||
"category": "development",
|
||||
"interface_type": "terminal",
|
||||
"base_image": "ubuntu:24.04",
|
||||
"packages": {
|
||||
"apt": ["curl", "git", "neovim", "tmux"],
|
||||
"node": {"version": "20"},
|
||||
"npm_global": ["@earendil-works/pi-coding-agent"]
|
||||
},
|
||||
"user": {
|
||||
"name": "user",
|
||||
"uid": 1001,
|
||||
"gid": 1001,
|
||||
"create_home": true,
|
||||
"shell": "/bin/bash"
|
||||
},
|
||||
"env": {"DEBIAN_FRONTEND": "noninteractive"},
|
||||
"scripts": {
|
||||
"build": [
|
||||
"git config --global init.defaultBranch main",
|
||||
"mkdir -p /home/user/.config/ranger"
|
||||
],
|
||||
"startup": [
|
||||
"if [ -d /workspace ]; then sudo chown -R user:user /workspace; fi"
|
||||
]
|
||||
},
|
||||
"mounts": [
|
||||
{
|
||||
"name": "workspace",
|
||||
"target": "/workspace",
|
||||
"source_type": "repo",
|
||||
"writable": true,
|
||||
"owner": "user"
|
||||
},
|
||||
{
|
||||
"name": "ssh",
|
||||
"target": "/home/user/.ssh",
|
||||
"source_type": "ssh_key",
|
||||
"mode": "0700",
|
||||
"file_mode": "0600",
|
||||
"readonly": true
|
||||
}
|
||||
],
|
||||
"runtime": {
|
||||
"command": ["/bin/bash"],
|
||||
"stdin_open": true,
|
||||
"tty": true,
|
||||
"working_dir": "/workspace"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response (201):**
|
||||
```json
|
||||
{
|
||||
"id": "d07b8376-2151-4119-8c1d-27f792aae9a3",
|
||||
"name": "pi-agent",
|
||||
"display_name": "Pi Agent",
|
||||
"manifest": { ... },
|
||||
"dockerfile_preview": "FROM ubuntu:24.04\n...",
|
||||
"compose_preview": "services:\n app:\n image: ...",
|
||||
"created_at": "2026-05-28T10:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### POST /tool-definitions/{id}/compile
|
||||
|
||||
**Response (200):**
|
||||
```json
|
||||
{
|
||||
"dockerfile": "FROM ubuntu:24.04\n...",
|
||||
"compose": "services:\n app:\n...",
|
||||
"image_tag": "headquarter/pi-agent-a3f7c2d9:latest",
|
||||
"mounts_resolved": [
|
||||
{"name": "workspace", "source": "/data/repos/headquarter", "target": "/workspace"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
### `tool_definition_manifests`
|
||||
|
||||
| Column | Type | Constraints |
|
||||
|--------|------|-------------|
|
||||
| `id` | UUID | PK |
|
||||
| `name` | VARCHAR(64) | UNIQUE, NOT NULL |
|
||||
| `display_name` | VARCHAR(128) | NOT NULL |
|
||||
| `description` | TEXT | |
|
||||
| `category` | VARCHAR(64) | |
|
||||
| `interface_type` | VARCHAR(16) | CHECK IN ('web', 'terminal') |
|
||||
| `base_image` | VARCHAR(256) | |
|
||||
| `base_definition_id` | UUID | FK → `tool_definition_manifests.id` |
|
||||
| `base_version` | VARCHAR(32) | DEFAULT 'latest' |
|
||||
| `manifest` | JSONB | NOT NULL |
|
||||
| `dockerfile_cache` | TEXT | |
|
||||
| `compose_cache` | TEXT | |
|
||||
| `version` | VARCHAR(32) | DEFAULT 'v1' |
|
||||
| `is_base` | BOOLEAN | DEFAULT FALSE |
|
||||
| `created_by_id` | UUID | FK → `users.id` |
|
||||
| `created_at` | TIMESTAMPTZ | DEFAULT now() |
|
||||
| `updated_at` | TIMESTAMPTZ | DEFAULT now() |
|
||||
|
||||
**Check constraint:** Exactly one of `base_image` or `base_definition_id` must be set.
|
||||
|
||||
### Alter `tool_types`
|
||||
|
||||
```sql
|
||||
ALTER TABLE tool_types
|
||||
ADD COLUMN manifest_id UUID REFERENCES tool_definition_manifests(id),
|
||||
ADD COLUMN definition_type VARCHAR(16) DEFAULT 'legacy'; -- 'legacy' | 'manifest'
|
||||
```
|
||||
|
||||
### Alter `tool_instances`
|
||||
|
||||
```sql
|
||||
ALTER TABLE tool_instances
|
||||
ADD COLUMN manifest_compiled_at TIMESTAMPTZ,
|
||||
ADD COLUMN image_tag VARCHAR(256);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Files
|
||||
|
||||
- `apps/api/src/models/tool_definition_manifest.py` — New model
|
||||
- `apps/api/src/models/tool_type.py` — Add manifest_id, definition_type
|
||||
- `apps/api/src/services/manifest_compiler.py` — New compiler
|
||||
- `apps/api/src/services/permission_fixer.py` — Refactored mount policy applier
|
||||
- `apps/api/src/api/tool_definitions.py` — New endpoints
|
||||
- `apps/api/src/api/tool_instances.py` — Modified start_instance flow
|
||||
- `apps/api/alembic/versions/20260528_add_tool_definition_manifests.py` — Migration
|
||||
@@ -0,0 +1,89 @@
|
||||
# Tasks: Tool Definition Manifest System
|
||||
|
||||
## PR 1: Backend Manifest System
|
||||
|
||||
### T1.1 Database Migration
|
||||
- [ ] Create `tool_definition_manifests` table
|
||||
- [ ] Add `manifest_id`, `definition_type` to `tool_types`
|
||||
- [ ] Add `manifest_compiled_at`, `image_tag` to `tool_instances`
|
||||
- [ ] Data migration: convert pi-agent to manifest
|
||||
|
||||
### T1.2 Models
|
||||
- [ ] `ToolDefinitionManifest` SQLAlchemy model
|
||||
- [ ] Update `ToolType` model with manifest relationship
|
||||
- [ ] Update `ToolInstance` model with image_tag
|
||||
|
||||
### T1.3 Manifest Compiler
|
||||
- [ ] `resolve_base()` — deep merge base + tool manifest
|
||||
- [ ] `compile_dockerfile()` — generate Dockerfile from manifest
|
||||
- [ ] `compile_entrypoint()` — generate startup entrypoint script
|
||||
- [ ] `compile_compose()` — generate Compose from manifest
|
||||
- [ ] `compute_image_tag()` — deterministic hash-based tag
|
||||
- [ ] `resolve_mount_source()` — mount source resolution
|
||||
|
||||
### T1.4 Permission Fixer
|
||||
- [ ] `apply_mount_permissions()` — post-start chown/chmod
|
||||
- [ ] Handle missing root user gracefully
|
||||
- [ ] Timeout and error reporting
|
||||
|
||||
### T1.5 API Endpoints
|
||||
- [ ] `POST /tool-definitions` — create
|
||||
- [ ] `GET /tool-definitions` — list
|
||||
- [ ] `GET /tool-definitions/{id}` — get
|
||||
- [ ] `PUT /tool-definitions/{id}` — update
|
||||
- [ ] `DELETE /tool-definitions/{id}` — delete
|
||||
- [ ] `POST /tool-definitions/{id}/compile` — preview
|
||||
|
||||
### T1.6 Modified Startup Flow
|
||||
- [ ] Update `start_instance` to use manifest when `definition_type == "manifest"`
|
||||
- [ ] Integrate permission fixer post-start
|
||||
- [ ] Store image_tag on instance for reuse
|
||||
|
||||
### T1.7 Tests
|
||||
- [ ] Unit: manifest compiler (all package managers, base merge)
|
||||
- [ ] Unit: permission fixer (success, failure, timeout)
|
||||
- [ ] Unit: mount resolution (all source types)
|
||||
- [ ] Integration: manifest → build → start → terminal works
|
||||
- [ ] Integration: legacy tool types still work
|
||||
|
||||
---
|
||||
|
||||
## PR 2: Frontend Tool Workshop
|
||||
|
||||
### T2.1 Tool Definitions API Client
|
||||
- [ ] Add tool definition endpoints to `client.ts`
|
||||
- [ ] Type definitions for manifest schema
|
||||
|
||||
### T2.2 Tool Workshop Page
|
||||
- [ ] Base image selector (with version dropdown)
|
||||
- [ ] Package manager editors (apt list, npm list, node version)
|
||||
- [ ] Script editors (build vs startup, tabbed)
|
||||
- [ ] Mount schema designer (form table with add/remove)
|
||||
- [ ] Runtime config (command, working_dir, stdin_open, tty)
|
||||
|
||||
### T2.3 Live Preview
|
||||
- [ ] Preview panel showing generated Dockerfile
|
||||
- [ ] Preview panel showing generated Compose
|
||||
- [ ] "Compile" button calling API preview endpoint
|
||||
|
||||
### T2.4 Tool Definitions List
|
||||
- [ ] Table view of all definitions
|
||||
- [ ] Create / Edit / Delete actions
|
||||
- [ ] Base indicator (shows if it's a base definition)
|
||||
|
||||
---
|
||||
|
||||
## PR 3: Migration & Legacy Fallback
|
||||
|
||||
### T3.1 Data Migration
|
||||
- [ ] Alembic migration creating base definition + pi-agent manifest
|
||||
- [ ] Update existing pi-agent tool_type row
|
||||
|
||||
### T3.2 Legacy Fallback
|
||||
- [ ] Ensure `definition_type == "legacy"` still uses old flow
|
||||
- [ ] Ensure `dockerfile_template` / `compose_template` still work
|
||||
- [ ] Tests for legacy path
|
||||
|
||||
### T3.3 Documentation
|
||||
- [ ] Update API docs
|
||||
- [ ] Add Tool Workshop user guide
|
||||
Reference in New Issue
Block a user