headquarter/openspec/specs/tool-definition-manifest.md

# Spec: Tool Definition Manifest System

## Status
**Phase:** spec  
**Date:** 2026-05-28  
**Owner:** el Gentleman  
**Based on:** Proposal `streamline-tool-container-definitions`

---

## Requirements

### R1. Declarative Tool Definitions
Users must be able to define a tool container without writing Dockerfiles or Compose files. The definition is a structured manifest specifying base image, packages, scripts, mounts, and runtime configuration.

### R2. Base Image Versioning
Base definitions must be versioned. Tool definitions reference a specific base version. Updating a base creates a new version; existing tools remain pinned to their version until explicitly updated.

### R3. Package Managers
The manifest must support multiple package managers: `apt`, `npm` (global), `pip`, and `node` (version installation).

### R4. Build vs Startup Scripts
Scripts are categorized by execution phase:
- **Build scripts**: Run during `docker build` (e.g., `git config`, config file setup)
- **Startup scripts**: Run when the container starts (e.g., permission fixes, dynamic setup)

### R5. Mount Schema with Permission Policies
Mounts declare:
- `target`: Container path
- `source_type`: How the source is resolved (`repo`, `ssh_key`, `instance`, `git_mount`, `host_path`)
- `writable`: Whether the mount is read-write
- `owner`: Container user to own the target path (post-start chown)
- `mode`: Directory permissions (post-start chmod)
- `file_mode`: File permissions inside the directory
- `readonly`: Whether mounted read-only in compose

### R6. Config Profile Compatibility
ConfigProfiles continue to add `env`, `files`, `mounts`, and `git_mounts` on top of the manifest defaults. The merge precedence is: manifest defaults → ToolConfig → ConfigProfile → user options.

### R7. Backward Compatibility
Existing `dockerfile_template` and `compose_template` columns remain functional. New tool types use the manifest system; old types continue to work. A data migration converts the existing pi-agent to the new format.

### R8. Local Builds
Images are built locally per instance using the standard `docker build` command. No registry integration in this phase.

### R9. Live Preview
The API provides a `compile` endpoint that returns the generated Dockerfile and Compose file without building.

### R10. Deterministic Image Tags
The image tag is derived from a hash of the manifest content, enabling build cache reuse when the manifest hasn't changed.

---

## Scenarios

### S1. Creating a New Tool Definition

**Given** a user on the Tool Workshop page  
**When** they select base "ubuntu-24.04-dev:v1", add packages `[neovim, tmux]`, add a build script for git config, and define mounts for workspace + ssh  
**Then** the system generates a manifest, compiles a Dockerfile + Compose preview, and upon save stores the manifest in the database.

### S2. Building an Instance from a Manifest

**Given** a tool instance created from a manifest-based tool type  
**When** `start_instance` is called  
**Then** the API compiles the manifest to a Dockerfile, builds the image, generates the Compose file with resolved mount paths, starts the container, and applies permission policies post-start.

### S3. Permission Fix on Non-Root Containers

**Given** a manifest with `user: {name: user, uid: 1001}` and a mount `target: /workspace, owner: user`  
**When** the container starts with the workspace bind-mounted from host (root-owned)  
**Then** the post-start permission fixer runs `docker exec --user root chown -R user:user /workspace`, making the directory writable for the container user.

### S4. SSH Key Mount for Non-Root User

**Given** a manifest with a mount `target: /home/user/.ssh, source_type: ssh_key, mode: "0700"`  
**When** the container starts  
**Then** SSH keys are mounted from the instance `.ssh` directory to `/home/user/.ssh`, and post-start fixes permissions to `0700` for the directory and `0600` for key files.

### S5. Config Profile Extends Manifest

**Given** a manifest with default mount `workspace: /workspace` and a ConfigProfile that adds `git_mounts: [{remote_url: "...", target_path: "/home/user/.config"}]`  
**When** the instance starts with that profile selected  
**Then** the final Compose includes both the workspace mount and the config git mount, merged in the correct precedence.

### S6. Base Version Pinning

**Given** a tool definition referencing `base_definition_id: "ubuntu-24.04-dev", base_version: "v1"`  
**When** the base definition is updated to "v2"  
**Then** the tool definition continues to use "v1" until explicitly updated. New tool definitions default to the latest version.

### S7. Deterministic Image Tag

**Given** a manifest with specific packages and scripts  
**When** compiled  
**Then** the generated image tag is `headquarter/{tool-name}-{manifest-hash}:latest`, and rebuilding the same manifest reuses the cached image layer.

### S8. Live Preview Without Build

**Given** a manifest being edited  
**When** the user clicks "Preview"  
**Then** the API returns the generated Dockerfile and Compose file within 500ms, without invoking Docker.

---

## Acceptance Criteria

### A1. Manifest Schema Validation
- [ ] The manifest JSON must validate against a defined JSON Schema
- [ ] Invalid manifests return 400 with detailed field-level errors
- [ ] Missing required fields (name, base_image or base_definition_id) are rejected

### A2. Dockerfile Compilation
- [ ] Generated Dockerfile builds successfully with `docker build`
- [ ] Build scripts appear as `RUN` commands in order
- [ ] Startup scripts appear in the generated entrypoint script
- [ ] Packages are installed in a single layer per package manager
- [ ] User creation uses the declared uid/gid

### A3. Compose Compilation
- [ ] Generated Compose file starts successfully with `docker compose up`
- [ ] Mounts are resolved from `source_type` to actual host paths
- [ ] `stdin_open` and `tty` are set for terminal interface types
- [ ] Ports are only included for web interface types

### A4. Permission Fixer
- [ ] Post-start chown runs for all mounts with an `owner` declared
- [ ] Post-start chmod runs for all mounts with `mode` or `file_mode` declared
- [ ] Permission fixes complete within 5 seconds of container start
- [ ] If the container has no `root` user, permission fixes are skipped with a warning

### A5. Config Profile Merge
- [ ] ConfigProfile env vars override manifest defaults
- [ ] ConfigProfile mounts are appended to manifest mounts
- [ ] ConfigProfile git_mounts are resolved and appended
- [ ] ToolConfig values override both manifest and ConfigProfile

### A6. Backward Compatibility
- [ ] Existing tool types with `dockerfile_template` continue to work
- [ ] Existing tool types with `compose_template` continue to work
- [ ] The pi-agent tool type is migrated to the new manifest format
- [ ] Old and new tool types can coexist in the same project

### A7. Base Versioning
- [ ] Base definitions store a version string
- [ ] Tool definitions store the base version they reference
- [ ] Updating a base creates a new version; old versions remain accessible
- [ ] The "latest" version can be referenced explicitly or by omission

### A8. Image Tag Determinism
- [ ] Same manifest produces the same image tag
- [ ] Changing any field (package, script, env) produces a different tag
- [ ] The tag is lowercased and valid as a Docker image reference

### A9. API Endpoints
- [ ] `POST /tool-definitions` creates a definition (201)
- [ ] `GET /tool-definitions/{id}` returns the definition with compiled preview
- [ ] `POST /tool-definitions/{id}/compile` returns Dockerfile + Compose (no build)
- [ ] `PUT /tool-definitions/{id}` updates and re-validates

### A10. Frontend Tool Workshop
- [ ] Users can create a tool definition via form (no raw JSON editing required)
- [ ] Live preview shows generated Dockerfile and Compose
- [ ] Package lists support add/remove/reorder
- [ ] Mount schema supports add/remove with visual feedback
- [ ] Base image selector shows available versions

---

## Non-Goals

- **Multi-stage builds** — Out of scope for this phase. The compiler generates single-stage Dockerfiles.
- **Docker registry integration** — Images are built locally per instance.
- **Binary build context files** — Build context is limited to text files stored in the DB.
- **Custom Dockerfile editing** — Users work exclusively through the manifest; no raw Dockerfile editing.
- **Container orchestration beyond Compose** — No Kubernetes, Swarm, or other orchestrators.
- **Real-time collaborative editing** — Tool Workshop is single-user editing.

---

## API Contract

### POST /tool-definitions

**Request:**
```json
{
  "name": "pi-agent",
  "display_name": "Pi Agent",
  "description": "Terminal coding harness",
  "category": "development",
  "interface_type": "terminal",
  "base_image": "ubuntu:24.04",
  "packages": {
    "apt": ["curl", "git", "neovim", "tmux"],
    "node": {"version": "20"},
    "npm_global": ["@earendil-works/pi-coding-agent"]
  },
  "user": {
    "name": "user",
    "uid": 1001,
    "gid": 1001,
    "create_home": true,
    "shell": "/bin/bash"
  },
  "env": {"DEBIAN_FRONTEND": "noninteractive"},
  "scripts": {
    "build": [
      "git config --global init.defaultBranch main",
      "mkdir -p /home/user/.config/ranger"
    ],
    "startup": [
      "if [ -d /workspace ]; then sudo chown -R user:user /workspace; fi"
    ]
  },
  "mounts": [
    {
      "name": "workspace",
      "target": "/workspace",
      "source_type": "repo",
      "writable": true,
      "owner": "user"
    },
    {
      "name": "ssh",
      "target": "/home/user/.ssh",
      "source_type": "ssh_key",
      "mode": "0700",
      "file_mode": "0600",
      "readonly": true
    }
  ],
  "runtime": {
    "command": ["/bin/bash"],
    "stdin_open": true,
    "tty": true,
    "working_dir": "/workspace"
  }
}
```

**Response (201):**
```json
{
  "id": "d07b8376-2151-4119-8c1d-27f792aae9a3",
  "name": "pi-agent",
  "display_name": "Pi Agent",
  "manifest": { ... },
  "dockerfile_preview": "FROM ubuntu:24.04\n...",
  "compose_preview": "services:\n  app:\n    image: ...",
  "created_at": "2026-05-28T10:00:00Z"
}
```

### POST /tool-definitions/{id}/compile

**Response (200):**
```json
{
  "dockerfile": "FROM ubuntu:24.04\n...",
  "compose": "services:\n  app:\n...",
  "image_tag": "headquarter/pi-agent-a3f7c2d9:latest",
  "mounts_resolved": [
    {"name": "workspace", "source": "/data/repos/headquarter", "target": "/workspace"}
  ]
}
```

---

## Database Schema

### `tool_definition_manifests`

| Column | Type | Constraints |
|--------|------|-------------|
| `id` | UUID | PK |
| `name` | VARCHAR(64) | UNIQUE, NOT NULL |
| `display_name` | VARCHAR(128) | NOT NULL |
| `description` | TEXT | |
| `category` | VARCHAR(64) | |
| `interface_type` | VARCHAR(16) | CHECK IN ('web', 'terminal') |
| `base_image` | VARCHAR(256) | |
| `base_definition_id` | UUID | FK → `tool_definition_manifests.id` |
| `base_version` | VARCHAR(32) | DEFAULT 'latest' |
| `manifest` | JSONB | NOT NULL |
| `dockerfile_cache` | TEXT | |
| `compose_cache` | TEXT | |
| `version` | VARCHAR(32) | DEFAULT 'v1' |
| `is_base` | BOOLEAN | DEFAULT FALSE |
| `created_by_id` | UUID | FK → `users.id` |
| `created_at` | TIMESTAMPTZ | DEFAULT now() |
| `updated_at` | TIMESTAMPTZ | DEFAULT now() |

**Check constraint:** Exactly one of `base_image` or `base_definition_id` must be set.

### Alter `tool_types`

```sql
ALTER TABLE tool_types
  ADD COLUMN manifest_id UUID REFERENCES tool_definition_manifests(id),
  ADD COLUMN definition_type VARCHAR(16) DEFAULT 'legacy';  -- 'legacy' | 'manifest'
```

### Alter `tool_instances`

```sql
ALTER TABLE tool_instances
  ADD COLUMN manifest_compiled_at TIMESTAMPTZ,
  ADD COLUMN image_tag VARCHAR(256);
```

---

## Related Files

- `apps/api/src/models/tool_definition_manifest.py` — New model
- `apps/api/src/models/tool_type.py` — Add manifest_id, definition_type
- `apps/api/src/services/manifest_compiler.py` — New compiler
- `apps/api/src/services/permission_fixer.py` — Refactored mount policy applier
- `apps/api/src/api/tool_definitions.py` — New endpoints
- `apps/api/src/api/tool_instances.py` — Modified start_instance flow
- `apps/api/alembic/versions/20260528_add_tool_definition_manifests.py` — Migration