headquarter/openspec/specs/tool-definition-manifest.md

Spec: Tool Definition Manifest System

Status

Phase: spec
Date: 2026-05-28
Owner: el Gentleman
Based on: Proposal streamline-tool-container-definitions

---

Requirements

R1. Declarative Tool Definitions

Users must be able to define a tool container without writing Dockerfiles or Compose files. The definition is a structured manifest specifying base image, packages, scripts, mounts, and runtime configuration.

R2. Base Image Versioning

Base definitions must be versioned. Tool definitions reference a specific base version. Updating a base creates a new version; existing tools remain pinned to their version until explicitly updated.

R3. Package Managers

The manifest must support multiple package managers: apt, npm (global), pip, and node (version installation).

R4. Build vs Startup Scripts

Scripts are categorized by execution phase:

• Build scripts: Run during docker build (e.g., git config, config file setup)
• Startup scripts: Run when the container starts (e.g., permission fixes, dynamic setup)

R5. Mount Schema with Permission Policies

Mounts declare:

• target: Container path
• source_type: How the source is resolved (repo, ssh_key, instance, git_mount, host_path)
• writable: Whether the mount is read-write
• owner: Container user to own the target path (post-start chown)
• mode: Directory permissions (post-start chmod)
• file_mode: File permissions inside the directory
• readonly: Whether mounted read-only in compose

R6. Config Profile Compatibility

ConfigProfiles continue to add env, files, mounts, and git_mounts on top of the manifest defaults. The merge precedence is: manifest defaults → ToolConfig → ConfigProfile → user options.

R7. Backward Compatibility

Existing dockerfile_template and compose_template columns remain functional. New tool types use the manifest system; old types continue to work. A data migration converts the existing pi-agent to the new format.

R8. Local Builds

Images are built locally per instance using the standard docker build command. No registry integration in this phase.

R9. Live Preview

The API provides a compile endpoint that returns the generated Dockerfile and Compose file without building.

R10. Deterministic Image Tags

The image tag is derived from a hash of the manifest content, enabling build cache reuse when the manifest hasn't changed.

---

Scenarios

S1. Creating a New Tool Definition

Given a user on the Tool Workshop page
When they select base "ubuntu-24.04-dev:v1", add packages [neovim, tmux], add a build script for git config, and define mounts for workspace + ssh
Then the system generates a manifest, compiles a Dockerfile + Compose preview, and upon save stores the manifest in the database.

S2. Building an Instance from a Manifest

Given a tool instance created from a manifest-based tool type
When start_instance is called
Then the API compiles the manifest to a Dockerfile, builds the image, generates the Compose file with resolved mount paths, starts the container, and applies permission policies post-start.

S3. Permission Fix on Non-Root Containers

Given a manifest with user: {name: user, uid: 1001} and a mount target: /workspace, owner: user
When the container starts with the workspace bind-mounted from host (root-owned)
Then the post-start permission fixer runs docker exec --user root chown -R user:user /workspace, making the directory writable for the container user.

S4. SSH Key Mount for Non-Root User

Given a manifest with a mount target: /home/user/.ssh, source_type: ssh_key, mode: "0700"
When the container starts
Then SSH keys are mounted from the instance .ssh directory to /home/user/.ssh, and post-start fixes permissions to 0700 for the directory and 0600 for key files.

S5. Config Profile Extends Manifest

Given a manifest with default mount workspace: /workspace and a ConfigProfile that adds git_mounts: [{remote_url: "...", target_path: "/home/user/.config"}]
When the instance starts with that profile selected
Then the final Compose includes both the workspace mount and the config git mount, merged in the correct precedence.

S6. Base Version Pinning

Given a tool definition referencing base_definition_id: "ubuntu-24.04-dev", base_version: "v1"
When the base definition is updated to "v2"
Then the tool definition continues to use "v1" until explicitly updated. New tool definitions default to the latest version.

S7. Deterministic Image Tag

Given a manifest with specific packages and scripts
When compiled
Then the generated image tag is headquarter/{tool-name}-{manifest-hash}:latest, and rebuilding the same manifest reuses the cached image layer.

S8. Live Preview Without Build

Given a manifest being edited
When the user clicks "Preview"
Then the API returns the generated Dockerfile and Compose file within 500ms, without invoking Docker.

---

Acceptance Criteria

A1. Manifest Schema Validation

• The manifest JSON must validate against a defined JSON Schema
• Invalid manifests return 400 with detailed field-level errors
• Missing required fields (name, base_image or base_definition_id) are rejected

A2. Dockerfile Compilation

• Generated Dockerfile builds successfully with docker build
• Build scripts appear as RUN commands in order
• Startup scripts appear in the generated entrypoint script
• Packages are installed in a single layer per package manager
• User creation uses the declared uid/gid

A3. Compose Compilation

• Generated Compose file starts successfully with docker compose up
• Mounts are resolved from source_type to actual host paths
• stdin_open and tty are set for terminal interface types
• Ports are only included for web interface types

A4. Permission Fixer

• Post-start chown runs for all mounts with an owner declared
• Post-start chmod runs for all mounts with mode or file_mode declared
• Permission fixes complete within 5 seconds of container start
• If the container has no root user, permission fixes are skipped with a warning

A5. Config Profile Merge

• ConfigProfile env vars override manifest defaults
• ConfigProfile mounts are appended to manifest mounts
• ConfigProfile git_mounts are resolved and appended
• ToolConfig values override both manifest and ConfigProfile

A6. Backward Compatibility

• Existing tool types with dockerfile_template continue to work
• Existing tool types with compose_template continue to work
• The pi-agent tool type is migrated to the new manifest format
• Old and new tool types can coexist in the same project

A7. Base Versioning

• Base definitions store a version string
• Tool definitions store the base version they reference
• Updating a base creates a new version; old versions remain accessible
• The "latest" version can be referenced explicitly or by omission

A8. Image Tag Determinism

• Same manifest produces the same image tag
• Changing any field (package, script, env) produces a different tag
• The tag is lowercased and valid as a Docker image reference

A9. API Endpoints

• POST /tool-definitions creates a definition (201)
• GET /tool-definitions/{id} returns the definition with compiled preview
• POST /tool-definitions/{id}/compile returns Dockerfile + Compose (no build)
• PUT /tool-definitions/{id} updates and re-validates

A10. Frontend Tool Workshop

• Users can create a tool definition via form (no raw JSON editing required)
• Live preview shows generated Dockerfile and Compose
• Package lists support add/remove/reorder
• Mount schema supports add/remove with visual feedback
• Base image selector shows available versions

---

Non-Goals

• Multi-stage builds — Out of scope for this phase. The compiler generates single-stage Dockerfiles.
• Docker registry integration — Images are built locally per instance.
• Binary build context files — Build context is limited to text files stored in the DB.
• Custom Dockerfile editing — Users work exclusively through the manifest; no raw Dockerfile editing.
• Container orchestration beyond Compose — No Kubernetes, Swarm, or other orchestrators.
• Real-time collaborative editing — Tool Workshop is single-user editing.

---

API Contract

POST /tool-definitions

Request:

{
  "name": "pi-agent",
  "display_name": "Pi Agent",
  "description": "Terminal coding harness",
  "category": "development",
  "interface_type": "terminal",
  "base_image": "ubuntu:24.04",
  "packages": {
    "apt": ["curl", "git", "neovim", "tmux"],
    "node": {"version": "20"},
    "npm_global": ["@earendil-works/pi-coding-agent"]
  },
  "user": {
    "name": "user",
    "uid": 1001,
    "gid": 1001,
    "create_home": true,
    "shell": "/bin/bash"
  },
  "env": {"DEBIAN_FRONTEND": "noninteractive"},
  "scripts": {
    "build": [
      "git config --global init.defaultBranch main",
      "mkdir -p /home/user/.config/ranger"
    ],
    "startup": [
      "if [ -d /workspace ]; then sudo chown -R user:user /workspace; fi"
    ]
  },
  "mounts": [
    {
      "name": "workspace",
      "target": "/workspace",
      "source_type": "repo",
      "writable": true,
      "owner": "user"
    },
    {
      "name": "ssh",
      "target": "/home/user/.ssh",
      "source_type": "ssh_key",
      "mode": "0700",
      "file_mode": "0600",
      "readonly": true
    }
  ],
  "runtime": {
    "command": ["/bin/bash"],
    "stdin_open": true,
    "tty": true,
    "working_dir": "/workspace"
  }
}

Response (201):

{
  "id": "d07b8376-2151-4119-8c1d-27f792aae9a3",
  "name": "pi-agent",
  "display_name": "Pi Agent",
  "manifest": { ... },
  "dockerfile_preview": "FROM ubuntu:24.04\n...",
  "compose_preview": "services:\n  app:\n    image: ...",
  "created_at": "2026-05-28T10:00:00Z"
}

POST /tool-definitions/{id}/compile

Response (200):

{
  "dockerfile": "FROM ubuntu:24.04\n...",
  "compose": "services:\n  app:\n...",
  "image_tag": "headquarter/pi-agent-a3f7c2d9:latest",
  "mounts_resolved": [
    {"name": "workspace", "source": "/data/repos/headquarter", "target": "/workspace"}
  ]
}

---

Database Schema

tool_definition_manifests

Column	Type	Constraints
id	UUID	PK
name	VARCHAR(64)	UNIQUE, NOT NULL
display_name	VARCHAR(128)	NOT NULL
description	TEXT
category	VARCHAR(64)
interface_type	VARCHAR(16)	CHECK IN ('web', 'terminal')
base_image	VARCHAR(256)
base_definition_id	UUID	FK → tool_definition_manifests.id
base_version	VARCHAR(32)	DEFAULT 'latest'
manifest	JSONB	NOT NULL
dockerfile_cache	TEXT
compose_cache	TEXT
version	VARCHAR(32)	DEFAULT 'v1'
is_base	BOOLEAN	DEFAULT FALSE
created_by_id	UUID	FK → users.id
created_at	TIMESTAMPTZ	DEFAULT now()
updated_at	TIMESTAMPTZ	DEFAULT now()

Check constraint: Exactly one of base_image or base_definition_id must be set.

Alter tool_types

ALTER TABLE tool_types
  ADD COLUMN manifest_id UUID REFERENCES tool_definition_manifests(id),
  ADD COLUMN definition_type VARCHAR(16) DEFAULT 'legacy';  -- 'legacy' | 'manifest'

Alter tool_instances

ALTER TABLE tool_instances
  ADD COLUMN manifest_compiled_at TIMESTAMPTZ,
  ADD COLUMN image_tag VARCHAR(256);

---

Related Files

• apps/api/src/models/tool_definition_manifest.py — New model
• apps/api/src/models/tool_type.py — Add manifest_id, definition_type
• apps/api/src/services/manifest_compiler.py — New compiler
• apps/api/src/services/permission_fixer.py — Refactored mount policy applier
• apps/api/src/api/tool_definitions.py — New endpoints
• apps/api/src/api/tool_instances.py — Modified start_instance flow
• apps/api/alembic/versions/20260528_add_tool_definition_manifests.py — Migration