headquarter/openspec/explorations/streamline-tool-container-definitions.md

# SDD Exploration: Streamline Tool Container Definitions

## Status
**Phase:** explore  
**Date:** 2026-05-28  
**Owner:** el Gentleman (parent session)  
**Scope:** Tool container definition, build, and mount system

---

## Executive Summary

The current tool container system works for the happy path (pi-agent on Ubuntu) but has deep structural inflexibility:

1. **Monolithic Dockerfile strings** in the database — impossible to review, version, or compose
2. **Ad-hoc compose generation** — string formatting with hardcoded fields (`stdin_open`, `tty`, `working_dir` missing for dockerfile types)
3. **Hardcoded mount paths** — `/workspace` and `/root/.ssh` don't adapt to the container's runtime user
4. **No package/base-image modularity** — every tool type carries a full Dockerfile copy
5. **Permission mismatch** — bind mounts come in as root-owned; non-root container users can't write
6. **Config overlap** — tool configs, config profiles, and compose templates fight for control of the same fields

This exploration proposes a **layered, declarative container definition system** where tool types compose from reusable base images, mount schemas, and permission policies.

---

## Current Architecture Map

### Data Model

```
ToolType (DB table)
  ├── name, display_name, description, category
  ├── interface_type: "web" | "terminal"
  ├── definition_type: "dockerfile" | "compose"
  ├── dockerfile_template: TEXT (giant Dockerfile string)
  ├── compose_template: TEXT (Jinja-like {{VAR}} string)
  ├── build_context: JSON {path: content}
  ├── required_variables: JSON ["REPO_PATH", ...]
  ├── default_port: int
  └── readiness_probe: JSON

ToolInstance (DB table)
  ├── name, display_name, status
  ├── tool_type_id → ToolType
  ├── repository_id → GitRepository
  ├── compose_path: str
  ├── container_id, container_name
  ├── port, url, public_url, tunnel_id
  ├── clone_mode: "mount" | "clone"
  ├── branch, new_branch
  └── selected_config_profile_id → ConfigProfile

ToolConfig (DB table, per-user per-tool-type)
  ├── config_type: "env" | "file"
  ├── key, value, file_path
  ├── port_override, start_command, working_directory
  ├── environment_variables: JSON
  └── volumes: JSON [{source, target, type}]

ConfigProfile (DB table)
  ├── name, description
  ├── user_id, project_id, tool_type_id
  ├── environment_variables: JSON
  ├── files: JSON {path: content}
  ├── mounts: JSON [{source, target, type}]
  ├── git_mounts: JSON [{remote_url, source_path, target_path, branch}]
  └── parent_profile_id → ConfigProfile (hierarchy)
```

### Creation Flow (`create_instance`)

```
POST /projects/{id}/repositories/{id}/instances
  → validate tool_type, repo, config_profile
  → generate instance_name = "{tool_type}-{repo}-{uuid8}"
  → ensure_instance_directory(instance_name)
  → find_free_port()
  → determine repo_path (mount = repo.path; clone = clone_repository())
  → IF tool_type.definition_type == "dockerfile":
       build_image(instance_dir, dockerfile_template, tag, build_context)
       generate compose_content (HARDCODED STRING FORMATTING)
     ELSE:
       render_compose_template(tool_type.compose_template, variables)
  → write_compose_file()
  → create ToolInstance DB record (status="pending")
```

### Startup Flow (`start_instance`)

```
POST /instances/{id}/start
  → fetch ToolConfigs (env, files, port_override, start_command, working_dir, volumes)
  → IF selected_config_profile:
       resolve_profile() → env, files, mounts, git_mounts, hints
  → write .env file, config files
  → IF clone_mode: mount SSH keys at /root/.ssh (HARDCODED)
  → _modify_compose_file(port, command, working_dir, extra_volumes)
  → _sanitize_compose_file()
  → execute_compose_command("up")
  → get_container_id(instance.name)  ← CASE-SENSITIVE BUG (fixed)
  → get_container_name(instance.name)
  → connect_container_to_network("backend")
  → wait_for_container_running()
  → IF web: start_cloudflared_tunnel()
  → instance.status = "running"
```

### Key Files

| File | Responsibility |
|------|---------------|
| `apps/api/src/api/tool_instances.py` | create_instance, start_instance, stop_instance, restart_instance, proxy, logs |
| `apps/api/src/api/tool_types.py` | CRUD for ToolType (DB strings) |
| `apps/api/src/services/docker.py` | compose execution, container queries, tunnel management |
| `apps/api/src/services/docker_build.py` | `docker build` wrapper |
| `apps/api/src/services/terminal_session.py` | PTY-based terminal over `docker exec` |
| `apps/api/src/services/terminal_manager.py` | WebSocket ↔ terminal session lifecycle |
| `apps/api/src/models/tool_type.py` | SQLAlchemy model |

---

## Pain Points (Detailed)

### 1. Monolithic Dockerfile Templates

The pi-agent Dockerfile template is a 40-line string stored in the DB migration:

```sql
INSERT INTO tool_types (... dockerfile_template ...)
VALUES ('...# Pi Coding Agent - Terminal-based coding harness\nFROM ubuntu:24.04\n...')
```

**Problems:**
- No syntax highlighting, linting, or `docker build` validation at edit time
- Every tool type copies the entire Dockerfile; no reuse of common layers
- Changes require a DB migration
- No way for users to customize packages without forking the whole template

### 2. Ad-Hoc Compose Generation

For `dockerfile` type tools, the compose is generated by Python f-string:

```python
compose_content = f"""version: "3.8"
services:
  app:
    image: {image_tag}
    container_name: {instance_name.lower()}
{ports_section}    volumes:
      - {repo_path}:/workspace
    restart: unless-stopped
"""
```

**Problems:**
- Missing `stdin_open: true` and `tty: true` (essential for terminal tools)
- Missing `working_dir: /workspace`
- No way to add labels, networks, healthchecks, or extra services
- Port section is conditionally included with awkward string concatenation

### 3. Hardcoded Mount Paths

| Mount | Current Target | Problem |
|-------|---------------|---------|
| Repository | `/workspace` | Always root-owned; no permission fix for non-root users |
| SSH keys (clone mode) | `/root/.ssh` | Invisible to containers running as `user` |
| Git-mount configs | `/tmp/.pi` | May be root-owned; conflicts with user's `.pi` |
| Config profile files | Instance-relative paths | No validation against container filesystem |

### 4. No Base Image / Layer Composition

Every tool type must specify a complete Dockerfile from `FROM` to `CMD`. There's no way to say:

```yaml
base: ubuntu-24.04-dev  # pre-built with curl, git, build-essential
layers:
  - nodejs-20
  - pi-coding-agent
  - custom-packages: [neovim, ranger, tmux]
```

### 5. Permission Mismatch (Non-Root Users)

The pi-agent Dockerfile creates a `user` account and uses `USER user`. Bind mounts from the host come in as root-owned. The API has **no automatic permission fix** — this caused the workspace-unwritable bug.

Workarounds considered:
- Post-start `docker exec --user root chown` (current fix)
- Dockerfile entrypoint script that chowns before dropping privileges
- Matching container UID to host UID

None of these are systematic or configurable.

### 6. Config Overlap and Precedence Confusion

Three systems control the same container aspects:

| System | Controls | Stored |
|--------|----------|--------|
| ToolConfig | env vars, files, port, command, working_dir, volumes | DB (per-user per-tool) |
| ConfigProfile | env vars, files, mounts, git_mounts, hints | DB (hierarchical) |
| Compose template / generation | volumes, ports, command, working_dir | DB string / Python f-string |

**Precedence is unclear:**
- ToolConfig `working_directory` vs ConfigProfile hint `working_directory` vs compose `working_dir`
- ToolConfig `volumes` vs ConfigProfile `mounts` vs compose `volumes`
- `start_command` from ToolConfig vs ConfigProfile vs Dockerfile `CMD`

### 7. Build Context Limitations

`build_context` is a JSON dictionary of `{relative_path: file_content}`. This is stored in the DB as text.

**Problems:**
- Binary files (images, tarballs) can't be stored
- Large files bloat the DB
- No versioning or external reference (e.g., "use file from git repo")

---

## Extensibility Gaps

| Want | Current State | Gap |
|------|--------------|-----|
| Add a new language runtime (e.g., Go, Rust) | Copy entire Dockerfile, edit | No modular package/layer system |
| Use a custom base image (e.g., `my-registry/dev-base:v2`) | Edit full Dockerfile | No base-image reference field |
| Mount a second repo or a secrets file | Write ConfigProfile or ToolConfig JSON | No declarative mount schema |
| Run as root instead of `user` | Edit full Dockerfile | No runtime-user field |
| Add a sidecar (e.g., postgres for integration tests) | Edit compose_template string | No multi-service compose support |
| Pre-install VS Code server | Edit full Dockerfile | No "feature" or "extension" mechanism |
| Custom entrypoint script | Edit full Dockerfile | No entrypoint field |

---

## Design Directions (Pre-Proposal)

### Direction A: Declarative Tool Manifests

Replace the monolithic `dockerfile_template` with a structured manifest:

```yaml
# Example: tool manifest for pi-agent
name: pi-agent
base_image: ubuntu:24.04
user:
  name: user
  uid: 1000
  home: /home/user
packages:
  apt: [curl, wget, git, neovim, ranger, tmux, htop, tree, jq, python3, python3-pip, build-essential]
  npm_global: [@earendil-works/pi-coding-agent]
  node_version: "20"
env:
  DEBIAN_FRONTEND: noninteractive
config_files:
  /home/user/.tmux.conf: "set -g mouse on\n..."
  /home/user/.config/ranger/rc.conf: "set preview_files true\n..."
working_directory: /workspace
command: ["/bin/bash"]
ports: []
mounts:
  repo: {target: /workspace, writable: true}
  ssh: {target: /home/user/.ssh, mode: "0600"}
  tmp_state: {target: /tmp/.pi, writable: true}
```

**Pros:** Structured, reviewable, composable  
**Cons:** Requires a manifest-to-Dockerfile compiler; migration complexity

### Direction B: Base Image Registry + Layers

Maintain a registry of pre-built base images:

```
headquarter/base/ubuntu-24.04-dev
headquarter/base/nodejs-20
headquarter/base/python-3.11
```

Tool types reference a base image and a list of layers:

```yaml
base_image: headquarter/base/ubuntu-24.04-dev
layers:
  - type: npm_install
    package: @earendil-works/pi-coding-agent
  - type: config_file
    path: /home/user/.tmux.conf
    content: "..."
```

**Pros:** Fast builds (base images cached), reusable, versioned  
**Cons:** Requires image registry management, layer ordering complexity

### Direction C: Compose-First with Dockerfile Overrides

Treat `compose_template` as the primary definition. For simple cases, use a pre-built image. For custom cases, allow an inline Dockerfile or a `build` section in the compose:

```yaml
services:
  app:
    build:
      context: .
      dockerfile_inline: |
        FROM ubuntu:24.04
        ...
    stdin_open: true
    tty: true
    working_dir: /workspace
    volumes:
      - ${REPO_PATH}:/workspace
      - ${SSH_PATH}:/home/user/.ssh:ro
    user: "${CONTAINER_USER:-user}"
```

**Pros:** Leverages Docker Compose native features, familiar to users  
**Cons:** Still string-based; inline Dockerfiles are hard to edit

### Direction D: Permission-Aware Mount Schema

Decouple mount declaration from mount implementation:

```python
class MountPolicy:
    source: str          # host path
    target: str          # container path
    owner: str | None    # container user to own the mount
    permissions: str     # chmod string
    readonly: bool
```

At startup, the API runs a post-start "permission fixer" that applies all policies:

```bash
docker exec --user root <container> chown -R <owner> <target>
docker exec --user root <container> chmod <permissions> <target>
```

**Pros:** Systematic, works with any base image, configurable per mount  
**Cons:** Adds startup latency, requires root to exist in container

---

## Recommended Next Steps

1. **Proposal phase:** Evaluate Direction A (Declarative Manifests) vs Direction C (Compose-First) for the primary architecture
2. **Design phase:** Detail the manifest schema or compose enhancement, migration path, and API changes
3. **Consider Direction D** as a cross-cutting concern regardless of primary direction

---

## Risks

- **Migration risk:** Existing `dockerfile_template` and `compose_template` columns need backward-compatible migration
- **Build cache invalidation:** Changing the build system may invalidate Docker layer caches
- **User confusion:** Adding a manifest layer on top of Dockerfiles may feel like "yet another abstraction"
- **Scope creep:** This touches tool types, tool configs, config profiles, compose generation, and the startup flow — high cross-cutting surface

---

## Artifacts

- `openspec/config.yaml` — SDD configuration
- `openspec/explorations/streamline-tool-container-definitions.md` — This document