Replace the subprocess-based tunnel implementation with Docker containers
running on the host network. This eliminates all container name resolution
bugs that caused tunnel 502 errors.
New design:
- Each tunnel is a docker run --network host cloudflare/cloudflared container
- cloudflared connects to localhost:{published_port} (Docker port forwarding)
- No dependency on container names, backend network DNS, or binding diagnostics
- Tunnels named predictably: tunnel-{instance_name}
- Start/stop/recreate use container names instead of PIDs
Files changed:
- NEW: apps/api/src/services/tunnel.py — clean tunnel module (start/stop/recreate/health)
- apps/api/src/services/docker.py — removed 250 lines of old tunnel code
- apps/api/src/api/tool_instances.py — use new tunnel module, store container_name
- apps/api/src/services/health_monitor.py — updated import
- apps/web/src/components/session-card.tsx — Recreate Tunnel button always visible
Quality gates: ruff clean, 13 tests passed (health_monitor + notifications)
- Inject explicit container_name into compose files at start/restart time
via _ensure_container_name_in_compose() to prevent Docker Compose from
generating UUID-based auto names that break backend network resolution.
- Use instance.name.lower() directly instead of get_container_name() lookups
which were unreliable with auto-generated names.
- Apply compose sanitization, bind-address fix, and container-name injection
on restart_instance as well so restarts pick up template fixes.
- Add --force-recreate to docker compose up to ensure container_name changes
take effect immediately.
- Fix notification lifecycle tests to match current behavior (success severity,
health_changed event for ownership test).
Quality gates: ruff clean, pytest (7 notification lifecycle tests passed)
FastAPI matches routes in declaration order. The DELETE /notifications
endpoint (bulk clear) was registered AFTER DELETE /notifications/{id},
so the path parameter route intercepted all requests to the bulk route,
causing a 422 UUID validation error instead of hitting clear_all.
Moved clear_all_notifications above dismiss_notification in the router.
Added regression test to verify route order.
Quality gates: pytest (22 passed)
Notification filtering:
- lifecycle_hooks.py: only instance.error and instance.health_changed
with status=running generate notifications. All other lifecycle events
(created, started, stopped, restarted, deleted) are filtered out.
- health_monitor.py: only error and unhealthy states generate notifications.
Running/recovered state no longer creates info notifications.
- _derive_title now maps instance.health_changed to "Container ready".
Clear-all button:
- Added dismiss_all() to NotificationService
- Added DELETE /notifications endpoint for bulk dismiss
- Frontend: clearAllNotifications API, clearAll in notification context,
"Clear all" button in notification drawer alongside "Mark all as read"
- Added CSS for .notification-clear-all with danger hover state
- Updated notification-center tests
Quality gates: pytest (21 passed), vitest (11 passed)
Root cause: code-server (and similar web tools) default to binding to
127.0.0.1 (localhost) inside their containers. This makes them unreachable
from the Docker network and from cloudflared, which connects via the
container's Docker network name.
Changes:
- Migration: Update code-server compose_template to include
--bind-addr 0.0.0.0:8443 command override
- Migration: Update jupyter-notebook compose_template to include
--ip=0.0.0.0 flag
- Runtime safety net: _ensure_web_bind_address() auto-injects bind
address for known web tools (code-server, jupyter-notebook) when
compose doesn't already specify a command
- Diagnostics: _check_app_binding() compares internal vs external
connectivity to detect 127.0.0.1 binding issues
- Improved readiness check: 30s timeout, checks HTTP status codes,
logs curl stderr for debugging
Files:
- apps/api/alembic/versions/2026_05_29_fix_web_tool_bind_address.py
- apps/api/src/services/docker.py
- apps/api/src/api/tool_instances.py
- apps/api/tests/integration/test_tool_types_api_extended.py
Quality gates: pytest 42 passed (5 pre-existing unrelated failures)
- Add _check_app_binding() to detect if app is bound to 127.0.0.1
instead of 0.0.0.0 (common cause of tunnel 'app error 0')
- Improve curl readiness check: wait up to 30s, check HTTP status codes
(accept 2xx, 3xx, 401, 403 as 'ready')
- Log curl stderr for connection debugging
- Log binding diagnosis when external connectivity fails
Quality gates: pytest 42 passed
The API container and tool instances share the 'backend' Docker network
(connect_container_to_network at tool_instances.py:1576). cloudflared
runs INSIDE the api container, so localhost:host_port is unreachable.
The original container_name:internal_port is correct for networking.
The 'app error 0' is an application-level issue, not networking.
This reverts commit a8fbca9.
- apply_mount_permissions now skips mounts with readonly=true to avoid
'Read-only file system' warnings on post-start chown/chmod
- Removed the ssh_keys mount from the pi-agent manifest definition;
instance-level SSH key mounting now handles this exclusively
- Added unit test for read-only mount skipping
Quality gates: pytest (15 passed)
Root cause: start_cloudflared_tunnel was trying to connect to
http://{container_name}:{container_port}, but:
1. The host OS cannot resolve Docker container names
2. cloudflared runs on the host, so it needs the host-mapped port
Changes:
- start_cloudflared_tunnel: changed signature to accept host_port only
- Connects cloudflared to localhost:{host_port} via Docker port mapping
- Connectivity check uses localhost:{host_port}
- recreate_tunnel updated to match new signature
- Callers in tool_instances.py pass instance.port (host port)
Quality gates: pytest 42 passed
- start_instance now deep-merges manifest with base definition before extracting user.uid/user.gid
- The user config is typically defined in the base image (ubuntu-24.04-dev), not the extending manifest
- Add debug logging to verify resolved uid/gid/home_dir
- Add logging to prepare_ssh_key_files for chown success/failure visibility
- Log current process uid when chown fails to diagnose permission issues
Quality gates: pytest 239 passed (6 pre-existing failures), tsc --noEmit clean
- Extend prepare_ssh_key_files() with optional uid/gid parameters
- Call os.chown on created files when uid/gid are provided
- Gracefully handle PermissionError if API process is not root
- In start_instance, extract container user UID/GID from manifest
- Pass container UID/GID when preparing instance-level SSH key mounts
- Legacy clone-mode SSH keys continue to use root (0,0)
- Add unit tests for prepare_ssh_key_files ownership logic
- Keep apply_ssh_permissions() as fallback for cases where host chown fails
Quality gates: pytest 239 passed (6 pre-existing failures), tsc --noEmit clean
- Replace apply_ssh_permissions internals with _exec_and_log for full visibility
- Log every docker exec command, stdout, and stderr at DEBUG level
- After chown/chmod, run ls -la and stat to verify final state
- Log verified state at INFO level so users can see exactly what happened
- Update tests to mock subprocess.run instead of _run_in_container
Quality gates: pytest 236 passed (6 pre-existing), tsc --noEmit clean
- lifecycle_hooks.publish_lifecycle_event now skips notification creation
when event_type='instance.started' and status='starting'
- Users only see notifications for terminal states:
- Failed: instance.error
- Successful: instance.health_changed with status='running'
- Updated integration tests to verify new behavior:
- test_lifecycle_started_intermediate_skips_notification
- test_lifecycle_running_creates_notification
Quality gates: pytest 42 passed, ruff clean
- Add ssh_key_id column to ConfigProfile model and migration
- Update config profile API to accept/return ssh_key_id
- Include ssh_key_id in ResolvedProfile and resolver logic
- Mount selected SSH key into container home dir at start_instance
- Frontend config profile form with SSH key selector dropdown
- Git mount URL validation defaults to profile's SSH key
Quality gates: pytest (231 passed, 6 pre-existing), tsc --noEmit clean
- Add notifications table with Alembic migration
- Notification model with user-scoped indexing and partial index on unread
- NotificationService singleton with create/list/count/mark-read/dismiss
- FastAPI router: GET /notifications, GET /unread, PATCH /{id}/read,
POST /mark-all-read, DELETE /{id}
- Mute categories filtering from UserConfig
- 13 unit tests for NotificationService
- 10 integration tests for API endpoints
- Updated test_models.py with new table registration
Quality gates: pytest 23 new passed, ruff clean
The previous sorting fix exposed a deeper bug: ResolvedMount always
mounted its staging directory as a single bind mount. When a config
profile mount targeted /workspace/x/y and contained a single file
z.json, the staging directory (containing only z.json) replaced the
ENTIRE /workspace/x/y directory, hiding all sibling files from git
repo mounts.
- Change apply_resolved_profile to mount each file individually:
- source: staging_dir/relative_path
- target: expanded_target/relative_path
- Sibling files from other mounts are preserved.
- Empty mounts produce no volume entries.
- Keep volume sorting (parent paths before child paths) which is
still necessary for directory mounts and ensures parent dirs exist
before file mounts inside them.
- Add 4 unit tests for file-level mount behavior.
Quality gates: pytest (218 passed, 6 pre-existing), tsc --noEmit (clean)
- Add instance_events and health_checks tables with Alembic migration
- InstanceEventBus: typed pub/sub singleton with wildcard support
- HealthMonitor: async background loop polling containers every 15s
- SSE endpoint GET /events/stream with auth and connection limits
- Lifecycle hooks in tool_instances.py (create/start/stop/restart/delete)
- Structured JSON logging with correlation IDs
- 15 new unit tests (EventBus, HealthMonitor, MonitoringModels)
Quality gates: pytest 15 new passed, ruff clean
- Add expand_container_path() helper that resolves ~/ and $HOME/ prefixes
- Add get_manifest_home_dir() to compute /home/{user.name} or /root from manifest
- Set ENV HOME=... and ENV USER=... in generated Dockerfile for runtime compatibility
- Pass home_dir through instance creation and startup pipeline
- Expand mount targets in apply_resolved_profile() for regular profile mounts
- Expand mapping targets in _resolve_git_mount_mappings() for git mounts
- Expand working_directory and volume targets in _modify_compose_file()
- Update _prepare_manifest_instance to return home_dir alongside image tag
- Fetch tool_type early in start_instance to determine home_dir before profile application
Quality gates: pytest 188 passed, frontend typecheck clean
Addresses: home-path-expansion
- Add mappings array support to git_mount entries
- Clone repository once per git_mount entry, mount multiple subdirectories
- Normalize legacy source_path+target_path to mappings on read
- Update _merge_git_mounts to dedup by (remote_url, branch) and concatenate mappings
- Add _normalize_git_mount, _clone_git_repo, _resolve_git_mount_mappings helpers
- Update GitMountItem Pydantic model with GitMountMapping and model_validator
- Update frontend GitMountEditor component with mappings UI
- Auto-convert legacy git mount entries to mappings format on load
- Add 15 backend unit tests for normalization, resolution, and glob expansion
- Update existing config profile resolver tests for new merge behavior
Quality gates: pytest 167 passed, frontend typecheck clean
Addresses: config-profile-multi-repo-mounts
Frontend:
- Remove 4 console.log statements from terminal.tsx that flooded the
browser console with WebSocket traffic (open, received X bytes, sending Y,
xterm focused)
Backend:
- Downgrade Dockerfile/entrypoint compilation logs from INFO to DEBUG in
_prepare_manifest_instance
- Remove hex-dump diagnostic logging from docker_build.py (was for
troubleshooting the backslash continuation bug, now fixed)
- Downgrade Dockerfile write log from INFO to DEBUG
The compile_dockerfile function used \\\\ in Python string literals,
which produces \ (two backslashes) in the Dockerfile output. Docker's
legacy builder requires a single backslash \ for line continuation.
This caused 'unknown instruction: curl' because Docker saw the first as the continuation and the second \ as a literal character before the
newline, breaking the RUN command parsing.
Fix: change all \\ to \ in Python string literals within
compile_dockerfile, producing the correct single-backslash continuation.
Verified with hex dump from container logs:
- Before: line ended with 5c5c (two backslashes)
- After: line ends with 5c (one backslash)
Docker's legacy builder treats \r as a literal character after a backslash
continuation, breaking RUN multi-line commands and producing
'unknown instruction: curl' errors.
Add defensive CRLF→LF normalisation for both Dockerfile and build context
files before writing. Also log hex representation of first 8 lines so we
can verify exactly what bytes Docker receives.
The container build fails with 'unknown instruction: curl' on line 6, which
suggests the Dockerfile continuation characters or line endings may be
malformed. Add defensive logging to diagnose:
- Force newline='\n' in all write_text calls in build_image for consistent
Unix line endings regardless of platform
- Log compiled Dockerfile and entrypoint content at INFO/DEBUG level
- Log Dockerfile byte count when written
This will let us see exactly what Docker is receiving in the next build attempt.
The terminal_sessions migration and drop_tool_configs migration both pointed
to add_tool_definition_manifests as their down_revision, creating two heads.
Update drop migration to depend on terminal_sessions instead, restoring a
single linear chain.
Three fixes for multi-session terminal bugs:
1. Race-condition double creation: The auto-create effect fired twice because
loadSessions returned 0 while an earlier createSession was still in flight.
Added hasAutoCreated guard ref to ensure only one auto-create happens.
2. Page reload spawns new sessions: After server restart, list_terminal_sessions
filtered out DB-only sessions (no in-memory counterpart), so the frontend
thought no sessions existed and auto-created new ones. Reverted the filter
so DB rows are always returned. The WebSocket handler now restores the
in-memory session from the DB row on demand when connecting.
3. No input after connection: The backend _read_loop would break on any send
error, causing asyncio.wait to cancel the _write_loop. Made _read_loop
retry up to 3 times before giving up, preventing transient send errors
from killing input handling.
Quality gates: pytest (15/15 passed), tsc clean
- Add ToolDefinitionManifest model with base image versioning
- Add manifest compiler: Dockerfile + Compose generation from JSON manifests
- Add permission fixer: post-start chown/chmod for mount policies
- Add tool definition CRUD API with live compile preview endpoint
- Integrate manifest-based startup flow in start_instance
- Add Alembic migration with data conversion for pi-agent
- Add 48 unit tests for manifest compiler, permission fixer, docker service
- Keep backward compatibility with legacy dockerfile_template/compose_template
Migration: applied successfully. Pi-agent converted to manifest.
Quality gates: pytest (146 passed, 4 pre-existing unrelated failures)
- Add TerminalSessionModel DB table with instance_id FK, name, status,
created_at, last_activity_at, closed_at columns
- Add Alembic migration for terminal_sessions table
- Refactor TerminalManager to use composite key (instance_id, session_id)
supporting up to 5 concurrent sessions per instance
- Add create_session, get_session, get_sessions_for_instance, close_session
- Preserve get_or_create_session for backward compatibility (default session)
- Fix attach_websocket to only close sockets within same session
- Add name (auto-generated 'Session N') and status tracking to TerminalSession
- Add 7 unit tests for multi-session logic
Quality gates: pytest (7 new passed, 174 total passed, 51 pre-existing failures)
- get_container_id() and get_container_name() now lowercase the
instance name before passing to docker ps --filter, because
Docker container names are lowercase internally and the filter
is case-sensitive. This caused container_id to never be captured
when instance.name contained uppercase chars (e.g. 'Headquarter'),
breaking terminal WebSocket connections.
- Also guard proc.stdout being None in start_cloudflared_tunnel().
- Add unit tests for get_container_id and get_container_name.
Quality gates: pytest (14 passed), python clean
- Add stdin_open: true and tty: true to dockerfile-based compose generation.
Without these, bash (PID 1) exits immediately, causing a container restart
loop that makes the instance invisible to docker ps and triggers 4004.
- Treat WebSocket close codes 4001/4003/4004 as permanent errors in the
frontend. Stop retrying and show the server reason to the user.
- Prevent visibilitychange handler from resetting retry attempts after a
permanent error has occurred.
- Use docker ps -a in get_container_id/get_container_name to find
stopped/exited containers for diagnostics.
Quality gates: tsc --noEmit (pass), pytest (98 passed, 4 pre-existing failures)
- Change git mount schema from repo_id to remote_url
- Remove database lookups for git mount resolution
- Clone directly from URL at instance startup
- Simplify frontend UI to text input for Git URL
- Fix route ordering in git_repositories.py to prevent 422 errors
- Update all tests to use remote_url field
Breaking change: Git mounts now use remote_url instead of repo_id
- Add startup_command field to ToolType model and API
- Execute startup command before interactive shell in terminal sessions
- Add tmux and ranger to OpenCode container spec
- Update Tool Workshop UI with startup_command input for terminal types
- Add backend tests for startup_command CRUD operations
- Sync specs: tool-terminal, tool-types-definition, opencode-web-server
- New spec: tool-terminal-startup-command
Quality gates: Frontend typecheck/lint passed. Backend tests blocked by environment (Python/Docker not available).
OpenSpec: terminal-startup-and-container-tools
Instead of sending stty commands through the user's terminal session
(which causes 'inappropriate ioctl' errors), send SIGWINCH signal to
the docker exec process on the host. Docker exec should forward this
to the container process, causing the shell to re-read its terminal size.
This avoids:
- Visible stty commands in the terminal
- ioctl errors from stty
- Interference with user's shell session
Reverted docker exec back to -it (required for interactive bash).
Instead, sends stty command with \r to hide it from the terminal display:
- \r moves cursor to start of line (overwrites prompt)
- stty command executes silently (no output on success)
- \r moves cursor back to start, hiding echoed command
This sends stty on EVERY resize so the container shell always matches
frontend dimensions.
alex
Alex Blank