- apps/api/src/services/tunnel.py: add 2-second sleep after discovering the
tunnel URL to allow Cloudflare DNS edge propagation before returning
- apps/web/src/hooks/use-instance-actions.ts: show alert() with the backend
error message when recreate tunnel fails, instead of silently swallowing
errors
Quality gates: ruff clean, tsc clean
Docker container names are case-sensitive for 'docker inspect' but case-
insensitive for Docker DNS. Compose templates may render container names
with mixed case (e.g. code-server-Headquarter-abc123), causing exact-name
docker inspect to fail while DNS resolution in tunnels works fine.
- apps/api/src/services/docker.py: get_container_id now tries exact match
first, then falls back to case-insensitive exact match via 'docker ps'
- apps/api/src/api/tool_instances.py: recreate_tunnel_endpoint uses
get_container_id instead of its own docker inspect call
Quality gates: ruff clean
docker ps --filter name= uses substring matching, so searching for
code-server-headquarter-abc123 also matches tunnel-code-server-headquarter-abc123.
This caused start_instance to store the tunnel container's ID instead of the
tool container's ID, breaking tunnel connectivity and all container operations.
Switched both helpers to docker inspect, which does exact name matching.
Quality gates: ruff clean
Adds INFO-level logging to trace exactly what happens during tunnel
recreation: container lookup, network membership, target IP/URL,
tunnel creation result, health check, and direct curl probe from API.
This will help diagnose why recreated tunnels return 502 while
original tunnels work.
Quality gates: ruff clean
Old instances may have auto-generated Docker Compose container names
that don't match instance.name.lower(), causing DNS resolution failures
for the tunnel. Also, old instances may not be on the backend network.
- apps/api/src/services/docker.py: add get_container_ip_on_network() and
is_container_on_network() helpers
- apps/api/src/services/tunnel.py: start_tunnel() and recreate_tunnel() now
accept an optional target_url parameter to override the default name-based URL
- apps/api/src/api/tool_instances.py: recreate_tunnel_endpoint now:
1. Looks up the tool container (by stored container_id or name)
2. Ensures it's connected to the backend network
3. Gets the container's IP on that network
4. Passes the IP as the explicit tunnel target
This guarantees the tunnel can reach the tool container regardless of
naming or network state.
Quality gates: ruff clean
The previous endpoint blocked recreation if the tunnel was 'healthy'
or returned an 'error_response', making the Recreate Tunnel button
ineffective in many cases.
- apps/api/src/api/tool_instances.py: removed the health-check guards
from recreate_tunnel_endpoint. It now unconditionally stops the old
tunnel and creates a new one, then commits the new URL to the DB.
- Frontend useInstanceActions already calls onRefresh() after success,
so the UI updates with the new tunnel URL automatically.
Quality gates: ruff clean
The post-creation 'docker network connect' was failing silently for
unknown reasons (race condition, container state, or Docker internals).
Instead of fighting with this, we now inject the backend network directly
into the compose file before 'docker compose up'. Docker Compose then
attaches the container to the network atomically during creation.
- apps/api/src/api/tool_instances.py: new _ensure_backend_network_in_compose()
adds 'networks: [backend_name]' to the service and declares the network
as external at the top level
- apps/api/src/api/tool_instances.py: call _ensure_backend_network_in_compose()
in both start_instance and restart_instance, right after
_ensure_container_name_in_compose()
- apps/api/src/api/tool_instances.py: removed connect_container_to_network
import and call entirely
- apps/api/src/api/tool_instances.py: import get_backend_network_name from
docker module for use in the new helper
Quality gates: ruff clean
Docker Compose prefixes network names with the project directory name
(e.g. 'headquarter_backend' instead of 'backend'). The previous code
hardcoded 'backend', causing 'network not found' errors.
- apps/api/src/services/docker.py: add get_backend_network_name() that
inspects the API container (hq-api) to find the actual network name
- apps/api/src/services/docker.py: connect_container_to_network() now
auto-detects the network name when not explicitly provided
- apps/api/src/services/tunnel.py: import and use get_backend_network_name()
- apps/api/src/api/tool_instances.py: remove explicit 'backend' arg from
connect_container_to_network() call
Quality gates: ruff clean
The host-network tunnel approach had issues because localhost inside
the tunnel container wasn't reaching the host-published ports correctly.
This reverts to running cloudflared as a Docker container on the
'backend' network, where Docker DNS resolves container names reliably.
The tunnel connects to http://{container_name}:{container_port}.
- apps/api/src/services/tunnel.py: use --network backend instead of host
- apps/api/src/api/tool_instances.py: pass container_port (default_port)
instead of published_port (host port) to tunnel functions
Quality gates: ruff clean
- Remove --rm from docker run so failed containers persist for inspection
- Add --no-autoupdate flag to prevent cloudflared from exiting on auto-update
- Capture both stdout and stderr from docker logs
- Check container exit code during wait loop; fail fast with logs if container exits early
- Include exit code in timeout error message for easier debugging
Replace the subprocess-based tunnel implementation with Docker containers
running on the host network. This eliminates all container name resolution
bugs that caused tunnel 502 errors.
New design:
- Each tunnel is a docker run --network host cloudflare/cloudflared container
- cloudflared connects to localhost:{published_port} (Docker port forwarding)
- No dependency on container names, backend network DNS, or binding diagnostics
- Tunnels named predictably: tunnel-{instance_name}
- Start/stop/recreate use container names instead of PIDs
Files changed:
- NEW: apps/api/src/services/tunnel.py — clean tunnel module (start/stop/recreate/health)
- apps/api/src/services/docker.py — removed 250 lines of old tunnel code
- apps/api/src/api/tool_instances.py — use new tunnel module, store container_name
- apps/api/src/services/health_monitor.py — updated import
- apps/web/src/components/session-card.tsx — Recreate Tunnel button always visible
Quality gates: ruff clean, 13 tests passed (health_monitor + notifications)
Show the Recreate Tunnel button on all active web-enabled session cards
(instead of only when tunnel_status is unreachable) so users can manually
trigger tunnel recreation at any time. Also adds it to the mobile action sheet.
Quality gates: eslint clean, tsc clean
Root cause: _ensure_web_bind_address injected --host 0.0.0.0 for code-server,
which only sets the bind host, not the port. code-server then listens on its
default port (8080) instead of the tool type's default_port (8443). Cloudflared
connects to port 8443 and gets connection refused, resulting in a 502.
Changes:
- _ensure_web_bind_address now accepts default_port and builds
--bind-addr 0.0.0.0:{port} for code-server
- Same fix for jupyter-notebook with explicit --port flag
- Existing broken --host commands are now detected and replaced
- New migration fixes tool_types templates and instance compose files on disk
- Test fixture updated to use correct --bind-addr 0.0.0.0:8443
- Inject explicit container_name into compose files at start/restart time
via _ensure_container_name_in_compose() to prevent Docker Compose from
generating UUID-based auto names that break backend network resolution.
- Use instance.name.lower() directly instead of get_container_name() lookups
which were unreliable with auto-generated names.
- Apply compose sanitization, bind-address fix, and container-name injection
on restart_instance as well so restarts pick up template fixes.
- Add --force-recreate to docker compose up to ensure container_name changes
take effect immediately.
- Fix notification lifecycle tests to match current behavior (success severity,
health_changed event for ownership test).
Quality gates: ruff clean, pytest (7 notification lifecycle tests passed)
Problem: linuxserver/code-server already binds to 0.0.0.0 by default.
Adding any command: override (--bind-addr or --host) breaks the LSIO
s6 init system with 'not found' errors.
Changes:
- _ensure_web_bind_address(): Skip LSIO images entirely (no command
override needed). If an existing override is found, remove it.
- New migration 2026_05_29_remove_lsio_command_override: Removes
--bind-addr and --host command overrides from both DB templates
and existing instance compose files on disk for LSIO images.
- Fixed migration to use correct column name (compose_path) and
check information_schema for column existence defensively.
Quality gates: ruff clean
Problem: The first migration already ran on the user's server with
--bind-addr (broken). Alembic won't re-run the fixed migration.
Changes:
- _ensure_web_bind_address(): Now detects existing --bind-addr commands
and replaces them with --host 0.0.0.0 instead of skipping
- New migration 2026_05_29_fix_code_server_bind_addr: Finds code-server
tool types with --bind-addr in compose_template and replaces with
--host 0.0.0.0
Quality gates: pytest 42 passed (2 pre-existing unrelated failures)
FastAPI matches routes in declaration order. The DELETE /notifications
endpoint (bulk clear) was registered AFTER DELETE /notifications/{id},
so the path parameter route intercepted all requests to the bulk route,
causing a 422 UUID validation error instead of hitting clear_all.
Moved clear_all_notifications above dismiss_notification in the router.
Added regression test to verify route order.
Quality gates: pytest (22 passed)
Frontend was sending empty string for config_profile_id when no profile
was selected, causing 'not compatible' validation error. Backend now
treats any falsy value (None, empty string) as 'no profile selected'.
Notification filtering:
- lifecycle_hooks.py: only instance.error and instance.health_changed
with status=running generate notifications. All other lifecycle events
(created, started, stopped, restarted, deleted) are filtered out.
- health_monitor.py: only error and unhealthy states generate notifications.
Running/recovered state no longer creates info notifications.
- _derive_title now maps instance.health_changed to "Container ready".
Clear-all button:
- Added dismiss_all() to NotificationService
- Added DELETE /notifications endpoint for bulk dismiss
- Frontend: clearAllNotifications API, clearAll in notification context,
"Clear all" button in notification drawer alongside "Mark all as read"
- Added CSS for .notification-clear-all with danger hover state
- Updated notification-center tests
Quality gates: pytest (21 passed), vitest (11 passed)
The --bind-addr flag caused code-server to fail entirely (app not
responding on any interface). The correct override for the
coder/code-server image is --host 0.0.0.0, which overrides the
entrypoint's --host 127.0.0.1.
Changes:
- Migration: Replace --bind-addr with --host 0.0.0.0, also handle
existing broken templates by detecting --bind-addr and replacing it
- Runtime safety net: _ensure_web_bind_address uses --host 0.0.0.0
- Test fixture: Updated compose template to match
Quality gates: pytest 42 passed
Root cause: code-server (and similar web tools) default to binding to
127.0.0.1 (localhost) inside their containers. This makes them unreachable
from the Docker network and from cloudflared, which connects via the
container's Docker network name.
Changes:
- Migration: Update code-server compose_template to include
--bind-addr 0.0.0.0:8443 command override
- Migration: Update jupyter-notebook compose_template to include
--ip=0.0.0.0 flag
- Runtime safety net: _ensure_web_bind_address() auto-injects bind
address for known web tools (code-server, jupyter-notebook) when
compose doesn't already specify a command
- Diagnostics: _check_app_binding() compares internal vs external
connectivity to detect 127.0.0.1 binding issues
- Improved readiness check: 30s timeout, checks HTTP status codes,
logs curl stderr for debugging
Files:
- apps/api/alembic/versions/2026_05_29_fix_web_tool_bind_address.py
- apps/api/src/services/docker.py
- apps/api/src/api/tool_instances.py
- apps/api/tests/integration/test_tool_types_api_extended.py
Quality gates: pytest 42 passed (5 pre-existing unrelated failures)
The create-session-form was calling startInstance() without passing the
selected config profile and SSH keys. This caused the backend to receive
ssh_key_ids=[] and clear the keys that were stored during createInstance.
The .ssh directory was never mounted because instance.ssh_key_ids was
wiped during the start call.
Also includes minor formatting cleanup on the data migration.
Quality gates: pytest (18 passed)
- Add _check_app_binding() to detect if app is bound to 127.0.0.1
instead of 0.0.0.0 (common cause of tunnel 'app error 0')
- Improve curl readiness check: wait up to 30s, check HTTP status codes
(accept 2xx, 3xx, 401, 403 as 'ready')
- Log curl stderr for connection debugging
- Log binding diagnosis when external connectivity fails
Quality gates: pytest 42 passed
The API container and tool instances share the 'backend' Docker network
(connect_container_to_network at tool_instances.py:1576). cloudflared
runs INSIDE the api container, so localhost:host_port is unreachable.
The original container_name:internal_port is correct for networking.
The 'app error 0' is an application-level issue, not networking.
This reverts commit a8fbca9.
The ssh_keys mount was already removed from the Alembic seed migration, but
that migration had already been applied to the DB. This data migration
removes the mount from the actual tool_definition_manifests row so that
instance-level SSH key mounting handles keys exclusively.
Quality gates: pytest (18 passed)
- apply_mount_permissions now skips mounts with readonly=true to avoid
'Read-only file system' warnings on post-start chown/chmod
- Removed the ssh_keys mount from the pi-agent manifest definition;
instance-level SSH key mounting now handles this exclusively
- Added unit test for read-only mount skipping
Quality gates: pytest (15 passed)
Root cause: start_cloudflared_tunnel was trying to connect to
http://{container_name}:{container_port}, but:
1. The host OS cannot resolve Docker container names
2. cloudflared runs on the host, so it needs the host-mapped port
Changes:
- start_cloudflared_tunnel: changed signature to accept host_port only
- Connects cloudflared to localhost:{host_port} via Docker port mapping
- Connectivity check uses localhost:{host_port}
- recreate_tunnel updated to match new signature
- Callers in tool_instances.py pass instance.port (host port)
Quality gates: pytest 42 passed
- start_instance now deep-merges manifest with base definition before extracting user.uid/user.gid
- The user config is typically defined in the base image (ubuntu-24.04-dev), not the extending manifest
- Add debug logging to verify resolved uid/gid/home_dir
- Add logging to prepare_ssh_key_files for chown success/failure visibility
- Log current process uid when chown fails to diagnose permission issues
Quality gates: pytest 239 passed (6 pre-existing failures), tsc --noEmit clean
- Extend prepare_ssh_key_files() with optional uid/gid parameters
- Call os.chown on created files when uid/gid are provided
- Gracefully handle PermissionError if API process is not root
- In start_instance, extract container user UID/GID from manifest
- Pass container UID/GID when preparing instance-level SSH key mounts
- Legacy clone-mode SSH keys continue to use root (0,0)
- Add unit tests for prepare_ssh_key_files ownership logic
- Keep apply_ssh_permissions() as fallback for cases where host chown fails
Quality gates: pytest 239 passed (6 pre-existing failures), tsc --noEmit clean
- Replace apply_ssh_permissions internals with _exec_and_log for full visibility
- Log every docker exec command, stdout, and stderr at DEBUG level
- After chown/chmod, run ls -la and stat to verify final state
- Log verified state at INFO level so users can see exactly what happened
- Update tests to mock subprocess.run instead of _run_in_container
Quality gates: pytest 236 passed (6 pre-existing), tsc --noEmit clean
- lifecycle_hooks.publish_lifecycle_event now skips notification creation
when event_type='instance.started' and status='starting'
- Users only see notifications for terminal states:
- Failed: instance.error
- Successful: instance.health_changed with status='running'
- Updated integration tests to verify new behavior:
- test_lifecycle_started_intermediate_skips_notification
- test_lifecycle_running_creates_notification
Quality gates: pytest 42 passed, ruff clean
- Add ssh_key_id column to ConfigProfile model and migration
- Update config profile API to accept/return ssh_key_id
- Include ssh_key_id in ResolvedProfile and resolver logic
- Mount selected SSH key into container home dir at start_instance
- Frontend config profile form with SSH key selector dropdown
- Git mount URL validation defaults to profile's SSH key
Quality gates: pytest (231 passed, 6 pre-existing), tsc --noEmit clean
- Add POST /config-profiles/validate-git-url endpoint:
- Parses URL using existing parse_git_url utility
- Suggests corrected URL for browser URLs
- Runs git ls-remote --heads to verify reachability
- Lists available branches from remote
- Supports SSH key for private repos
- Returns structured response: valid, suggested_url, branches,
default_branch, error, error_code
- Update frontend GitMountEditor:
- Add Check button next to URL field with loading state
- Show validation result: valid (green), suggestion (yellow),
invalid (red)
- Suggestion includes Use this button to apply corrected URL
- Branch field becomes dropdown when URL is validated,
populated with remote branches
- Mappings section disabled until URL is validated
- Shows hint: Validate the URL first
- Quality gates: pytest (218 passed, 6 pre-existing),
tsc --noEmit (clean)
- Add notifications table with Alembic migration
- Notification model with user-scoped indexing and partial index on unread
- NotificationService singleton with create/list/count/mark-read/dismiss
- FastAPI router: GET /notifications, GET /unread, PATCH /{id}/read,
POST /mark-all-read, DELETE /{id}
- Mute categories filtering from UserConfig
- 13 unit tests for NotificationService
- 10 integration tests for API endpoints
- Updated test_models.py with new table registration
Quality gates: pytest 23 new passed, ruff clean
The previous sorting fix exposed a deeper bug: ResolvedMount always
mounted its staging directory as a single bind mount. When a config
profile mount targeted /workspace/x/y and contained a single file
z.json, the staging directory (containing only z.json) replaced the
ENTIRE /workspace/x/y directory, hiding all sibling files from git
repo mounts.
- Change apply_resolved_profile to mount each file individually:
- source: staging_dir/relative_path
- target: expanded_target/relative_path
- Sibling files from other mounts are preserved.
- Empty mounts produce no volume entries.
- Keep volume sorting (parent paths before child paths) which is
still necessary for directory mounts and ensures parent dirs exist
before file mounts inside them.
- Add 4 unit tests for file-level mount behavior.
Quality gates: pytest (218 passed, 6 pre-existing), tsc --noEmit (clean)
- Add instance_events and health_checks tables with Alembic migration
- InstanceEventBus: typed pub/sub singleton with wildcard support
- HealthMonitor: async background loop polling containers every 15s
- SSE endpoint GET /events/stream with auth and connection limits
- Lifecycle hooks in tool_instances.py (create/start/stop/restart/delete)
- Structured JSON logging with correlation IDs
- 15 new unit tests (EventBus, HealthMonitor, MonitoringModels)
Quality gates: pytest 15 new passed, ruff clean
alex
Developer