headquarter/openspec/changes/cloudflare-tunnel-instances/design.md

Context

Currently, tool instances are exposed via an API proxy endpoint that forwards requests from /instances/{id}/proxy/ to the internal Docker container. This creates latency, adds load to the API service, and doesn't support WebSocket features well. Cloudflare Tunnel offers a better architecture where each instance gets its own HTTPS subdomain.

Goals / Non-Goals

Goals:

• Each running tool instance gets a unique public HTTPS subdomain
• No manual DNS or reverse proxy configuration per instance
• Automatic cleanup when instances are stopped or deleted
• Support for WebSocket and real-time features (code-server terminal, jupyter kernels)
• Minimal latency compared to API proxy approach

Non-Goals:

• Custom domains per instance (use Cloudflare zone's wildcard)
• Advanced tunnel features (load balancing, failover, ingress rules)
• Replacing Traefik for the main app (API + frontend)
• Supporting non-HTTP protocols (TCP/UDP raw tunneling)

Decisions

Cloudflare API vs cloudflared CLI

Decision: Use the Cloudflare REST API to create/manage tunnels, not the cloudflared CLI.

Rationale:

• The API gives us programmatic control without parsing CLI output
• We can use httpx (already a dependency) instead of subprocess calls
• Easier to test and mock

Alternative considered: Running cloudflared tunnel create via subprocess

• Rejected: Fragile, harder to test, requires cloudflared binary in API container

Architecture: cloudflared as a separate container

Decision: Run cloudflared as a standalone Docker service that connects to Cloudflare and routes traffic.

Rationale:

• Separation of concerns: API manages tunnels, cloudflared handles connectivity
• The cloudflared container can access the Docker internal network where instances run
• Easier to scale/restart independently

┌─────────────────────────────────────────────────────────────┐
│                      Cloudflare Edge                        │
└──────────────────────┬──────────────────────────────────────┘
                       │ HTTPS
┌──────────────────────▼──────────────────────────────────────┐
│                   cloudflared container                     │
│  (connects to Cloudflare, receives traffic for *.zone)     │
└──────────┬──────────────────────────────────────────────────┘
           │ Docker network
┌──────────▼──────────────────────────────────────────────────┐
│  code-server container:8443  jupyter container:8888        │
│  (tool instances on Docker network with DNS names)          │
└─────────────────────────────────────────────────────────────┘

Subdomain naming

Decision: Use instance-{short-uuid}.{zone} format (e.g., instance-a1b2c3d4.headquarter.commumedia.org)

Rationale:

• Predictable and URL-safe
• Short enough to be readable
• UUID ensures uniqueness without exposing internal IDs

Tunnel lifecycle

Decision: Create tunnel on instance start, delete on instance stop/delete.

Flow:

1. User clicks "Start"
2. Backend creates Cloudflare tunnel via API
3. Backend creates DNS CNAME record: instance-abc123 → {tunnel-id}.cfargotunnel.com
4. Backend stores tunnel_id and public_url in ToolInstance
5. cloudflared container routes traffic to container:port
6. On stop: delete DNS record, delete tunnel

cloudflared configuration

Decision: Use a single cloudflared container with dynamic config file updates.

Approach:

• The cloudflared container reads an config.yml file mounted as a volume
• The API writes ingress rules to this file when instances start/stop
• cloudflared automatically reloads the config (or we restart the container)

# /etc/cloudflared/config.yml
tunnel: {tunnel-token}
credentials-file: /etc/cloudflared/credentials.json
ingress:
  - hostname: instance-abc123.headquarter.commumedia.org
    service: http://code-server-repo-abc123:8443
  - hostname: instance-xyz789.headquarter.commumedia.org
    service: http://jupyter-repo-def:8888
  - service: http_status:404

Authentication

Decision: Cloudflare tunnels provide HTTPS but do NOT handle app-level auth. Tool instances without built-in auth (like code-server) will be publicly accessible.

Rationale:

• Cloudflare Access could add auth, but adds complexity
• Many tools (code-server) have their own password/auth mechanisms
• Users should configure tool-level auth via ToolConfig

Mitigation: Document that users must configure tool passwords via ToolConfig (e.g., PASSWORD env for code-server).

Risks / Trade-offs

[Risk] Cloudflare API rate limits (1200 requests/5 min) → Mitigation: Tunnel creation is infrequent (user-initiated), unlikely to hit limits

[Risk] cloudflared container becomes a single point of failure → Mitigation: It's stateless; can be restarted quickly. All instances share one cloudflared.

[Risk] Subdomain enumeration exposes running instances → Mitigation: UUID-based names are hard to guess. Consider adding Cloudflare Access in future.

[Risk] cloudflared config file updates require container restart → Mitigation: Investigate cloudflared --no-autoupdate with config watch, or accept brief restart

[Risk] Tool instances publicly accessible without auth → Mitigation: Document security best practices, recommend setting tool passwords

Migration Plan

1. Deploy cloudflared container with base config
2. Add Cloudflare env vars to API container
3. Deploy backend changes (tunnel service, updated lifecycle)
4. Deploy frontend changes (use public_url instead of proxy)
5. Test with code-server instance
6. Remove old proxy endpoint code

Open Questions

• Should we add Cloudflare Access (Zero Trust) to protect instances?
• Do we need to support custom subdomains (e.g., myproject.headquarter.commumedia.org)?
• Should we keep the proxy endpoint as a fallback?