headquarter/docs/superpowers/plans/2024-05-16-docker-infrastructure.md

Docker Infrastructure Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Set up complete Docker-based development environment with PostgreSQL, Redis, Traefik, Authentik, FastAPI backend, and React frontend.

Architecture: Multi-service Docker Compose setup with Traefik as reverse proxy, PostgreSQL for data, Redis for caching, Authentik for auth, FastAPI backend, and Vite React frontend. All services include health checks and persistent volumes.

Tech Stack: Docker 24.0+, Docker Compose 2.20+, Make, Python 3.11+, Node.js 20+

---

File Structure

/
├── docker-compose.yml          # All services orchestration
├── .env.example                # Required environment variables
├── Makefile                    # Common commands
├── apps/
│   ├── api/
│   │   ├── Dockerfile          # Multi-stage Python build
│   │   └── pyproject.toml      # Python dependencies (placeholder)
│   └── web/
│       ├── Dockerfile          # Multi-stage Node build
│       └── package.json        # Node dependencies (placeholder)
└── data/
    └── repos/                  # Git repository storage volume

---

Task 1: Create Project Directory Structure

Files:

• Create: apps/api/

• Create: apps/web/

• Create: data/repos/

• Step 1: Create directory structure

mkdir -p apps/api apps/web data/repos

• Step 2: Create placeholder files for Docker build context

Create apps/api/pyproject.toml:

[project]
name = "headquarter-api"
version = "0.1.0"
description = "Headquarter platform API"
requires-python = ">=3.11"
dependencies = [
    "fastapi>=0.104.0",
    "uvicorn[standard]>=0.24.0",
    "sqlalchemy>=2.0.0",
    "asyncpg>=0.29.0",
    "alembic>=1.12.0",
    "pydantic>=2.5.0",
    "pydantic-settings>=2.1.0",
    "python-jose[cryptography]>=3.3.0",
    "python-multipart>=0.0.6",
    "httpx>=0.25.0",
    "structlog>=23.2.0",
    "cryptography>=41.0.0",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.4.0",
    "pytest-asyncio>=0.21.0",
    "mypy>=1.7.0",
    "ruff>=0.1.0",
    "httpx>=0.25.0",
]

Create apps/web/package.json:

{
  "name": "headquarter-web",
  "version": "0.1.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc && vite build",
    "preview": "vite preview",
    "typecheck": "tsc --noEmit",
    "lint": "eslint . --ext ts,tsx --report-unused-disable-directives --max-warnings 0"
  },
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-router-dom": "^6.20.0",
    "axios": "^1.6.0",
    "tailwindcss": "^3.3.0"
  },
  "devDependencies": {
    "@types/react": "^18.2.0",
    "@types/react-dom": "^18.2.0",
    "@vitejs/plugin-react": "^4.2.0",
    "typescript": "^5.3.0",
    "vite": "^5.0.0",
    "eslint": "^8.55.0",
    "@typescript-eslint/eslint-plugin": "^6.14.0",
    "@typescript-eslint/parser": "^6.14.0",
    "autoprefixer": "^10.4.16",
    "postcss": "^8.4.32"
  }
}

• Step 3: Commit

git add apps/ data/
git commit -m "chore: create project directory structure"

---

Task 2: Create API Dockerfile

Files:

• Create: apps/api/Dockerfile

• Step 1: Write multi-stage API Dockerfile

# Build stage
FROM python:3.11-slim as builder

WORKDIR /build

# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Install Python dependencies
COPY pyproject.toml .
RUN pip install --no-cache-dir --user -e ".[dev]"

# Production stage
FROM python:3.11-slim

# Create non-root user
RUN groupadd -r appgroup && useradd -r -g appgroup appuser

WORKDIR /app

# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq5 \
    git \
    && rm -rf /var/lib/apt/lists/*

# Copy dependencies from builder
COPY --from=builder /root/.local /home/appuser/.local
ENV PATH=/home/appuser/.local/bin:$PATH

# Copy application code
COPY --chown=appuser:appgroup . .

# Create directories for repo storage
RUN mkdir -p /data/repos && chown -R appuser:appgroup /data/repos

# Switch to non-root user
USER appuser

# Expose port
EXPOSE 8000

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1

# Run the application
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

• Step 2: Commit

git add apps/api/Dockerfile
git commit -m "feat: add multi-stage API Dockerfile"

---

Task 3: Create Web Frontend Dockerfile

Files:

• Create: apps/web/Dockerfile

• Create: apps/web/nginx.conf

• Step 1: Write multi-stage Web Dockerfile

# Build stage
FROM node:20-alpine as builder

WORKDIR /build

# Copy package files
COPY package.json package-lock.json* ./

# Install dependencies
RUN npm ci

# Copy source code
COPY . .

# Build production bundle
RUN npm run build

# Production stage
FROM nginx:alpine

# Create non-root user
RUN addgroup -g 1001 -S nodejs && adduser -S nextjs -u 1001

# Copy custom nginx config
COPY nginx.conf /etc/nginx/conf.d/default.conf

# Copy built assets from builder
COPY --from=builder --chown=nextjs:nodejs /build/dist /usr/share/nginx/html

# Create required directories
RUN mkdir -p /var/cache/nginx /var/run && \
    chown -R nextjs:nodejs /var/cache/nginx /var/run /usr/share/nginx/html

# Switch to non-root user
USER nextjs

# Expose port
EXPOSE 80

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD wget --quiet --tries=1 --spider http://localhost/ || exit 1

# Start nginx
CMD ["nginx", "-g", "daemon off;"]

• Step 2: Write nginx configuration

server {
    listen 80;
    server_name localhost;
    root /usr/share/nginx/html;
    index index.html;

    # Enable gzip compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml;

    # Handle client-side routing
    location / {
        try_files $uri $uri/ /index.html;
    }

    # Cache static assets
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }

    # Health check endpoint
    location /health {
        access_log off;
        return 200 "healthy\n";
        add_header Content-Type text/plain;
    }
}

• Step 3: Commit

git add apps/web/Dockerfile apps/web/nginx.conf
git commit -m "feat: add multi-stage web frontend Dockerfile with nginx"

---

Task 4: Create Docker Compose Configuration

Files:

• Create: docker-compose.yml

• Step 1: Write Docker Compose file

version: '3.8'

services:
  # PostgreSQL Database
  postgres:
    image: postgres:15-alpine
    container_name: hq-postgres
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-headquarter}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-headquarter}
      POSTGRES_DB: ${POSTGRES_DB:-headquarter}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d:ro
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-headquarter} -d ${POSTGRES_DB:-headquarter}"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 10s
    networks:
      - backend
    restart: unless-stopped

  # Redis Cache
  redis:
    image: redis:7-alpine
    container_name: hq-redis
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 5s
    networks:
      - backend
    restart: unless-stopped

  # Traefik Reverse Proxy
  traefik:
    image: traefik:v3.0
    container_name: hq-traefik
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--ping=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"  # Dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik:/etc/traefik:ro
    healthcheck:
      test: ["CMD", "traefik", "healthcheck"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 5s
    networks:
      - frontend
      - backend
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.hq.local`)"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.entrypoints=web"

  # Authentik - Authentication Server
  authentik-server:
    image: ghcr.io/goauthentik/server:2024.2
    container_name: hq-authentik
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY:-change-me-in-production}
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgres
      AUTHENTIK_POSTGRESQL__NAME: ${POSTGRES_DB:-headquarter}
      AUTHENTIK_POSTGRESQL__USER: ${POSTGRES_USER:-headquarter}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${POSTGRES_PASSWORD:-headquarter}
    volumes:
      - ./authentik/media:/media
      - ./authentik/custom-templates:/templates
    ports:
      - "9000:9000"
      - "9443:9443"
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - backend
      - frontend
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.authentik.rule=Host(`auth.hq.local`)"
      - "traefik.http.routers.authentik.entrypoints=web"
      - "traefik.http.services.authentik.loadbalancer.server.port=9000"

  # API Service
  api:
    build:
      context: ./apps/api
      dockerfile: Dockerfile
    container_name: hq-api
    environment:
      DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-headquarter}:${POSTGRES_PASSWORD:-headquarter}@postgres:5432/${POSTGRES_DB:-headquarter}
      REDIS_URL: redis://redis:6379/0
      AUTHENTIK_URL: http://authentik-server:9000
      AUTHENTIK_CLIENT_ID: ${AUTHENTIK_CLIENT_ID}
      AUTHENTIK_CLIENT_SECRET: ${AUTHENTIK_CLIENT_SECRET}
      JWT_SECRET: ${JWT_SECRET:-change-me-in-production}
      REPO_BASE_PATH: /data/repos
    volumes:
      - repo_data:/data/repos
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
      authentik-server:
        condition: service_started
    networks:
      - backend
      - frontend
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`api.hq.local`)"
      - "traefik.http.routers.api.entrypoints=web"
      - "traefik.http.services.api.loadbalancer.server.port=8000"

  # Web Frontend
  web:
    build:
      context: ./apps/web
      dockerfile: Dockerfile
    container_name: hq-web
    environment:
      VITE_API_URL: http://api.hq.local
      VITE_AUTH_URL: http://auth.hq.local
    depends_on:
      - api
    networks:
      - frontend
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.web.rule=Host(`hq.local`) || Host(`www.hq.local`)"
      - "traefik.http.routers.web.entrypoints=web"
      - "traefik.http.services.web.loadbalancer.server.port=80"

volumes:
  postgres_data:
  redis_data:
  repo_data:

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge

• Step 2: Commit

git add docker-compose.yml
git commit -m "feat: add Docker Compose with all services"

---

Task 5: Create Environment Configuration Template

Files:

• Create: .env.example

• Step 1: Write environment template

# Database Configuration
POSTGRES_USER=headquarter
POSTGRES_PASSWORD=change-me-in-production
POSTGRES_DB=headquarter

# Redis Configuration
REDIS_URL=redis://redis:6379/0

# Authentik Configuration
AUTHENTIK_SECRET_KEY=change-me-in-production
AUTHENTIK_CLIENT_ID=your-authentik-client-id
AUTHENTIK_CLIENT_SECRET=your-authentik-client-secret
AUTHENTIK_URL=http://auth.hq.local

# JWT Configuration
JWT_SECRET=change-me-in-production
JWT_ALGORITHM=HS256
JWT_EXPIRATION_HOURS=24

# Application Configuration
APP_ENV=development
DEBUG=true
LOG_LEVEL=info
REPO_BASE_PATH=/data/repos

# Frontend Configuration
VITE_API_URL=http://api.hq.local
VITE_AUTH_URL=http://auth.hq.local

# Docker Configuration
COMPOSE_PROJECT_NAME=headquarter
DOCKER_NETWORK=headquarter_default

• Step 2: Commit

git add .env.example
git commit -m "docs: add environment configuration template"

---

Task 6: Create Makefile

Files:

• Create: Makefile

• Step 1: Write Makefile

.PHONY: help up down logs migrate test lint clean build

# Default target
help:
	@echo "Headquarter Development Commands"
	@echo "================================"
	@echo "make up       - Start all services"
	@echo "make down     - Stop all services"
	@echo "make logs     - View service logs"
	@echo "make migrate  - Run database migrations"
	@echo "make test     - Run test suites"
	@echo "make lint     - Run linting"
	@echo "make build    - Build all Docker images"
	@echo "make clean    - Remove containers and volumes"
	@echo "make shell    - Open shell in API container"

# Start services
up:
	docker compose up -d
	@echo "Services starting..."
	@echo "API: http://api.hq.local"
	@echo "Web: http://hq.local"
	@echo "Auth: http://auth.hq.local"
	@echo "Traefik: http://traefik.hq.local:8080"

# Stop services
down:
	docker compose down

# View logs
logs:
	docker compose logs -f

# View specific service logs
logs-api:
	docker compose logs -f api

logs-web:
	docker compose logs -f web

logs-db:
	docker compose logs -f postgres

# Run database migrations
migrate:
	docker compose exec api alembic upgrade head

# Create new migration
migration:
	docker compose exec api alembic revision --autogenerate -m "$(message)"

# Run tests
test:
	docker compose exec api pytest -v

# Run linting
lint:
	docker compose exec api ruff check .
	docker compose exec api mypy .
	cd apps/web && npm run lint

# Type checking
typecheck:
	docker compose exec api mypy .
	cd apps/web && npm run typecheck

# Build all images
build:
	docker compose build

# Build specific service
build-api:
	docker compose build api

build-web:
	docker compose build web

# Clean up
clean:
	docker compose down -v --remove-orphans
	docker system prune -f

# Open shell in API container
shell:
	docker compose exec api /bin/sh

# Database shell
db-shell:
	docker compose exec postgres psql -U $(POSTGRES_USER) -d $(POSTGRES_DB)

# Health check
health:
	@echo "Checking service health..."
	@docker compose ps
	@docker compose exec api wget -qO- http://localhost:8000/health || echo "API health check failed"

• Step 2: Commit

git add Makefile
git commit -m "feat: add Makefile with common development commands"

---

Task 7: Validate Docker Configuration

Files:

• Test: docker-compose.yml

• Step 1: Validate Docker Compose syntax

docker compose config

Expected: Valid YAML output with all services configured

• Step 2: Test build

make build

Expected: Both API and web images build successfully (may warn about missing source files - that's OK)

• Step 3: Test start/stop

make up
sleep 10
make down

Expected: Services start (Postgres and Redis should be healthy), then stop cleanly

• Step 4: Commit

git add -A
git commit -m "test: validate Docker infrastructure"

---

Verification

Run these checks to verify everything works:

1. Syntax validation:

   docker compose config > /dev/null && echo "Valid"
   

2. Health checks:

   make up
   docker compose ps
   

   All services should show "healthy" or "running"

3. Makefile commands:

   make help      # Shows usage
   make build     # Builds images
   make up        # Starts services
   make logs      # Shows logs
   make down      # Stops services
   

Quality Gates

• docker compose config validates without errors
• All services have health checks defined
• API Dockerfile uses multi-stage build with non-root user
• Web Dockerfile uses multi-stage build with non-root user
• Makefile includes all required commands (up, down, logs, migrate, test, lint)
• .env.example documents all required variables
• Persistent volume for /data/repos