# Design Doc: Hierarchical Project Analysis Skill for Pi ## 1. Goals and Success Criteria ### Primary Goal Enable a Pi coding agent to understand a software project's architecture and code relationships without scanning the entire repository. The agent should have a compact, hierarchical "internal representation" of the project that it can consume in-context. ### Success Criteria - The agent can orient itself in a new or familiar project without reading dozens of source files. - The agent understands cross-package dependencies, data flows, and architectural patterns from the analysis files alone. - Analysis files stay sufficiently fresh that the agent does not make decisions based on stale information. - The representation is token-dense: maximum information per token, optimized for LLM consumption, not human readability. ## 2. Format Specification: Dense Markdown with Conventions ### Design Rationale - **Not JSON/YAML**: Brackets, quotes, and indentation add token overhead with no benefit to LLM comprehension. - **Not a custom DSL**: Fragile, requires a parser, and LLMs may hallucinate syntax. - **Dense markdown**: Hierarchical headings, bullet points, and abbreviations are natively understood by LLMs and extremely token-efficient. ### Structure Each directory in the project gets one analysis file named `.pi-map.md` (hidden by default, excluded from git via `.gitignore`). ```markdown # ## role | Dep: ## files - | | exp: | dep: - | | exp: | dep: ## arch ## dirty ``` ### Abbreviation Conventions | Abbreviation | Meaning | |-------------|---------| | `exp:` | exported symbols (functions, classes, types, constants) | | `dep:` | dependencies (other packages, files, or external libs) | | `pkg/` | project-internal package reference | | `ext/` | external dependency reference | | `->` | data flow direction | | `|` | field delimiter within a line | ### Example ```markdown # pkg/auth ## role Auth layer: JWT issuance, validation, refresh. Stateless. Dep: pkg/crypto, pkg/db. ## files - tokens.ts | JWT gen/val | exp: issueToken, verifyToken, refreshToken | dep: crypto/hmac, db/sessions - middleware.ts | HTTP auth guard | exp: requireAuth, requireRole | dep: tokens/verifyToken - types.ts | shared auth types | exp: AuthToken, UserClaims, Role ## arch Guard pattern on routes. Tokens short-lived (15m), refresh long-lived (7d). Rotation on every use. Session state stored in Redis via db/sessions. No server-side JWT storage. ## dirty - ``` ### Rules - One file per directory, placed inside that directory. - Every non-excluded file in the directory gets one bullet under `## files`. - Subdirectories are referenced in `## role` via `Dep:` or in `## arch` as structural notes, not duplicated. - The `## dirty` section is empty (`-`) when clean, or contains a timestamp/flag when stale. ## 3. Pipeline Architecture ### Hybrid Extraction: LLM + AST Two independent extraction layers contribute to the same output file. #### Layer 1: LLM-Based Extraction (All Files) - **Input**: Raw file contents of every non-excluded file in the directory. - **Output**: Purpose description, architectural role, and cross-file relationships. - **Applies to**: Code files, config files, Dockerfiles, READMEs, YAML, JSON, shell scripts — everything. - **When it runs**: Once per file during init; again on changed files during patching. - **Implementation**: Calls an actual LLM (not regex heuristics). Inside Pi, it uses Pi's built-in LLM via the ExtensionAPI. Standalone CLI falls back to an external LLM API (OpenAI-compatible). #### Layer 2: AST-Based Extraction (Code Files Only) - **Input**: Source code of files where a tree-sitter or LSP parser is available. - **Output**: Precise symbol lists (functions, classes, types), signatures, import/export graphs, class hierarchies. - **Applies to**: Supported languages only (TypeScript, Python, Go, Rust, etc.). - **When it runs**: Once per file during init; again on changed files during patching. #### Merging The two layers merge into a single line per file under `## files`: ``` - tokens.ts | JWT gen/val | exp: issueToken, verifyToken, refreshToken | dep: crypto/hmac, db/sessions ^ LLM ^ LLM ^ AST ^ AST + LLM ``` - File name and purpose: LLM. - Exported symbols and signatures: AST (augmented by LLM if AST unavailable). - Dependency list: AST for imports; LLM for inferred architectural dependencies. ### LLM Client Architecture The LLM client is abstracted behind a unified interface: ```typescript interface LLMClient { complete(prompt: string): Promise; } ``` Two implementations: 1. **PiLLMClient** (Pi extension): Uses `ctx.model` or `ctx.modelRegistry` to invoke Pi's configured LLM. Called from `pi-extension.ts` when the skill runs inside Pi. 2. **ExternalLLMClient** (standalone CLI): Calls an external OpenAI-compatible API. Configured via environment variable (e.g., `OPENAI_API_KEY`) or config file. ### Caching LLM results are cached to avoid re-querying unchanged files. - **Key**: SHA-256 hash of file contents. - **Storage**: JSON file at `~/.cache/pi-project-map/llm-cache.json`. - **Behavior**: Before calling the LLM, compute the file hash and check the cache. If hit, reuse the cached result. If miss, call the LLM and store the result. - **Invalidation**: Cache entries are implicitly invalidated when the file content changes (because the hash changes). There is no TTL; the cache is append-only. ### Parallelization and Rate Limiting - **Concurrency**: 4-8 LLM calls in parallel, controlled by `p-limit`. - **Batch delays**: A small delay (e.g., 100ms) is inserted between batches to avoid triggering rate limits. - **Retry policy**: Each LLM call retries up to 3 times with exponential backoff (1s, 2s, 4s). If all retries fail, the entire operation stops with a hard error. ### Error Handling - **Hard error on failure**: If an LLM call fails after all retries, `init` or `patch` stops immediately and prints a clear error. There is no heuristic fallback. The user must resolve the issue (set API key, wait for rate limit, check network). - **Context limit protection**: Files larger than the LLM's context window are truncated from the end (with a note in the prompt) before being sent. ### Init Pipeline ``` For each directory (depth-first): 1. List all non-excluded files. 2. For each file (parallel, 4-8 concurrent): a. Compute SHA-256 of file contents. b. Check disk cache. If hit, use cached result. c. If miss: call LLM (with retries/backoff) to extract purpose and role. d. Store result in cache. e. If code file + parser available: run AST extraction (symbols, imports). 3. Merge per-file outputs into lines. 4. Run LLM on merged lines + directory context to generate: - `## role` (package-level summary) - `## arch` (architectural notes) 5. Write `.pi-map.md` to directory. ``` ### Patch Pipeline ``` When agent edits file(s) in directory: 1. Determine patch strategy: - If directory has < 10 files: full rewrite. - Else: section-level patch for changed file(s) only. 2. For each changed file: a. Recompute SHA-256. b. Check cache. If miss or stale, call LLM with retries/backoff. 3. Re-run AST extraction on changed file(s) if applicable. 4. Update `## files` section (rewrite or patch). 5. Update `## dirty` flag if full regeneration is deferred. ``` ## 4. LLM Prompt Design ### File-Level Prompt The LLM prompt for a single file is designed to produce a structured, concise analysis. ``` You are analyzing a source file for a project map. Read the file below and summarize: 1. PURPOSE: What does this file do? Describe its role in the project (2-3 sentences max). 2. DEPENDENCIES: What does this file depend on? List internal modules/packages and external libraries. 3. KEY CONCEPTS: Mention any important patterns, algorithms, or domain concepts. File path: ``` ``` Respond in this exact format: PURPOSE: DEPS: CONCEPTS: ``` ### Package-Level Prompt After all file summaries are collected for a directory, a second LLM call synthesizes the package role and architecture. ``` You are analyzing a directory in a software project. Below is a list of files in this directory with their purposes. Directory: Files: - : - : ... Respond in this exact format: ROLE: ARCH: <2-4 sentences describing architecture, data flow, patterns, and design decisions> ``` ### Output Parsing The LLM client's response is parsed to extract `PURPOSE`, `DEPS`, `CONCEPTS`, `ROLE`, and `ARCH` fields. These are merged with AST data into the final `.pi-map.md` format. ### Context Limit Protection - Files are truncated from the end if they exceed a configurable max token budget (default: 4000 tokens of source). - A marker `[...truncated]` is appended to the truncated content so the LLM knows it is not seeing the full file. - Very large binary or generated files are skipped entirely for LLM analysis (they still appear in `.pi-map.md` with a note like "Large/generated file"). ## 5. Consumption Model ### Session Start 1. Agent discovers all `.pi-map.md` files (e.g., via `find . -name ".pi-map.md"`). 2. Agent reads **all** files into context. This is a one-time cost at session start. 3. Agent constructs an internal mental model of the project hierarchy. ### During Session - An **auto-injected summary** stays in context (e.g., a condensed top-level `.pi-map.md` or a synthesized project overview). - When the agent needs deeper detail about a specific package, it already has the full `.pi-map.md` in memory from step 2. - If the agent enters a new package not yet loaded, it reads that package's `.pi-map.md` on demand. ### Context Management - For very large projects, the agent may summarize or prune the initial read, keeping only the top N levels of the hierarchy in active context. - The skill can provide a "context budget" parameter: max tokens to spend on analysis files. ## 6. Stale Data Mitigation ### Combined Strategy #### 5.1 Dirty Markers - Whenever the agent edits a file, it appends a dirty flag to the directory's `.pi-map.md`: ```markdown ## dirty 2024-06-09T14:32:00Z: tokens.ts modified ``` - A background or post-session reconciliation step regenerates dirty files. - The agent can also be instructed to reconcile before making architectural decisions. #### 5.2 Periodic Full Re-init - On every new session start, or on a configurable schedule (e.g., daily), the skill offers to run a full re-scan. - This catches any changes made outside the agent's awareness (e.g., by other developers). #### 5.3 Validation Command - A `validate` tool/command that the agent can invoke: - Checks for missing files (new files not in `.pi-map.md`). - Checks for orphaned entries (files listed but deleted). - Checks for changed signatures (AST mismatch between listed symbols and actual code). - Reports discrepancies and suggests corrections. ### Recovery - If validation finds staleness beyond a threshold (e.g., > 3 dirty packages), the skill recommends a full re-init. - The agent can also trigger re-init for a specific subtree. ## 7. Scope Boundaries and Non-Goals ### In Scope - Every directory in the project gets a `.pi-map.md` file. - Every non-excluded file gets analyzed by the LLM layer. - Code files get augmented by the AST layer where parsers exist. - Respect `.gitignore` and known junk patterns (node_modules, .git, dist, build, coverage, .next, .venv, __pycache__, .DS_Store). ### Out of Scope (Non-Goals) - **Human-readable documentation**: These files are machine-only. Human docs live elsewhere. - **Line-by-line code explanation**: The format captures symbols and architecture, not implementation details. - **Auto-regeneration on filesystem events**: The skill relies on agent-initiated updates and periodic re-init, not filesystem watchers. - **Cross-project analysis**: Each project is independent. No global index across repos. - **IDE integration**: This is a Pi agent skill, not a VS Code extension or LSP server. ## 8. Pi Skill Package Structure ``` pi-project-map/ ├── SKILL.md # Skill definition for Pi ├── package.json # npm package metadata ├── src/ │ ├── init.ts # Full project scan + generation │ ├── patch.ts # Incremental patch logic │ ├── validate.ts # Consistency checker │ ├── ast-extract.ts # Tree-sitter / LSP wrappers │ ├── llm-extract.ts # LLM prompt templates for extraction │ ├── merge.ts # Merge AST + LLM outputs │ ├── format.ts # Dense markdown formatter │ └── config.ts # Skill configuration (thresholds, ignore patterns) ├── hooks/ │ └── on-prompt.ts # Injects maintenance command into prompts └── README.md # Setup and usage for humans ``` ### Custom Tools - `project-map:init` — Run full project scan. Creates all `.pi-map.md` files. - `project-map:patch ` — Update analysis for a specific file/directory. - `project-map:validate` — Run consistency check across all `.pi-map.md` files. - `project-map:reinit [path]` — Force re-initialization of entire project or subtree. ### Prompt Hook - On every prompt, the skill appends a lightweight instruction: > "If you modify any source file, run `project-map:patch ` to update the analysis. If you suspect staleness, run `project-map:validate`." ## 9. Risks and Tradeoffs | Risk | Likelihood | Impact | Mitigation | |------|-----------|--------|------------| | Token bloat (1000+ dirs) | Medium | High | Summary mode, lazy loading, context budget | | Stale analysis files | High | High | Dirty markers + periodic re-init + validation | | Agent trusts stale data | Medium | High | Clear instructions to validate before architectural decisions | | Expensive init on large repos | Medium | Medium | Parallelization, caching, optional incremental init | | Overlap with LSP/typedoc | Low | Low | This is agent-context, not IDE tooling. Different use case. | | AST parser unavailable | Medium | Low | Graceful fallback to LLM-only extraction | ## 10. Concrete Example: Full Project Snapshot ``` project-root/ ├── .pi-map.md ├── src/ │ ├── .pi-map.md │ ├── auth/ │ │ ├── .pi-map.md │ │ ├── tokens.ts │ │ ├── middleware.ts │ │ └── types.ts │ └── db/ │ ├── .pi-map.md │ ├── connection.ts │ └── migrations/ │ ├── .pi-map.md │ └── 001_init.sql ├── docker/ │ ├── .pi-map.md │ ├── Dockerfile │ └── docker-compose.yml └── README.md ``` Each `.pi-map.md` follows the format in Section 2, creating a navigable hierarchy. ## 11. Future Extensions - **Cross-reference graph**: A top-level `project-graph.md` linking all packages with dependency arrows. - **Search index**: A lightweight FTS5 index over all `.pi-map.md` files for fast symbol lookup. - **Diff-aware patching**: Only re-run LLM on changed functions, not entire files. - **Multi-repo workspaces**: Support monorepos with independent package boundaries.