442 lines
20 KiB
Markdown
442 lines
20 KiB
Markdown
# Design Doc: Hierarchical Project Analysis Skill for Pi
|
|
|
|
## 1. Goals and Success Criteria
|
|
|
|
### Primary Goal
|
|
Enable a Pi coding agent to understand a software project's architecture and code relationships without scanning the entire repository. The agent should have a compact, hierarchical "internal representation" of the project that it can consume in-context.
|
|
|
|
### Success Criteria
|
|
- The agent can orient itself in a new or familiar project without reading dozens of source files.
|
|
- The agent understands cross-package dependencies, data flows, and architectural patterns from the analysis files alone.
|
|
- Analysis files stay sufficiently fresh that the agent does not make decisions based on stale information.
|
|
- The representation is token-dense: maximum information per token, optimized for LLM consumption, not human readability.
|
|
|
|
## 2. Format Specification: Dense Markdown with Conventions
|
|
|
|
### Design Rationale
|
|
- **Not JSON/YAML**: Brackets, quotes, and indentation add token overhead with no benefit to LLM comprehension.
|
|
- **Not a custom DSL**: Fragile, requires a parser, and LLMs may hallucinate syntax.
|
|
- **Dense markdown**: Hierarchical headings, bullet points, and abbreviations are natively understood by LLMs and extremely token-efficient.
|
|
|
|
### Structure
|
|
Each non-ignored directory in the project gets **two** hidden analysis files:
|
|
|
|
- `.pi-map.index.md` — routing-first index
|
|
- `.pi-map.md` — orientation-first rich map
|
|
|
|
```markdown
|
|
# <relative-path> (index)
|
|
dir: <relative-path>
|
|
## role
|
|
<short routing summary>
|
|
## parent
|
|
<parent links or ->
|
|
## children
|
|
<child links or ->
|
|
## files
|
|
<likely files>
|
|
## links
|
|
index: <self-index>
|
|
map: <self-map>
|
|
## workflows
|
|
<compact task -> route hints>
|
|
## dirty
|
|
<timestamp or ->
|
|
```
|
|
|
|
```markdown
|
|
# <relative-path>
|
|
dir: <relative-path>
|
|
index: <sibling-index>
|
|
## role
|
|
<one-line package role>
|
|
## files
|
|
- <filename> | <one-line purpose> | exp: <exported symbols> | dep: <internal/external deps>
|
|
## arch
|
|
<free-form architectural notes>
|
|
## tags
|
|
<compact tags>
|
|
## symbols
|
|
<prioritized symbols>
|
|
## workflows
|
|
<compact workflow hints>
|
|
## dirty
|
|
<timestamp or ->
|
|
```
|
|
|
|
### Abbreviation Conventions
|
|
| Abbreviation | Meaning |
|
|
|-------------|---------|
|
|
| `exp:` | exported symbols (functions, classes, types, constants) |
|
|
| `dep:` | dependencies (other packages, files, or external libs) |
|
|
| `pkg/` | project-internal package reference |
|
|
| `ext/` | external dependency reference |
|
|
| `->` | data flow direction |
|
|
| `|` | field delimiter within a line |
|
|
|
|
### Example
|
|
|
|
```markdown
|
|
# pkg/auth
|
|
## role
|
|
Auth layer: JWT issuance, validation, refresh. Stateless. Dep: pkg/crypto, pkg/db.
|
|
## files
|
|
- tokens.ts | JWT gen/val | exp: issueToken, verifyToken, refreshToken | dep: crypto/hmac, db/sessions
|
|
- middleware.ts | HTTP auth guard | exp: requireAuth, requireRole | dep: tokens/verifyToken
|
|
- types.ts | shared auth types | exp: AuthToken, UserClaims, Role
|
|
## arch
|
|
Guard pattern on routes. Tokens short-lived (15m), refresh long-lived (7d). Rotation on every use.
|
|
Session state stored in Redis via db/sessions. No server-side JWT storage.
|
|
## dirty
|
|
-
|
|
```
|
|
|
|
### Rules
|
|
- One file per directory, placed inside that directory.
|
|
- Every non-excluded file in the directory gets one bullet under `## files`.
|
|
- Subdirectories are referenced in `## role` via `Dep:` or in `## arch` as structural notes, not duplicated.
|
|
- The `## dirty` section is empty (`-`) when clean, or contains a timestamp/flag when stale.
|
|
|
|
## 3. Pipeline Architecture
|
|
|
|
### Hybrid Extraction: LLM + AST
|
|
|
|
Two independent extraction layers contribute to the same output file.
|
|
|
|
#### Layer 1: LLM-Based Extraction (All Files)
|
|
- **Input**: Raw file contents of every non-excluded file in the directory.
|
|
- **Output**: Purpose description, architectural role, and cross-file relationships.
|
|
- **Applies to**: Code files, config files, Dockerfiles, READMEs, YAML, JSON, shell scripts — everything.
|
|
- **When it runs**: Once per file during init; again on changed files during patching.
|
|
- **Implementation**: Calls an actual LLM (not regex heuristics). Inside Pi, it uses Pi's built-in LLM via the ExtensionAPI. Standalone CLI falls back to an external LLM API (OpenAI-compatible).
|
|
|
|
#### Layer 2: AST-Based Extraction (Code Files Only)
|
|
- **Input**: Source code of files where a tree-sitter or LSP parser is available.
|
|
- **Output**: Precise symbol lists (functions, classes, types), signatures, import/export graphs, class hierarchies.
|
|
- **Applies to**: Supported languages only (TypeScript, Python, Go, Rust, etc.).
|
|
- **When it runs**: Once per file during init; again on changed files during patching.
|
|
|
|
#### Merging
|
|
The two layers merge into a single line per file under `## files`:
|
|
|
|
```
|
|
- tokens.ts | JWT gen/val | exp: issueToken, verifyToken, refreshToken | dep: crypto/hmac, db/sessions
|
|
^ LLM ^ LLM ^ AST ^ AST + LLM
|
|
```
|
|
|
|
- File name and purpose: LLM.
|
|
- Exported symbols and signatures: AST (augmented by LLM if AST unavailable).
|
|
- Dependency list: AST for imports; LLM for inferred architectural dependencies.
|
|
|
|
### LLM Client Architecture
|
|
|
|
The LLM client is abstracted behind a unified interface:
|
|
|
|
```typescript
|
|
interface LLMClient {
|
|
complete(prompt: string): Promise<string>;
|
|
}
|
|
```
|
|
|
|
Two implementations:
|
|
|
|
1. **PiLLMClient** (Pi extension): Uses `ctx.model` or `ctx.modelRegistry` to invoke Pi's configured LLM. Called from `pi-extension.ts` when the skill runs inside Pi.
|
|
2. **ExternalLLMClient** (standalone CLI): Calls an external OpenAI-compatible API. Configured via environment variable (e.g., `OPENAI_API_KEY`) or config file.
|
|
|
|
### Caching
|
|
|
|
LLM results are cached to avoid re-querying unchanged files.
|
|
|
|
- **Key**: SHA-256 hash of file contents.
|
|
- **Storage**: JSON file at `~/.cache/pi-project-map/llm-cache.json`.
|
|
- **Behavior**: Before calling the LLM, compute the file hash and check the cache. If hit, reuse the cached result. If miss, call the LLM and store the result.
|
|
- **Invalidation**: Cache entries are implicitly invalidated when the file content changes (because the hash changes). There is no TTL; the cache is append-only.
|
|
|
|
### Parallelization and Rate Limiting
|
|
|
|
- **Concurrency**: 4-8 LLM calls in parallel, controlled by `p-limit`.
|
|
- **Batch delays**: A small delay (e.g., 100ms) is inserted between batches to avoid triggering rate limits.
|
|
- **Retry policy**: Each LLM call retries up to 3 times with exponential backoff (1s, 2s, 4s). If all retries fail, the entire operation stops with a hard error.
|
|
|
|
### Error Handling
|
|
|
|
- **Hard error on failure**: If an LLM call fails after all retries, `init` or `patch` stops immediately and prints a clear error. There is no heuristic fallback. The user must resolve the issue (set API key, wait for rate limit, check network).
|
|
- **Context limit protection**: Files larger than the LLM's context window are truncated from the end (with a note in the prompt) before being sent.
|
|
|
|
### Init Pipeline
|
|
```
|
|
For each directory (depth-first):
|
|
1. List all non-excluded files.
|
|
2. For each file (parallel, 4-8 concurrent):
|
|
a. Compute SHA-256 of file contents.
|
|
b. Check disk cache. If hit, use cached result.
|
|
c. If miss: call LLM (with retries/backoff) to extract purpose and role.
|
|
d. Store result in cache.
|
|
e. If code file + parser available: run AST extraction (symbols, imports).
|
|
3. Merge per-file outputs into lines.
|
|
4. Run LLM on merged lines + directory context to generate:
|
|
- `## role` (package-level summary)
|
|
- `## arch` (architectural notes)
|
|
5. Write `.pi-map.md` to directory.
|
|
```
|
|
|
|
### Patch Pipeline
|
|
```
|
|
When agent edits file(s):
|
|
1. Classify patch mode: auto, small, or structural.
|
|
2. Always regenerate the changed directory pair:
|
|
- `.pi-map.md`
|
|
- `.pi-map.index.md`
|
|
3. For small changes: refresh ancestor indexes.
|
|
4. For structural changes: refresh ancestor map/index pairs.
|
|
5. Use `validate --fix` to repair affected chains when paired artifacts are stale or missing.
|
|
```
|
|
|
|
## 4. LLM Prompt Design
|
|
|
|
### File-Level Prompt
|
|
|
|
The LLM prompt for a single file is designed to produce a structured, concise analysis.
|
|
|
|
```
|
|
You are analyzing a source file for a project map. Read the file below and summarize:
|
|
|
|
1. PURPOSE: What does this file do? Describe its role in the project (2-3 sentences max).
|
|
2. DEPENDENCIES: What does this file depend on? List internal modules/packages and external libraries.
|
|
3. KEY CONCEPTS: Mention any important patterns, algorithms, or domain concepts.
|
|
|
|
File path: <file-path>
|
|
|
|
```
|
|
<file-contents-truncated>
|
|
```
|
|
|
|
Respond in this exact format:
|
|
PURPOSE: <concise description>
|
|
DEPS: <comma-separated list, or "none">
|
|
CONCEPTS: <comma-separated list, or "none">
|
|
```
|
|
|
|
### Package-Level Prompt
|
|
|
|
After all file summaries are collected for a directory, a second LLM call synthesizes the package role and architecture.
|
|
|
|
```
|
|
You are analyzing a directory in a software project. Below is a list of files in this directory with their purposes.
|
|
|
|
Directory: <dir-path>
|
|
Files:
|
|
- <file1>: <purpose1>
|
|
- <file2>: <purpose2>
|
|
...
|
|
|
|
Respond in this exact format:
|
|
ROLE: <one-line description of this directory's role in the project>
|
|
ARCH: <2-4 sentences describing architecture, data flow, patterns, and design decisions>
|
|
```
|
|
|
|
### Output Parsing
|
|
|
|
The LLM client's response is parsed to extract `PURPOSE`, `DEPS`, `CONCEPTS`, `ROLE`, and `ARCH` fields. These are merged with AST data into the final `.pi-map.md` format.
|
|
|
|
### Context Limit Protection
|
|
|
|
- Files are truncated from the end if they exceed a configurable max token budget (default: 4000 tokens of source).
|
|
- A marker `[...truncated]` is appended to the truncated content so the LLM knows it is not seeing the full file.
|
|
- Very large binary or generated files are skipped entirely for LLM analysis (they still appear in `.pi-map.md` with a note like "Large/generated file").
|
|
|
|
## 5. Consumption Model
|
|
|
|
### Session Start
|
|
1. Agent reads the root `Project Map Protocol` and root `.pi-map.index.md`.
|
|
2. For architecture/system or ambiguous tasks, agent also reads the root `.pi-map.md`.
|
|
3. Agent does **not** preload every directory map by default.
|
|
|
|
In the Pi extension, this session-start consumption is assisted by automatic prompt injection:
|
|
|
|
- **Before init**: only a lightweight visible startup hint is injected, telling the agent to run `project_map_init`. No synthetic map content is injected.
|
|
- **After init**: the root pair (`.pi-map.index.md` + `.pi-map.md`) is guaranteed to be preloaded automatically. Additional directory pairs are expanded only while the configured context budget allows.
|
|
|
|
### Automatic Prompt Injection
|
|
|
|
The Pi extension uses event hooks (`before_agent_start`, `context`, etc.) to maintain guidance context.
|
|
|
|
- The **mode ladder** controls how much is injected:
|
|
- `off`: no automatic injection.
|
|
- `advisory`: visible startup/init hints and optional root-pair preload.
|
|
- `strong` (default): root pair + budgeted expansion + relevant-turn reinjection checks.
|
|
- `strict`: same as `strong`, plus explicit bypass justification for sensitive edits/architectural claims when the protocol path is missing.
|
|
- The **protocol path** requires both the canonical injected root-pair block and the trust-boundary instruction (`index routes, map orients, source decides`) to be present in outgoing context.
|
|
- **Reinjection avoidance** scans actual outgoing messages (and falls back to provider payload) for a stable canonical marker before adding the root pair again.
|
|
- **Relevant-turn triggers** are: agent start, before edits, before architecture-sensitive reasoning, after compaction, and after root-pair artifact changes.
|
|
- **Visibility** is mixed: startup/init hints are user-visible; raw injected artifact blocks are agent-visible by default.
|
|
|
|
### Context Budget
|
|
|
|
Default automatic-injection budget: **15% of the active model context window**, capped at **100k tokens**. The smaller of the relative and absolute values wins. If the runtime cannot discover the active model's context window, it falls back to the absolute cap.
|
|
|
|
Configurable via `.pi-project-map.json`:
|
|
|
|
```json
|
|
{
|
|
"promptInjectionMode": "strong",
|
|
"contextBudgetPercent": 15,
|
|
"contextBudgetMaxTokens": 100000
|
|
}
|
|
```
|
|
|
|
### During Session
|
|
- Use directory indexes first to decide what to open next.
|
|
- For **targeted queries**, run `project_map_context` (tool) or `project-map context` (CLI). The retrieval engine scores all paired metadata and returns a compact markdown bundle with the top-3 strongest matches: indexes, maps, likely files, and symbols.
|
|
- Open the strongest-match `.pi-map.md` files for richer orientation.
|
|
- Read actual source before editing or making exact runtime claims.
|
|
|
|
### Context Management
|
|
- Tier 0 stays tiny and stable.
|
|
- Tier 1 loads only likely relevant indexes/maps.
|
|
- Tier 2 is real source, tests, config, and docs.
|
|
- Automatic injection stays within the configured budget and avoids redundant reinjection by scanning outgoing context.
|
|
|
|
## 6. Stale Data Mitigation
|
|
|
|
### Combined Strategy
|
|
|
|
#### 5.1 Dirty Markers
|
|
- Whenever the agent edits a file, it appends a dirty flag to the directory's `.pi-map.md`:
|
|
```markdown
|
|
## dirty
|
|
2024-06-09T14:32:00Z: tokens.ts modified
|
|
```
|
|
- A background or post-session reconciliation step regenerates dirty files.
|
|
- The agent can also be instructed to reconcile before making architectural decisions.
|
|
|
|
#### 5.2 Periodic Full Re-init
|
|
- On every new session start, or on a configurable schedule (e.g., daily), the skill offers to run a full re-scan.
|
|
- This catches any changes made outside the agent's awareness (e.g., by other developers).
|
|
|
|
#### 5.3 Validation Command
|
|
- A `validate` tool/command that the agent can invoke:
|
|
- Checks for missing files (new files not in `.pi-map.md`).
|
|
- Checks for orphaned entries (files listed but deleted).
|
|
- Checks for changed signatures (AST mismatch between listed symbols and actual code).
|
|
- Reports discrepancies and suggests corrections.
|
|
|
|
### Recovery
|
|
- If validation finds staleness beyond a threshold (e.g., > 3 dirty packages), the skill recommends a full re-init.
|
|
- The agent can also trigger re-init for a specific subtree.
|
|
|
|
## 7. Scope Boundaries and Non-Goals
|
|
|
|
### In Scope
|
|
- Every directory in the project gets a `.pi-map.md` file.
|
|
- Every non-excluded file gets analyzed by the LLM layer.
|
|
- Code files get augmented by the AST layer where parsers exist.
|
|
- Respect `.gitignore` and known junk patterns (node_modules, .git, dist, build, coverage, .next, .venv, __pycache__, .DS_Store).
|
|
|
|
### Out of Scope (Non-Goals)
|
|
- **Human-readable documentation**: These files are machine-only. Human docs live elsewhere.
|
|
- **Line-by-line code explanation**: The format captures symbols and architecture, not implementation details.
|
|
- **Auto-regeneration on filesystem events**: The skill relies on agent-initiated updates and periodic re-init, not filesystem watchers.
|
|
- **Cross-project analysis**: Each project is independent. No global index across repos.
|
|
- **IDE integration**: This is a Pi agent skill, not a VS Code extension or LSP server.
|
|
|
|
## 8. Pi Skill Package Structure
|
|
|
|
```
|
|
pi-project-map/
|
|
├── SKILL.md # Skill definition for Pi
|
|
├── package.json # npm package metadata
|
|
├── src/
|
|
│ ├── init.ts # Full project scan + generation
|
|
│ ├── patch.ts # Incremental patch logic
|
|
│ ├── validate.ts # Consistency checker
|
|
│ ├── ast-extract.ts # Tree-sitter / LSP wrappers
|
|
│ ├── llm-extract.ts # LLM prompt templates for extraction
|
|
│ ├── merge.ts # Merge AST + LLM outputs
|
|
│ ├── format.ts # Dense markdown formatter
|
|
│ ├── config.ts # Skill configuration (thresholds, ignore patterns)
|
|
│ └── prompt-injection.ts # Runtime guidance injection policy and helpers
|
|
├── hooks/
|
|
│ └── on-prompt.ts # Injects maintenance command into prompts (legacy; Pi extension uses event hooks)
|
|
└── README.md # Setup and usage for humans
|
|
```
|
|
|
|
### Custom Tools
|
|
- `project-map:init` — Run full project scan. Creates all paired map/index artifacts.
|
|
- `project-map:patch <file-path>` — Update analysis for a specific file/directory.
|
|
- `project-map:validate` — Run consistency check across all paired artifacts.
|
|
- `project-map:context <query>` — Retrieve a compact markdown bundle of the most relevant directories, files, and symbols for a natural-language query.
|
|
- `project-map:reinit [path]` — Force re-initialization of entire project or subtree.
|
|
|
|
### Prompt Injection Hooks
|
|
|
|
The Pi extension registers event hooks instead of a single per-prompt append:
|
|
|
|
- `before_agent_start`: emits the pre-init hint when no artifacts exist, or preloads the root pair (plus budgeted expansion) after init.
|
|
- `context`: performs relevant-turn reinjection checks, detects compaction/artifact-change invalidation, and enforces `strict`-mode bypass guards.
|
|
- `before_provider_request`: optional fallback for marker scanning when message-layer detection is insufficient.
|
|
|
|
The injected maintenance reminder is:
|
|
|
|
> Start with the root `.pi-map.index.md`, use indexes first for routing, read the local `.pi-map.md` plus source before edits, run `project_map_patch` after source edits, and run `project_map_validate` before freshness-sensitive architectural handoff.
|
|
|
|
This is layered on top of the canonical root-pair block, which includes the trust boundary (`index routes, map orients, source decides`).
|
|
|
|
## 9. Risks and Tradeoffs
|
|
|
|
| Risk | Likelihood | Impact | Mitigation |
|
|
|------|-----------|--------|------------|
|
|
| Token bloat (1000+ dirs) | Medium | High | Summary mode, lazy loading, context budget |
|
|
| Stale analysis files | High | High | Dirty markers + periodic re-init + validation |
|
|
| Agent trusts stale data | Medium | High | Clear instructions to validate before architectural decisions |
|
|
| Expensive init on large repos | Medium | Medium | Parallelization, caching, optional incremental init |
|
|
| Overlap with LSP/typedoc | Low | Low | This is agent-context, not IDE tooling. Different use case. |
|
|
| AST parser unavailable | Medium | Low | Graceful fallback to LLM-only extraction |
|
|
| Message-level marker scanning misses provider serialization quirks | Medium | Medium | Add payload fallback scanning |
|
|
| Root-pair marker becomes brittle | Low | Medium | Stable deterministic boundaries and normalized artifact identity lines |
|
|
| 15% / 100k default budget too aggressive for some fleets | Low | Medium | Both knobs are configurable |
|
|
| Relevant-turn detection fuzzy | Medium | Medium | Centralized heuristics + extensive integration tests |
|
|
| `strict` mode friction | Low | Medium | Keep `strong` as default; isolate strict-only bypass behavior |
|
|
|
|
### Prompt Injection Known Limitations
|
|
|
|
- Token estimation is best-effort (≈ 4 chars per token); actual provider token counts may differ.
|
|
- Relevant-turn detection relies on explicit event types when available, with heuristic fallback for generic turns.
|
|
- Provider payload serialization may require the fallback scan path.
|
|
- The mode ladder is config-driven in v1; future UX may expose runtime controls.
|
|
- Retrieval (`project_map_context`) remains separate from automatic injection.
|
|
|
|
## 10. Concrete Example: Full Project Snapshot
|
|
|
|
```
|
|
project-root/
|
|
├── .pi-map.md
|
|
├── src/
|
|
│ ├── .pi-map.md
|
|
│ ├── auth/
|
|
│ │ ├── .pi-map.md
|
|
│ │ ├── tokens.ts
|
|
│ │ ├── middleware.ts
|
|
│ │ └── types.ts
|
|
│ └── db/
|
|
│ ├── .pi-map.md
|
|
│ ├── connection.ts
|
|
│ └── migrations/
|
|
│ ├── .pi-map.md
|
|
│ └── 001_init.sql
|
|
├── docker/
|
|
│ ├── .pi-map.md
|
|
│ ├── Dockerfile
|
|
│ └── docker-compose.yml
|
|
└── README.md
|
|
```
|
|
|
|
Each `.pi-map.md` follows the format in Section 2, creating a navigable hierarchy.
|
|
|
|
## 11. Future Extensions
|
|
|
|
- **Cross-reference graph**: A top-level `project-graph.md` linking all packages with dependency arrows.
|
|
- **Search index**: A lightweight FTS5 index over all `.pi-map.md` files for fast symbol lookup.
|
|
- **Diff-aware patching**: Only re-run LLM on changed functions, not entire files.
|
|
- **Multi-repo workspaces**: Support monorepos with independent package boundaries.
|