alex/pi-map
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: pi-project-map skill scaffolding

Browse Source 
- Design doc with full spec (dense markdown format, hybrid AST+LLM pipeline,
  consumption model, stale data mitigation)
- Implementation plan with 6 milestones and rollout strategy
- TypeScript package structure with all source stubs
- CLI entry point, formatter, discover, init, patch, validate, extract, merge
- Pi SKILL.md with tool definitions and format documentation
This commit is contained in:
Alex
2026-06-09 17:45:19 +02:00
commit fd958671ca
20 changed files with 4667 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
design-doc.md
+258
View File
@@ -0,0 +1,258 @@
# Design Doc: Hierarchical Project Analysis Skill for Pi
## 1. Goals and Success Criteria
### Primary Goal
Enable a Pi coding agent to understand a software project's architecture and code relationships without scanning the entire repository. The agent should have a compact, hierarchical "internal representation" of the project that it can consume in-context.
### Success Criteria
- The agent can orient itself in a new or familiar project without reading dozens of source files.
- The agent understands cross-package dependencies, data flows, and architectural patterns from the analysis files alone.
- Analysis files stay sufficiently fresh that the agent does not make decisions based on stale information.
- The representation is token-dense: maximum information per token, optimized for LLM consumption, not human readability.
## 2. Format Specification: Dense Markdown with Conventions
### Design Rationale
- **Not JSON/YAML**: Brackets, quotes, and indentation add token overhead with no benefit to LLM comprehension.
- **Not a custom DSL**: Fragile, requires a parser, and LLMs may hallucinate syntax.
- **Dense markdown**: Hierarchical headings, bullet points, and abbreviations are natively understood by LLMs and extremely token-efficient.
### Structure
Each directory in the project gets one analysis file named `.pi-map.md` (hidden by default, excluded from git via `.gitignore`).
```markdown
# <relative-path>
## role
<one-line package role> | Dep: <comma-separated upstream deps>
## files
- <filename> | <one-line purpose> | exp: <exported symbols> | dep: <internal/external deps>
- <filename> | <one-line purpose> | exp: <exported symbols> | dep: <internal/external deps>
## arch
<free-form architectural notes: patterns, data flow, invariants, design decisions>
## dirty
<timestamp or flag indicating staleness>
```
### Abbreviation Conventions
| Abbreviation | Meaning |
|-------------|---------|
| `exp:` | exported symbols (functions, classes, types, constants) |
| `dep:` | dependencies (other packages, files, or external libs) |
| `pkg/` | project-internal package reference |
| `ext/` | external dependency reference |
| `->` | data flow direction |
| `|` | field delimiter within a line |
### Example
```markdown
# pkg/auth
## role
Auth layer: JWT issuance, validation, refresh. Stateless. Dep: pkg/crypto, pkg/db.
## files
- tokens.ts | JWT gen/val | exp: issueToken, verifyToken, refreshToken | dep: crypto/hmac, db/sessions
- middleware.ts | HTTP auth guard | exp: requireAuth, requireRole | dep: tokens/verifyToken
- types.ts | shared auth types | exp: AuthToken, UserClaims, Role
## arch
Guard pattern on routes. Tokens short-lived (15m), refresh long-lived (7d). Rotation on every use.
Session state stored in Redis via db/sessions. No server-side JWT storage.
## dirty
-
```
### Rules
- One file per directory, placed inside that directory.
- Every non-excluded file in the directory gets one bullet under `## files`.
- Subdirectories are referenced in `## role` via `Dep:` or in `## arch` as structural notes, not duplicated.
- The `## dirty` section is empty (`-`) when clean, or contains a timestamp/flag when stale.
## 3. Pipeline Architecture
### Hybrid Extraction: LLM + AST
Two independent extraction layers contribute to the same output file.
#### Layer 1: LLM-Based Extraction (All Files)
- **Input**: Raw file contents of every non-excluded file in the directory.
- **Output**: One-line purpose description, architectural role, and cross-file relationships.
- **Applies to**: Code files, config files, Dockerfiles, READMEs, YAML, JSON, shell scripts — everything.
- **When it runs**: Once per file during init; again on changed files during patching.
#### Layer 2: AST-Based Extraction (Code Files Only)
- **Input**: Source code of files where a tree-sitter or LSP parser is available.
- **Output**: Precise symbol lists (functions, classes, types), signatures, import/export graphs, class hierarchies.
- **Applies to**: Supported languages only (TypeScript, Python, Go, Rust, etc.).
- **When it runs**: Once per file during init; again on changed files during patching.
#### Merging
The two layers merge into a single line per file under `## files`:
```
- tokens.ts | JWT gen/val | exp: issueToken, verifyToken, refreshToken | dep: crypto/hmac, db/sessions
^ LLM ^ LLM ^ AST ^ AST + LLM
```
- File name and one-line purpose: LLM.
- Exported symbols and signatures: AST (augmented by LLM if AST unavailable).
- Dependency list: AST for imports; LLM for inferred architectural dependencies.
### Init Pipeline
```
For each directory (depth-first):
1. List all non-excluded files.
2. For each file:
a. Run LLM extraction (purpose, role).
b. If code file + parser available: run AST extraction (symbols, imports).
3. Merge per-file outputs into lines.
4. Run LLM on merged lines + directory context to generate:
- `## role` (package-level summary)
- `## arch` (architectural notes)
5. Write `.pi-map.md` to directory.
```
### Patch Pipeline
```
When agent edits file(s) in directory:
1. Determine patch strategy:
- If directory has < 10 files: full rewrite.
- Else: section-level patch for changed file(s) only.
2. Re-run LLM extraction on changed file(s).
3. Re-run AST extraction on changed file(s) if applicable.
4. Update `## files` section (rewrite or patch).
5. Update `## dirty` flag if full regeneration is deferred.
```
## 4. Consumption Model
### Session Start
1. Agent discovers all `.pi-map.md` files (e.g., via `find . -name ".pi-map.md"`).
2. Agent reads **all** files into context. This is a one-time cost at session start.
3. Agent constructs an internal mental model of the project hierarchy.
### During Session
- An **auto-injected summary** stays in context (e.g., a condensed top-level `.pi-map.md` or a synthesized project overview).
- When the agent needs deeper detail about a specific package, it already has the full `.pi-map.md` in memory from step 2.
- If the agent enters a new package not yet loaded, it reads that package's `.pi-map.md` on demand.
### Context Management
- For very large projects, the agent may summarize or prune the initial read, keeping only the top N levels of the hierarchy in active context.
- The skill can provide a "context budget" parameter: max tokens to spend on analysis files.
## 5. Stale Data Mitigation
### Combined Strategy
#### 5.1 Dirty Markers
- Whenever the agent edits a file, it appends a dirty flag to the directory's `.pi-map.md`:
```markdown
## dirty
2024-06-09T14:32:00Z: tokens.ts modified
```
- A background or post-session reconciliation step regenerates dirty files.
- The agent can also be instructed to reconcile before making architectural decisions.
#### 5.2 Periodic Full Re-init
- On every new session start, or on a configurable schedule (e.g., daily), the skill offers to run a full re-scan.
- This catches any changes made outside the agent's awareness (e.g., by other developers).
#### 5.3 Validation Command
- A `validate` tool/command that the agent can invoke:
- Checks for missing files (new files not in `.pi-map.md`).
- Checks for orphaned entries (files listed but deleted).
- Checks for changed signatures (AST mismatch between listed symbols and actual code).
- Reports discrepancies and suggests corrections.
### Recovery
- If validation finds staleness beyond a threshold (e.g., > 3 dirty packages), the skill recommends a full re-init.
- The agent can also trigger re-init for a specific subtree.
## 6. Scope Boundaries and Non-Goals
### In Scope
- Every directory in the project gets a `.pi-map.md` file.
- Every non-excluded file gets analyzed by the LLM layer.
- Code files get augmented by the AST layer where parsers exist.
- Respect `.gitignore` and known junk patterns (node_modules, .git, dist, build, coverage, .next, .venv, __pycache__, .DS_Store).
### Out of Scope (Non-Goals)
- **Human-readable documentation**: These files are machine-only. Human docs live elsewhere.
- **Line-by-line code explanation**: The format captures symbols and architecture, not implementation details.
- **Auto-regeneration on filesystem events**: The skill relies on agent-initiated updates and periodic re-init, not filesystem watchers.
- **Cross-project analysis**: Each project is independent. No global index across repos.
- **IDE integration**: This is a Pi agent skill, not a VS Code extension or LSP server.
## 7. Pi Skill Package Structure
```
pi-project-map/
├── SKILL.md # Skill definition for Pi
├── package.json # npm package metadata
├── src/
│ ├── init.ts # Full project scan + generation
│ ├── patch.ts # Incremental patch logic
│ ├── validate.ts # Consistency checker
│ ├── ast-extract.ts # Tree-sitter / LSP wrappers
│ ├── llm-extract.ts # LLM prompt templates for extraction
│ ├── merge.ts # Merge AST + LLM outputs
│ ├── format.ts # Dense markdown formatter
│ └── config.ts # Skill configuration (thresholds, ignore patterns)
├── hooks/
│ └── on-prompt.ts # Injects maintenance command into prompts
└── README.md # Setup and usage for humans
```
### Custom Tools
- `project-map:init` — Run full project scan. Creates all `.pi-map.md` files.
- `project-map:patch <file-path>` — Update analysis for a specific file/directory.
- `project-map:validate` — Run consistency check across all `.pi-map.md` files.
- `project-map:reinit [path]` — Force re-initialization of entire project or subtree.
### Prompt Hook
- On every prompt, the skill appends a lightweight instruction:
> "If you modify any source file, run `project-map:patch <path>` to update the analysis. If you suspect staleness, run `project-map:validate`."
## 8. Risks and Tradeoffs
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Token bloat (1000+ dirs) | Medium | High | Summary mode, lazy loading, context budget |
| Stale analysis files | High | High | Dirty markers + periodic re-init + validation |
| Agent trusts stale data | Medium | High | Clear instructions to validate before architectural decisions |
| Expensive init on large repos | Medium | Medium | Parallelization, caching, optional incremental init |
| Overlap with LSP/typedoc | Low | Low | This is agent-context, not IDE tooling. Different use case. |
| AST parser unavailable | Medium | Low | Graceful fallback to LLM-only extraction |
## 9. Concrete Example: Full Project Snapshot
```
project-root/
├── .pi-map.md
├── src/
│ ├── .pi-map.md
│ ├── auth/
│ │ ├── .pi-map.md
│ │ ├── tokens.ts
│ │ ├── middleware.ts
│ │ └── types.ts
│ └── db/
│ ├── .pi-map.md
│ ├── connection.ts
│ └── migrations/
│ ├── .pi-map.md
│ └── 001_init.sql
├── docker/
│ ├── .pi-map.md
│ ├── Dockerfile
│ └── docker-compose.yml
└── README.md
```
Each `.pi-map.md` follows the format in Section 2, creating a navigable hierarchy.
## 10. Future Extensions
- **Cross-reference graph**: A top-level `project-graph.md` linking all packages with dependency arrows.
- **Search index**: A lightweight FTS5 index over all `.pi-map.md` files for fast symbol lookup.
- **Diff-aware patching**: Only re-run LLM on changed functions, not entire files.
- **Multi-repo workspaces**: Support monorepos with independent package boundaries.