Initial commit: pi-project-map skill scaffolding

- Design doc with full spec (dense markdown format, hybrid AST+LLM pipeline, consumption model, stale data mitigation) - Implementation plan with 6 milestones and rollout strategy - TypeScript package structure with all source stubs - CLI entry point, formatter, discover, init, patch, validate, extract, merge - Pi SKILL.md with tool definitions and format documentation
2026-06-09 17:45:19 +02:00
commit fd958671ca
20 changed files with 4667 additions and 0 deletions
@@ -0,0 +1,246 @@
+# Implementation Plan: Hierarchical Project Analysis Skill for Pi
+
+## Overview
+
+Build a Pi skill package (`pi-project-map`) that generates and maintains a hierarchical, machine-readable analysis of a software project. Each directory gets a `.pi-map.md` file. The skill provides custom tools for init, patch, validate, and re-init, plus a prompt hook that ensures the agent keeps analysis files in sync.
+
+---
+
+## Milestones
+
+### M1: Foundation and Format (Week 1)
+**Goal**: A working dense-markdown formatter and the ability to generate `.pi-map.md` files from a directory listing.
+
+#### Tasks
+1. **Bootstrap the npm package**
+   - Create `package.json`, `tsconfig.json`, basic CLI entry point.
+   - Set up test runner (vitest or jest).
+   - Create stub `SKILL.md` for Pi integration.
+
+2. **Implement the formatter (`src/format.ts`)**
+   - Define the `.pi-map.md` schema as a TypeScript interface.
+   - Implement `renderPackageMap(data) -> string` that outputs dense markdown.
+   - Implement `parsePackageMap(markdown) -> data` for reading existing files.
+   - Unit tests for round-trip serialization.
+
+3. **Implement file discovery (`src/discover.ts`)**
+   - Walk a directory tree.
+   - Respect `.gitignore` and a built-in ignore list (node_modules, .git, dist, etc.).
+   - Return a flat list of `(dirPath, filePaths[])` tuples.
+   - Unit tests with fixture directories.
+
+4. **Implement init skeleton (`src/init.ts`)**
+   - Given a project root, create one `.pi-map.md` per directory.
+   - Populate `## files` with filenames only (no analysis yet).
+   - Populate `## role` and `## arch` with placeholder text.
+   - Command: `project-map:init --root <path>`
+
+**Acceptance Criteria**
+- Running `project-map:init` on a test repo creates `.pi-map.md` in every directory.
+- Files match `.gitignore` rules correctly.
+- Output format follows the dense-markdown spec from the design doc.
+
+**Dependencies**: None.
+
+---
+
+### M2: LLM Extraction Layer (Week 2)
+**Goal**: The LLM layer generates one-line purposes and package-level architecture notes.
+
+#### Tasks
+1. **Implement LLM file extraction (`src/llm-extract.ts`)**
+   - Prompt template: given a file's contents and path, return a one-line purpose and a list of exported symbols (if any).
+   - Prompt template: given a file's contents, infer its dependencies.
+   - Integrate with Pi's LLM tool calls (or a generic OpenAI-compatible client for standalone testing).
+   - Add caching: hash file contents, skip LLM call if unchanged.
+
+2. **Implement LLM package extraction (`src/llm-extract.ts`)**
+   - Prompt template: given all per-file summaries in a directory, generate `## role` and `## arch`.
+   - Keep prompts token-efficient (truncate large files, focus on headers/exports).
+
+3. **Wire LLM layer into init (`src/init.ts`)**
+   - After file discovery, run LLM extraction on every file.
+   - Then run package-level LLM extraction.
+   - Write fully populated `.pi-map.md` files.
+
+4. **Add configuration (`src/config.ts`)**
+   - Configurable ignore patterns.
+   - Configurable LLM model and token limits.
+   - Configurable context budget for large projects.
+
+**Acceptance Criteria**
+- `project-map:init` produces `.pi-map.md` files with meaningful `## files`, `## role`, and `## arch` sections.
+- Caching works: second run on unchanged repo is fast.
+- Unit tests mock LLM responses to verify prompt structure and output parsing.
+
+**Dependencies**: M1.
+
+---
+
+### M3: AST Extraction Layer (Week 3)
+**Goal**: Code files get precise symbol and dependency data from tree-sitter or LSP.
+
+#### Tasks
+1. **Set up tree-sitter (`src/ast-extract.ts`)**
+   - Add tree-sitter dependencies for target languages (TypeScript, Python, Go, Rust as first-class).
+   - Implement `extractSymbols(filePath, language) -> { exports, imports, types }`.
+   - Implement `extractDependencies(filePath, language) -> string[]`.
+
+2. **Language detection**
+   - Map file extensions to tree-sitter grammars.
+   - Graceful fallback to LLM-only if no parser available.
+
+3. **Merge AST + LLM outputs (`src/merge.ts`)**
+   - Combine AST symbols with LLM purpose descriptions.
+   - AST provides `exp:` and `dep:`; LLM provides the one-line purpose.
+   - If AST and LLM disagree on dependencies, prefer AST for imports, LLM for inferred architectural deps.
+
+4. **Wire AST into init (`src/init.ts`)**
+   - For each code file, run AST extraction in parallel with LLM extraction.
+   - Merge results before formatting.
+
+**Acceptance Criteria**
+- TypeScript/Python files show precise exported function/class names in `exp:`.
+- Import statements are captured in `dep:`.
+- Files with no AST parser still get LLM-only analysis.
+- Unit tests with sample source files verify symbol extraction accuracy.
+
+**Dependencies**: M2.
+
+---
+
+### M4: Patch and Update (Week 4)
+**Goal**: The agent can incrementally update `.pi-map.md` files after editing source files.
+
+#### Tasks
+1. **Implement patch strategy (`src/patch.ts`)**
+   - Determine directory size (file count).
+   - If < 10 files: trigger full rewrite of that directory's `.pi-map.md`.
+   - If >= 10 files: parse existing `.pi-map.md`, identify the section(s) for changed file(s), and rewrite only those lines.
+
+2. **Implement dirty markers (`src/patch.ts`)**
+   - When a section-level patch occurs, append a dirty note:
+     ```markdown
+     ## dirty
+     2024-06-09T14:32:00Z: tokens.ts patched (section-level)
+     ```
+
+3. **Command: `project-map:patch <file-path>`**
+   - Given a changed file path, find its directory.
+   - Re-run LLM + AST extraction on that file only.
+   - Apply patch strategy.
+   - Update `.pi-map.md`.
+
+4. **Command: `project-map:reinit [path]`**
+   - Force full re-init of entire project or a subtree.
+   - Clear all dirty markers.
+
+**Acceptance Criteria**
+- Editing one file in a small package triggers full rewrite of that package's `.pi-map.md`.
+- Editing one file in a large package patches only the relevant line.
+- Dirty markers are correctly added on section-level patches.
+- Re-init clears dirty markers and regenerates everything.
+
+**Dependencies**: M3.
+
+---
+
+### M5: Validation and Consistency (Week 5)
+**Goal**: The agent can detect and recover from stale analysis files.
+
+#### Tasks
+1. **Implement validation (`src/validate.ts`)**
+   - Walk all `.pi-map.md` files.
+   - For each listed file, check existence (catch deletions).
+   - For each directory, check for new files not in `.pi-map.md`.
+   - For code files with AST support, compare listed `exp:` against actual symbols.
+   - Report discrepancies with severity (missing, orphaned, stale-signature).
+
+2. **Implement reconciliation (`src/validate.ts`)**
+   - Option `--fix`: automatically patch or flag discrepancies.
+   - Option `--reinit-if <threshold>`: if > N discrepancies, recommend full re-init.
+
+3. **Command: `project-map:validate`**
+   - Run checks and print a report.
+   - Exit code 0 if clean, 1 if discrepancies found.
+
+**Acceptance Criteria**
+- Validation detects a deleted file that still appears in `.pi-map.md`.
+- Validation detects a new file not yet in `.pi-map.md`.
+- Validation detects a changed function signature.
+- `--fix` patches or flags all found issues.
+
+**Dependencies**: M4.
+
+---
+
+### M6: Pi Integration (Week 6)
+**Goal**: The skill is installable and functional within the Pi agent harness.
+
+#### Tasks
+1. **Finalize `SKILL.md`**
+   - Document all custom tools: `project-map:init`, `project-map:patch`, `project-map:validate`, `project-map:reinit`.
+   - Document the dense-markdown format for the agent.
+   - Include usage examples and configuration options.
+
+2. **Implement prompt hook (`hooks/on-prompt.ts`)**
+   - Inject a lightweight instruction into every prompt:
+     > "If you modify any source file, run `project-map:patch <path>`. If you suspect staleness, run `project-map:validate`."
+   - Make injection configurable (toggle on/off, customize message).
+
+3. **Auto-inject session-start behavior**
+   - On session start, if `.pi-map.md` files exist, the skill auto-reads them into context.
+   - If they don't exist, the skill offers to run `project-map:init`.
+   - If dirty markers exist, the skill warns the agent.
+
+4. **Packaging and publishing**
+   - Ensure `package.json` has correct `bin` entries.
+   - Write human README with install instructions.
+   - Tag and publish to npm (or internal registry).
+
+**Acceptance Criteria**
+- Installing the skill in Pi makes the four tools available.
+- The prompt hook injects maintenance instructions reliably.
+- Session start auto-detects and loads `.pi-map.md` files.
+- The skill can be published and installed via npm.
+
+**Dependencies**: M5.
+
+---
+
+## Sequencing and Dependencies
+
+```
+M1 (Foundation) → M2 (LLM) → M3 (AST) → M4 (Patch) → M5 (Validate) → M6 (Pi Integration)
+```
+
+No parallel tracks — each milestone builds on the previous. Total estimated time: 6 weeks at a steady pace, or 3-4 weeks with focused effort.
+
+---
+
+## Validation Criteria (Overall)
+
+1. **Functional**: `project-map:init` correctly generates `.pi-map.md` for a test repo of 50+ files across 10+ directories.
+2. **Accuracy**: AST-extracted symbols match actual source code (verified by unit tests).
+3. **Freshness**: After editing a file, `project-map:patch` updates the analysis within 5 seconds.
+4. **Consistency**: `project-map:validate` detects 100% of synthetic staleness scenarios (deleted files, new files, signature changes).
+5. **Token efficiency**: A 100-file project's full `.pi-map.md` set fits within a 4k token budget.
+6. **Pi integration**: The skill installs cleanly, tools are discoverable, and the prompt hook works.
+
+---
+
+## Rollout Plan
+
+1. **Internal dogfooding** (Week 7): Use the skill on 2-3 real projects. Gather feedback on format density and agent behavior.
+2. **Format refinement** (Week 8): Adjust abbreviations, section structure, and dirty-marker format based on dogfooding.
+3. **Open beta** (Week 9): Share with a small group of Pi users. Collect bug reports and feature requests.
+4. **v1.0 release** (Week 10): Stable API, documented format, published to npm.
+
+---
+
+## Open Questions for Implementation
+
+- Which tree-sitter grammars to bundle by default? (Start with TypeScript, Python, Go.)
+- Should the LLM extraction use the agent's current model or a dedicated cheaper model?
+- How to handle monorepos with multiple `.gitignore` files at different depths?
+- Should `.pi-map.md` files be committed to git or kept in `.gitignore`? (Recommendation: `.gitignore` — they are derived artifacts.)