pi-map/implementation-plan.md

Implementation Plan: Hierarchical Project Analysis Skill for Pi

Overview

Build a Pi skill package (pi-project-map) that generates and maintains a hierarchical, machine-readable analysis of a software project. Each directory gets a .pi-map.md file. The skill provides custom tools for init, patch, validate, and re-init, plus a prompt hook that ensures the agent keeps analysis files in sync.

---

Milestones

M1: Foundation and Format (Week 1)

Goal: A working dense-markdown formatter and the ability to generate .pi-map.md files from a directory listing.

Tasks

1. Bootstrap the npm package

   • Create package.json, tsconfig.json, basic CLI entry point.
   • Set up test runner (vitest or jest).
   • Create stub SKILL.md for Pi integration.

2. Implement the formatter (src/format.ts)

   • Define the .pi-map.md schema as a TypeScript interface.
   • Implement renderPackageMap(data) -> string that outputs dense markdown.
   • Implement parsePackageMap(markdown) -> data for reading existing files.
   • Unit tests for round-trip serialization.

3. Implement file discovery (src/discover.ts)

   • Walk a directory tree.
   • Respect .gitignore and a built-in ignore list (node_modules, .git, dist, etc.).
   • Return a flat list of (dirPath, filePaths[]) tuples.
   • Unit tests with fixture directories.

4. Implement init skeleton (src/init.ts)

   • Given a project root, create one .pi-map.md per directory.
   • Populate ## files with filenames only (no analysis yet).
   • Populate ## role and ## arch with placeholder text.
   • Command: project-map:init --root <path>

Acceptance Criteria

• Running project-map:init on a test repo creates .pi-map.md in every directory.
• Files match .gitignore rules correctly.
• Output format follows the dense-markdown spec from the design doc.

Dependencies: None.

---

M2: LLM Extraction Layer (Week 2)

Goal: The LLM layer generates one-line purposes and package-level architecture notes.

Tasks

1. Implement LLM file extraction (src/llm-extract.ts)

   • Prompt template: given a file's contents and path, return a one-line purpose and a list of exported symbols (if any).
   • Prompt template: given a file's contents, infer its dependencies.
   • Integrate with Pi's LLM tool calls (or a generic OpenAI-compatible client for standalone testing).
   • Add caching: hash file contents, skip LLM call if unchanged.

2. Implement LLM package extraction (src/llm-extract.ts)

   • Prompt template: given all per-file summaries in a directory, generate ## role and ## arch.
   • Keep prompts token-efficient (truncate large files, focus on headers/exports).

3. Wire LLM layer into init (src/init.ts)

   • After file discovery, run LLM extraction on every file.
   • Then run package-level LLM extraction.
   • Write fully populated .pi-map.md files.

4. Add configuration (src/config.ts)

   • Configurable ignore patterns.
   • Configurable LLM model and token limits.
   • Configurable context budget for large projects.

Acceptance Criteria

• project-map:init produces .pi-map.md files with meaningful ## files, ## role, and ## arch sections.
• Caching works: second run on unchanged repo is fast.
• Unit tests mock LLM responses to verify prompt structure and output parsing.

Dependencies: M1.

---

M3: AST Extraction Layer (Week 3)

Goal: Code files get precise symbol and dependency data from tree-sitter or LSP.

Tasks

1. Set up tree-sitter (src/ast-extract.ts)

   • Add tree-sitter dependencies for target languages (TypeScript, Python, Go, Rust as first-class).
   • Implement extractSymbols(filePath, language) -> { exports, imports, types }.
   • Implement extractDependencies(filePath, language) -> string[].

2. Language detection

   • Map file extensions to tree-sitter grammars.
   • Graceful fallback to LLM-only if no parser available.

3. Merge AST + LLM outputs (src/merge.ts)

   • Combine AST symbols with LLM purpose descriptions.
   • AST provides exp: and dep:; LLM provides the one-line purpose.
   • If AST and LLM disagree on dependencies, prefer AST for imports, LLM for inferred architectural deps.

4. Wire AST into init (src/init.ts)

   • For each code file, run AST extraction in parallel with LLM extraction.
   • Merge results before formatting.

Acceptance Criteria

• TypeScript/Python files show precise exported function/class names in exp:.
• Import statements are captured in dep:.
• Files with no AST parser still get LLM-only analysis.
• Unit tests with sample source files verify symbol extraction accuracy.

Dependencies: M2.

---

M4: Patch and Update (Week 4)

Goal: The agent can incrementally update .pi-map.md files after editing source files.

Tasks

1. Implement patch strategy (src/patch.ts)

   • Determine directory size (file count).
   • If < 10 files: trigger full rewrite of that directory's .pi-map.md.
   • If >= 10 files: parse existing .pi-map.md, identify the section(s) for changed file(s), and rewrite only those lines.

2. Implement dirty markers (src/patch.ts)

   • When a section-level patch occurs, append a dirty note:

     ## dirty
     2024-06-09T14:32:00Z: tokens.ts patched (section-level)
     

3. Command: project-map:patch <file-path>

   • Given a changed file path, find its directory.
   • Re-run LLM + AST extraction on that file only.
   • Apply patch strategy.
   • Update .pi-map.md.

4. Command: project-map:reinit [path]

   • Force full re-init of entire project or a subtree.
   • Clear all dirty markers.

Acceptance Criteria

• Editing one file in a small package triggers full rewrite of that package's .pi-map.md.
• Editing one file in a large package patches only the relevant line.
• Dirty markers are correctly added on section-level patches.
• Re-init clears dirty markers and regenerates everything.

Dependencies: M3.

---

M5: Validation and Consistency (Week 5)

Goal: The agent can detect and recover from stale analysis files.

Tasks

1. Implement validation (src/validate.ts)

   • Walk all .pi-map.md files.
   • For each listed file, check existence (catch deletions).
   • For each directory, check for new files not in .pi-map.md.
   • For code files with AST support, compare listed exp: against actual symbols.
   • Report discrepancies with severity (missing, orphaned, stale-signature).

2. Implement reconciliation (src/validate.ts)

   • Option --fix: automatically patch or flag discrepancies.
   • Option --reinit-if <threshold>: if > N discrepancies, recommend full re-init.

3. Command: project-map:validate

   • Run checks and print a report.
   • Exit code 0 if clean, 1 if discrepancies found.

Acceptance Criteria

• Validation detects a deleted file that still appears in .pi-map.md.
• Validation detects a new file not yet in .pi-map.md.
• Validation detects a changed function signature.
• --fix patches or flags all found issues.

Dependencies: M4.

---

M6: Pi Integration (Week 6)

Goal: The skill is installable and functional within the Pi agent harness.

Tasks

1. Finalize SKILL.md

   • Document all custom tools: project-map:init, project-map:patch, project-map:validate, project-map:reinit.
   • Document the dense-markdown format for the agent.
   • Include usage examples and configuration options.

2. Implement prompt hook (hooks/on-prompt.ts)

   • Inject a lightweight instruction into every prompt:

     "If you modify any source file, run project-map:patch <path>. If you suspect staleness, run project-map:validate."

   • Make injection configurable (toggle on/off, customize message).

3. Auto-inject session-start behavior

   • On session start, if .pi-map.md files exist, the skill auto-reads them into context.
   • If they don't exist, the skill offers to run project-map:init.
   • If dirty markers exist, the skill warns the agent.

4. Packaging and publishing

   • Ensure package.json has correct bin entries.
   • Write human README with install instructions.
   • Tag and publish to npm (or internal registry).

Acceptance Criteria

• Installing the skill in Pi makes the four tools available.
• The prompt hook injects maintenance instructions reliably.
• Session start auto-detects and loads .pi-map.md files.
• The skill can be published and installed via npm.

Dependencies: M5.

---

Sequencing and Dependencies

M1 (Foundation) → M2 (LLM) → M3 (AST) → M4 (Patch) → M5 (Validate) → M6 (Pi Integration)

No parallel tracks — each milestone builds on the previous. Total estimated time: 6 weeks at a steady pace, or 3-4 weeks with focused effort.

---

Validation Criteria (Overall)

1. Functional: project-map:init correctly generates .pi-map.md for a test repo of 50+ files across 10+ directories.
2. Accuracy: AST-extracted symbols match actual source code (verified by unit tests).
3. Freshness: After editing a file, project-map:patch updates the analysis within 5 seconds.
4. Consistency: project-map:validate detects 100% of synthetic staleness scenarios (deleted files, new files, signature changes).
5. Token efficiency: A 100-file project's full .pi-map.md set fits within a 4k token budget.
6. Pi integration: The skill installs cleanly, tools are discoverable, and the prompt hook works.

---

Rollout Plan

1. Internal dogfooding (Week 7): Use the skill on 2-3 real projects. Gather feedback on format density and agent behavior.
2. Format refinement (Week 8): Adjust abbreviations, section structure, and dirty-marker format based on dogfooding.
3. Open beta (Week 9): Share with a small group of Pi users. Collect bug reports and feature requests.
4. v1.0 release (Week 10): Stable API, documented format, published to npm.

---

Open Questions for Implementation

• Which tree-sitter grammars to bundle by default? (Start with TypeScript, Python, Go.)
• Should the LLM extraction use the agent's current model or a dedicated cheaper model?
• How to handle monorepos with multiple .gitignore files at different depths?
• Should .pi-map.md files be committed to git or kept in .gitignore? (Recommendation: .gitignore — they are derived artifacts.)