alex
7b67205d43
- design-doc.md: Rewrote Section 3 (Pipeline Architecture) to describe the real LLM integration: dual-provider LLM client (Pi native + external), SHA-256 disk cache in ~/.cache/pi-project-map/, parallelization with 4-8 concurrent requests, retries with exponential backoff, hard-error policy, context limit protection. Added new Section 4 with concrete file-level and package-level prompts. - implementation-plan.md: Replaced old milestone schedule with current status and detailed M7 tasks for LLM integration: LLM client abstraction, external API client, Pi LLM client, disk cache, parallel batching with retries, rewriting llm-extract.ts, context limits, CLI/extension wiring, and tests. - Minor formatting cleanup in pi-extension.ts and src/cli.ts All 16 tests pass. TypeScript compiles clean.
242 lines
8.3 KiB
Markdown
242 lines
8.3 KiB
Markdown
# Implementation Plan: Hierarchical Project Analysis Skill for Pi
|
|
|
|
## Overview
|
|
|
|
Build a Pi skill package (`pi-project-map`) that generates and maintains a hierarchical, machine-readable analysis of a software project. Each directory gets a `.pi-map.md` file. The skill provides custom tools for init, patch, validate, and re-init, plus a prompt hook that ensures the agent keeps analysis files in sync.
|
|
|
|
---
|
|
|
|
## Current Status
|
|
|
|
- **M1: Foundation and Format** — ✅ Complete
|
|
- **M2: Heuristic Extraction** — ✅ Complete (placeholder only; to be replaced by real LLM)
|
|
- **M3: AST Extraction** — ✅ Complete
|
|
- **M4: Patch and Update** — ✅ Complete
|
|
- **M5: Validation and `--fix`** — ✅ Complete
|
|
- **M6: Pi Skill Integration** — ✅ Complete (basic)
|
|
- **M7: Proper LLM Integration** — 🔄 In Progress / Next up
|
|
|
|
The remaining major work is to replace the heuristic "LLM" extraction with **actual LLM API calls**.
|
|
|
|
---
|
|
|
|
## Remaining Milestone: M7 — Proper LLM Integration
|
|
|
|
**Goal**: Replace the regex heuristic `llm-extract.ts` with real LLM calls, including dual-provider support (Pi native + external fallback), disk caching, parallelization, and retries.
|
|
|
|
**Estimated time**: 1 week of focused work.
|
|
|
|
---
|
|
|
|
### Task 1: LLM Client Abstraction (`src/llm-client.ts`)
|
|
|
|
Create a unified interface for LLM calls.
|
|
|
|
```typescript
|
|
interface LLMClient {
|
|
complete(prompt: string): Promise<string>;
|
|
}
|
|
```
|
|
|
|
Sub-tasks:
|
|
- Define the `LLMClient` interface.
|
|
- Add a factory function `createLLMClient(mode: 'pi' | 'external', options)`.
|
|
- Handle mode selection:
|
|
- `'pi'`: used inside `pi-extension.ts` with Pi's model registry.
|
|
- `'external'`: used in standalone CLI with OpenAI-compatible API.
|
|
|
|
**Acceptance Criteria**
|
|
- `LLMClient` interface exists and compiles.
|
|
- Factory correctly selects implementation based on mode.
|
|
|
|
---
|
|
|
|
### Task 2: External LLM Client (`src/external-llm-client.ts`)
|
|
|
|
Implement the standalone CLI LLM client using an OpenAI-compatible API.
|
|
|
|
Sub-tasks:
|
|
- Add `openai` (or a lightweight fetch-based client) as a dependency.
|
|
- Read configuration from:
|
|
- Environment variable `OPENAI_API_KEY` (or `ANTHROPIC_API_KEY`, etc.)
|
|
- Config file `.pi-project-map.json` fields: `llmProvider`, `llmModel`, `llmBaseUrl`
|
|
- Implement `complete(prompt)` using chat completions API.
|
|
- Default to `gpt-4o-mini` or similar cheap model.
|
|
|
|
**Acceptance Criteria**
|
|
- `project-map init` works standalone with `OPENAI_API_KEY` set.
|
|
- Missing API key produces a clear, actionable error message.
|
|
- Failed API call throws a descriptive `LLMError`.
|
|
|
|
---
|
|
|
|
### Task 3: Pi LLM Client (`src/pi-llm-client.ts`)
|
|
|
|
Implement the Pi-native LLM client for use inside the extension.
|
|
|
|
Sub-tasks:
|
|
- Accept Pi's `ExtensionContext` or model registry as a constructor argument.
|
|
- Use `ctx.model` / `ctx.modelRegistry` to get the configured model and API key.
|
|
- Call the provider directly (likely using the same OpenAI-compatible endpoint Pi uses).
|
|
- If Pi does not expose direct LLM calls, fall back to emitting a tool call / follow-up message pattern.
|
|
|
|
**Acceptance Criteria**
|
|
- Pi extension can successfully call an LLM when running inside Pi.
|
|
- Errors surface clearly to the user.
|
|
|
|
---
|
|
|
|
### Task 4: Disk Cache (`src/llm-cache.ts`)
|
|
|
|
Implement persistent SHA-256 → LLM result caching.
|
|
|
|
Sub-tasks:
|
|
- Cache directory: `~/.cache/pi-project-map/` (create if missing).
|
|
- Cache file: `llm-cache.json` (simple JSON object).
|
|
- Functions:
|
|
- `getCached(hash: string): string | null`
|
|
- `setCached(hash: string, result: string): void`
|
|
- Ensure atomic writes (write to temp file, rename).
|
|
- Add cache size limit (e.g., 10,000 entries, LRU eviction).
|
|
|
|
**Acceptance Criteria**
|
|
- Second `init` run on unchanged files does not call LLM.
|
|
- Cache persists across process restarts.
|
|
- Corrupted cache file does not crash the tool.
|
|
|
|
---
|
|
|
|
### Task 5: Parallelization and Retries (`src/llm-batch.ts`)
|
|
|
|
Run LLM requests in parallel with retries and backoff.
|
|
|
|
Sub-tasks:
|
|
- Add `p-limit` dependency for concurrency control.
|
|
- Default concurrency: 4 (configurable via `.pi-project-map.json` `llmConcurrency`).
|
|
- Add small delay (100ms) between batches.
|
|
- Implement retry logic: 3 retries, delays 1s → 2s → 4s.
|
|
- On final failure, throw a hard error and stop the entire process.
|
|
|
|
**Acceptance Criteria**
|
|
- 100-file project completes significantly faster than sequential.
|
|
- Simulated transient failures are retried and recovered.
|
|
- Persistent failure stops the tool with a clear error.
|
|
|
|
---
|
|
|
|
### Task 6: Rewrite `llm-extract.ts` to Use Real LLM
|
|
|
|
Replace regex heuristics with LLM calls.
|
|
|
|
Sub-tasks:
|
|
- Accept an `LLMClient` in `extractFileLLM` and `extractPackageLLM`.
|
|
- Check disk cache before calling LLM.
|
|
- Construct file-level prompt (see design doc Section 4).
|
|
- Parse response into `purpose`, `deps`, `concepts`.
|
|
- Construct package-level prompt.
|
|
- Parse response into `role`, `arch`.
|
|
- Remove heuristic code from production path; keep only for test mocks if useful.
|
|
|
|
**Acceptance Criteria**
|
|
- `llm-extract.ts` calls the LLM client for every file.
|
|
- Output includes rich, non-trivial descriptions for typical files.
|
|
- Unit tests mock the LLM client to verify prompt structure and parsing.
|
|
|
|
---
|
|
|
|
### Task 7: Context Limit Handling
|
|
|
|
Protect against oversized files.
|
|
|
|
Sub-tasks:
|
|
- Measure prompt + file content tokens (approximate: 1 token ≈ 4 chars for ASCII).
|
|
- If file exceeds max context budget (configurable, default 4000 tokens), truncate from the end.
|
|
- Append `[...truncated]` marker in the prompt.
|
|
- Skip LLM for binary/generated files over a hard limit (e.g., 50KB) and mark them as "Large/generated file".
|
|
|
|
**Acceptance Criteria**
|
|
- A 1MB minified JS file does not crash or consume excessive tokens.
|
|
- Truncated files still produce useful output.
|
|
|
|
---
|
|
|
|
### Task 8: Update CLI and Extension
|
|
|
|
Wire the new LLM client into all entry points.
|
|
|
|
Sub-tasks:
|
|
- `src/cli.ts`: create external LLM client, pass into `initProject` / `patchFile`.
|
|
- `pi-extension.ts`: create Pi LLM client, pass into tools.
|
|
- Update `init.ts` and `patch.ts` signatures to accept an optional `LLMClient`.
|
|
- Add CLI flag `--llm-provider=openai` for explicit selection.
|
|
- Update error handling to catch `LLMError` and print helpful messages.
|
|
|
|
**Acceptance Criteria**
|
|
- CLI works with external API key.
|
|
- Extension works inside Pi (if Pi exposes LLM access).
|
|
- Clear errors on misconfiguration.
|
|
|
|
---
|
|
|
|
### Task 9: Update Tests
|
|
|
|
Sub-tasks:
|
|
- Replace heuristic tests with mocked LLM client tests.
|
|
- Add integration test: create a fake LLM client, run `initProject`, verify output contains LLM-provided text.
|
|
- Add cache test: verify cache hit skips LLM call.
|
|
- Add retry test: verify transient failures retry, persistent failures hard-stop.
|
|
|
|
**Acceptance Criteria**
|
|
- All tests pass.
|
|
- Test coverage includes: LLM client, cache, batching, prompt parsing, error handling.
|
|
|
|
---
|
|
|
|
## Sequencing and Dependencies
|
|
|
|
```
|
|
Task 1 (LLMClient interface)
|
|
├── Task 2 (External client)
|
|
├── Task 3 (Pi client)
|
|
├── Task 4 (Cache)
|
|
├── Task 5 (Batch + retries)
|
|
│
|
|
├── Task 6 (Rewrite llm-extract.ts)
|
|
├── Task 7 (Context limits)
|
|
│
|
|
├── Task 8 (Wire CLI + extension)
|
|
└── Task 9 (Tests)
|
|
```
|
|
|
|
---
|
|
|
|
## Validation Criteria (for LLM Integration)
|
|
|
|
1. **Real LLM calls**: `llm-extract.ts` invokes the configured LLM client for every file.
|
|
2. **Cache hit**: Second `init` on unchanged repo completes with zero LLM calls.
|
|
3. **Parallel speed**: 100-file project init completes in under 30 seconds (assuming average LLM latency 500ms).
|
|
4. **Retry works**: Transient 429/5xx errors are retried; permanent failures stop with a clear error.
|
|
5. **Context limit safety**: Files > max token budget are truncated, not rejected.
|
|
6. **Dual provider**: CLI uses external API; Pi extension uses Pi's LLM.
|
|
7. **Quality**: LLM output is visibly richer than the old heuristic output (verified by manual inspection).
|
|
|
|
---
|
|
|
|
## Rollout Plan
|
|
|
|
1. **Test on real projects** (Day 1-2): Run `init` on 2-3 real codebases with the new LLM integration.
|
|
2. **Cost audit** (Day 3): Measure token usage per project; adjust defaults if too expensive.
|
|
3. **Prompt tuning** (Day 4-5): Iterate prompt design based on output quality.
|
|
4. **Release** (Day 6-7): Publish updated npm package, update Pi extension docs.
|
|
|
|
---
|
|
|
|
## Completed Milestones (for reference)
|
|
|
|
- **M1**: Foundation and Format
|
|
- **M2**: Heuristic Extraction (placeholder, to be replaced by M7)
|
|
- **M3**: AST Extraction
|
|
- **M4**: Patch and Update
|
|
- **M5**: Validation and `--fix`
|
|
- **M6**: Pi Skill Integration
|