Implement layered maps and context retrieval
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# Design: Map Context Retrieval
|
||||
|
||||
## Status
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | **Design** |
|
||||
| Based on | [Spec](spec.md) |
|
||||
| Next | Tasks |
|
||||
|
||||
## Design summary
|
||||
|
||||
This change adds a lightweight retrieval layer on top of paired project-map metadata. It should not require a new external storage system, vector store, or Engram dependency. Instead, it should scan root and directory indexes first, expand to rich maps and files from the strongest candidates, and emit an agent-friendly bundle.
|
||||
|
||||
## Likely implementation areas
|
||||
|
||||
- `pi-extension.ts` for the first-class tool surface
|
||||
- `src/index.ts` for exports
|
||||
- new retrieval module, e.g. `src/context.ts` or `src/retrieve.ts`
|
||||
- shared parsing/model code from the layered protocol
|
||||
- `src/cli/*` for follow-up CLI wiring
|
||||
- docs and skill guidance
|
||||
|
||||
## Retrieval pipeline
|
||||
|
||||
1. Accept a natural-language `query`
|
||||
2. Read root `.pi-map.index.md` and, when needed, root `.pi-map.md`
|
||||
3. Parse paired index/map metadata through the shared format/model layer
|
||||
4. Score candidate directories primarily from indexes
|
||||
5. Keep the top 3 candidates by default
|
||||
6. Expand strongest candidates to rich maps, likely files, and symbols
|
||||
7. Emit a compact markdown bundle with stable section order and retrieval-specific title `# Context bundle: <query>`
|
||||
|
||||
## Candidate scoring inputs
|
||||
|
||||
- direct term matches in workflow hints
|
||||
- direct term matches in likely files
|
||||
- parent/child/sibling link context
|
||||
- direct term matches in tags
|
||||
- direct term matches in symbols
|
||||
- path/name similarity
|
||||
- root workflow routing hits
|
||||
|
||||
A first implementation can use deterministic weighted lexical scoring.
|
||||
|
||||
Candidate-selection reasoning does not need to be exposed by default.
|
||||
|
||||
## Parsing strategy
|
||||
|
||||
Reuse the paired-artifact parser/model from the layered protocol. Avoid retrieval-specific ad hoc parsing.
|
||||
|
||||
## Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Retrieval becomes too fuzzy to trust | Keep output advisory and always direct agent back to source |
|
||||
| Pair parser complexity grows | Extend shared format/model logic instead of command-local parsing |
|
||||
| Command output becomes too large | Limit result count and keep instructions terse |
|
||||
|
||||
## Open choices
|
||||
|
||||
1. Exact scoring weights for workflow hints vs files vs tags vs symbols
|
||||
2. Whether the tool should support optional structured output later
|
||||
3. Whether broad architecture queries should be allowed to exceed the usual top-3 default in a later version
|
||||
4. Whether later versions should support optional root-rich-map inclusion as a flag
|
||||
@@ -0,0 +1,53 @@
|
||||
# Proposal: Map Context Retrieval
|
||||
|
||||
## Status
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | **Proposal** |
|
||||
| Based on | Follow-up to `layered-map-protocol` |
|
||||
| Next | Spec |
|
||||
|
||||
## Problem
|
||||
|
||||
After the layered map protocol lands, the agent will know it should load root protocol + root index first, then route through per-directory indexes and rich maps. But the repo will still lack an ergonomic retrieval primitive that can turn a natural-language task into a compact context bundle.
|
||||
|
||||
Without that helper, the agent still has to manually inspect directory indexes, rank candidate areas, and expand to rich maps/files. That weakens the value of the new routing layer.
|
||||
|
||||
## Proposed change
|
||||
|
||||
Add a retrieval tool and command shape:
|
||||
|
||||
```bash
|
||||
project-map context "<user task>"
|
||||
```
|
||||
|
||||
The first-class surface should be a Pi tool. CLI support can follow the same shape.
|
||||
|
||||
The retrieval result should return a compact context bundle containing:
|
||||
- relevant indexes,
|
||||
- strongest-match rich maps,
|
||||
- likely source files,
|
||||
- relevant symbols when useful,
|
||||
- short instructions on what to read next.
|
||||
|
||||
This remains a local metadata-driven retrieval feature, not a vector-store or Engram-backed memory system.
|
||||
## In scope
|
||||
|
||||
- [ ] Add Pi tool support for `project-map context <query>`
|
||||
- [ ] Add CLI support using the same shape after the tool contract is stable
|
||||
- [ ] Rank results using paired index/map metadata from the layered protocol
|
||||
- [ ] Return a compact, markdown-first, agent-friendly context bundle
|
||||
- [ ] Document recommended usage from prompts and docs
|
||||
## Out of scope
|
||||
|
||||
- [ ] Full semantic search across arbitrary source contents
|
||||
- [ ] Replacing source verification with map-based answers
|
||||
- [ ] Building a heavyweight external indexer or vector database
|
||||
|
||||
## Success criteria
|
||||
|
||||
- [ ] A natural-language task can resolve to likely indexes/maps/files/symbols without manual repo scanning
|
||||
- [ ] Output is compact enough to inject directly into the next step
|
||||
- [ ] Retrieval builds on generated paired metadata rather than bypassing it
|
||||
- [ ] The first version is tool-first and markdown-first
|
||||
@@ -0,0 +1,127 @@
|
||||
# Spec: Map Context Retrieval
|
||||
|
||||
## Status
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | **Spec** |
|
||||
| Based on | [Proposal](proposal.md) |
|
||||
| Next | Design |
|
||||
|
||||
## Overview
|
||||
|
||||
Add a retrieval-oriented `context` command that turns a user task into a compact project-map routing bundle. This change depends on the paired root/per-directory index+map metadata introduced by `layered-map-protocol`.
|
||||
|
||||
## Dependency
|
||||
|
||||
This change should not begin implementation before `layered-map-protocol` has landed enough metadata to support routing and ranking.
|
||||
|
||||
## Functional requirements
|
||||
|
||||
### 1. Tool-first surface
|
||||
Pi should expose `project-map context` as a first-class tool action so agents can request a context bundle directly.
|
||||
|
||||
For v1, the tool input should be minimal:
|
||||
- `query: string`
|
||||
|
||||
### 2. CLI follow-up surface
|
||||
The CLI should support the same conceptual shape:
|
||||
|
||||
```bash
|
||||
project-map context "<query>"
|
||||
```
|
||||
|
||||
CLI support may follow after the tool contract is stable.
|
||||
The CLI should mirror the same main input name: `query`.
|
||||
### 3. Bundle contents
|
||||
A context bundle must contain, at minimum:
|
||||
- relevant indexes
|
||||
- strongest-match relevant maps
|
||||
- likely files
|
||||
- relevant symbols when useful
|
||||
- short next-step instructions
|
||||
### 4. Ranking behavior
|
||||
The command must rank candidates using generated metadata such as:
|
||||
- root workflow hints
|
||||
- directory index routing hints
|
||||
- parent/child/sibling link structure
|
||||
- package tags
|
||||
- symbol references
|
||||
- task/workflow entries
|
||||
- `files:` / `index:` / `map:` targets
|
||||
|
||||
Indexes should be the first routing layer. Rich maps and symbols should expand from the strongest routed candidates.
|
||||
|
||||
By default, retrieval should usually surface the top 3 candidate directories, with adaptive omission of weaker rich maps when confidence drops.
|
||||
### 5. Trust boundary
|
||||
The bundle must instruct the agent to:
|
||||
- read indexes first for orientation,
|
||||
- read rich maps next when deeper context is needed,
|
||||
- read source before editing or asserting exact behavior.
|
||||
|
||||
## Output shape
|
||||
|
||||
The first version should be markdown-first, with stable, schema-like sections optimized for LLM consumption rather than human prose.
|
||||
|
||||
Default section order:
|
||||
1. `query`
|
||||
2. `relevant indexes`
|
||||
3. `relevant maps`
|
||||
4. `likely files`
|
||||
5. `relevant symbols`
|
||||
6. `instructions`
|
||||
|
||||
The retrieval title format should remain retrieval-specific:
|
||||
- `# Context bundle: <query>`
|
||||
|
||||
A representative response shape:
|
||||
|
||||
```markdown
|
||||
# Context bundle: validation stale signatures
|
||||
|
||||
## query
|
||||
validation stale signatures
|
||||
|
||||
## relevant indexes
|
||||
- .pi-map.index.md
|
||||
- src/.pi-map.index.md
|
||||
- tests/.pi-map.index.md
|
||||
|
||||
## relevant maps
|
||||
- src/.pi-map.md
|
||||
- tests/.pi-map.md
|
||||
|
||||
## likely files
|
||||
- src/validate.ts
|
||||
- src/ast/ast-extract.ts
|
||||
- tests/validate.test.ts
|
||||
|
||||
## relevant symbols
|
||||
- validateMaps
|
||||
- generateDirectoryMap
|
||||
- extractFileAST
|
||||
|
||||
## instructions
|
||||
Read the indexes first, then the strongest-match rich maps, then verify behavior from source before editing.
|
||||
```
|
||||
|
||||
The tool should omit weak/unhelpful symbol output rather than forcing a noisy symbol section.
|
||||
## Non-functional requirements
|
||||
|
||||
- Output must be compact enough for prompt injection.
|
||||
- Ranking should be deterministic enough to test.
|
||||
- The implementation should prefer lightweight heuristics over heavyweight indexing.
|
||||
- Retrieval should reuse the paired-artifact parsing layer instead of ad hoc string slicing.
|
||||
- Retrieval should not require a vector store, Engram, or another external memory backend.
|
||||
- The first version should accept natural language queries first; structured filters can come later.
|
||||
- Candidate explanations are not required by default.
|
||||
- Result-count override and root-rich-map inclusion can come later if needed, but are not part of v1.
|
||||
## Acceptance criteria
|
||||
|
||||
- [ ] Pi exposes `project-map context <query>` as a tool-first surface
|
||||
- [ ] CLI support follows the same shape after the tool contract is stable
|
||||
- [ ] Retrieval output includes indexes, strongest-match maps, files, and instructions
|
||||
- [ ] Symbol hints appear only when paired metadata makes them useful
|
||||
- [ ] Ranking respects the index-first routing model and usually returns top 3 candidates
|
||||
- [ ] Docs show when to use the retrieval command
|
||||
- [ ] Trust-boundary guidance remains explicit
|
||||
@@ -0,0 +1,34 @@
|
||||
# Tasks: Map Context Retrieval
|
||||
|
||||
## Status
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Phase | **Tasks** |
|
||||
| Based on | [Design](design.md) |
|
||||
| Next | Apply |
|
||||
|
||||
## Tasks
|
||||
|
||||
1. [ ] Add Pi tool support for `project-map context` with `query` as the main input
|
||||
2. [ ] Add retrieval module for scoring paired index/map metadata
|
||||
3. [ ] Reuse or extend shared paired-artifact parsing/model code
|
||||
4. [ ] Rank candidate directories from indexes first, then expand to maps/files/symbols
|
||||
5. [ ] Emit compact markdown-first context bundles with stable section order and retrieval-specific `Context bundle` title
|
||||
6. [ ] Add tests for retrieval ranking, top-3 default behavior, and output shape
|
||||
7. [ ] Add CLI support after the tool contract is stable
|
||||
8. [ ] Update docs and skill guidance for retrieval usage
|
||||
## Acceptance checklist
|
||||
|
||||
- [ ] Tool works for natural-language queries via `query`
|
||||
- [ ] Output includes relevant indexes, strongest-match maps, likely files, symbols when useful, and instructions
|
||||
- [ ] Retrieval depends on paired metadata rather than raw source scanning
|
||||
- [ ] Ranking follows the index-first routing model with a top-3 default
|
||||
- [ ] `npm run typecheck` passes
|
||||
- [ ] `npm test` passes
|
||||
- [ ] `npm run lint` passes
|
||||
|
||||
## Sequencing note
|
||||
This change is intentionally follow-up work. It should start after the layered map protocol produces reliable paired routing metadata.
|
||||
|
||||
The first delivery surface is the Pi tool. CLI support follows the stabilized tool contract.
|
||||
Reference in New Issue
Block a user