joakimp
fd8b15f570
ARCHITECTURE.md and README.md still carried the claim from954c3f2(initial commit) that the convos miner "keys on source_file path alone (convos miner doesn't check mtime)", and told the operator to delete the staging dir to force a re-mine.29e660ecorrected exactly that claim in AGENTS.md and SKILL.md — and missed these two files, so the repo has been documenting both behaviours at once ever since. Two files said mtime is checked, two said it is not. Ground truth, read off the deployed mempalace 3.7.1 rather than inferred: convo_miner.py:657 calls file_already_mined(..., check_mtime=True) inside mine_lock(source_file), and palace.py:1430 re-mines when no drawers exist for the source_file, when the stored normalize_version predates the current schema, or when the mtime differs. On a mismatch the file's existing drawers are purged (_source_file_delete_ids -> collection.delete) before refiling, so a changed transcript is replaced rather than doubled. The docstring names the case outright: transcripts are not assumed immutable, since a session keeps appending to its own file while active and /compact or /clear can rewrite one in place. The stale advice was not merely out of date, it was expensive. "Delete the staging dir to force a re-mine" is the one gesture that re-keys dedup: the staged path IS the key, so a stage that is wiped or recreated elsewhere makes the palace refile a whole wing as duplicates instead of replacing it. The docs now say so, name `touch` as the non-destructive way to force a single session, and record why staged copies must carry the source's mtime — with the corollary that an old mtime in a stage or a remote inbox says nothing about when the file was shipped, so a ship is judged by the feeder's log instead. Sample output blocks quoting "(dedup by source_file)" are deliberately left verbatim: that is what bin/mempalace-session:426 and bin/mempalace-pi-session:857 actually print. Tightening the wrappers' own wording is a separate change, because the samples have to move with it.
191 lines
12 KiB
Markdown
191 lines
12 KiB
Markdown
# AGENTS.md
|
|
|
|
## What this is
|
|
|
|
Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace). Three thin wrappers in `bin/` (opencode + pi session feeders, a docs miner), a companion agent skill, and an `extensions/` tree with per-harness bridges (currently `pi/`). Pairs with the consumer-side `mempalace` skill.
|
|
|
|
Read [`ARCHITECTURE.md`](ARCHITECTURE.md) first — it's the canonical spec for what this repo does and why.
|
|
|
|
## Structure
|
|
|
|
```
|
|
install.sh # Idempotent installer — see "What install.sh does" below
|
|
ARCHITECTURE.md # Canonical spec: diagrams, setup recipe, ops notes, upstream roadmap
|
|
README.md # Human-facing quickstart + per-tool usage reference
|
|
SKILL.md # Agent skill (symlinked into ~/.agents/skills/ on install)
|
|
docs/
|
|
rfc-001-global-palace.md # Design: one primary palace + per-machine local fallback (mempalace-edge)
|
|
rfc-002-joiner.md # Design: joining a machine-local palace into the primary
|
|
bin/
|
|
mempalace-docs # Docs-only MemPalace miner (bash wrapper)
|
|
mempalace-session # Opencode session → MemPalace bridge (bash + inline Python)
|
|
mempalace-pi-session # pi session → MemPalace bridge (bash + inline Python)
|
|
mempalace-census # RFC 002 Phase A: read-only join census of a palace on disk
|
|
# (classifies mined / diary / agent-authored, self-verifies
|
|
# ids, emits --json manifest). Reads, never writes.
|
|
contrib/ # systemd / launchd / cron templates for scheduling feeders
|
|
extensions/
|
|
pi/ # pi↔mempalace MCP bridge (bridge-only; pi's own config is in the pi-toolkit repo)
|
|
mempalace.ts # Symlinked into ~/.pi/agent/extensions/ (MCP <→ pi glue)
|
|
README.md # Bridge internals, Type.Unsafe gotcha, pi+mempalace deploy recipe
|
|
```
|
|
|
|
## What `install.sh` does
|
|
|
|
Idempotent, safe to re-run. Always:
|
|
|
|
- Symlinks `bin/*` into `~/.local/bin/`.
|
|
- Creates `~/.agents/skills/opencode-mempalace-bridge/` with a `SKILL.md` symlink and a `.skill-source` marker.
|
|
|
|
Gated on pi being installed (`~/.pi/agent/extensions/` exists):
|
|
|
|
- Symlinks `extensions/pi/mempalace.ts` into `~/.pi/agent/extensions/`. Backs up any real file in the way.
|
|
|
|
Probes (never halt, `warn` + `return 0`):
|
|
|
|
- `~/.local/bin` is on `$PATH`.
|
|
- `~/.config/opencode/instructions/mempalace.md` exists (opencode wake-up protocol).
|
|
- `mempalace` is registered as an MCP server in `~/.config/opencode/opencode.json`.
|
|
- If pi is installed: pi-toolkit artifacts (`~/.pi/agent/keybindings.json` symlink, `~/.oh-my-zsh/custom/pi-env.zsh`) exist. Warns with a `git clone ssh://...pi-toolkit.git` pointer if missing.
|
|
|
|
All non-destructive: if something is already in place and points into this repo, prints "already linked" and moves on. If a non-symlink real file is in the way, backs it up with a timestamp.
|
|
|
|
**Not handled here any more** (split to [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) on 2026-05-05):
|
|
|
|
- `keybindings.json` symlink into `~/.pi/agent/`
|
|
- `pi-env.zsh` cp into `~/.oh-my-zsh/custom/`
|
|
- `settings.example.json` template + `check_pi_settings` probe
|
|
- `check_aws_env` probe
|
|
|
|
Those are pi-generic concerns. This toolkit installs **only** the pi↔mempalace MCP bridge on top of whatever pi-toolkit set up.
|
|
|
|
|
|
## Conventions
|
|
|
|
- **Standalone executables** in `bin/` with `#!/usr/bin/env bash` shebang, no extension, `chmod +x`. Must work in non-interactive contexts (agent processes, cron, CI).
|
|
- **Thin wrappers only.** Neither tool reimplements the mempalace miner. Both follow the **stage-to-cache-then-mine** idiom: curate input to `~/.cache/…/<wing>/`, then delegate to `mempalace mine`.
|
|
- **Idempotent + dry-runnable.** Every tool supports `--dry-run`. Second invocation on unchanged input is a no-op (dedup via `source_file` path + `mtime` — all three miner modes pass `check_mtime=True`).
|
|
- **No external Python deps.** Stdlib only (`sqlite3`, `json`, `pathlib`). Inline in the bash wrapper via heredoc.
|
|
- Argument parsing: `--help`/`-h` first, then mode flags, then positional args.
|
|
- Comment sections use `# ── Section Name ──────` style (matches sibling `cli_utils` repo).
|
|
|
|
## Adding a new wrapper
|
|
|
|
Three wrappers live happily as standalone scripts — no shared helper library yet, because each one's stage-to-cache logic differs enough that the common surface is thin (arg parsing + `mempalace mine` invocation). A fourth wrapper might tip the balance; re-evaluate then. Until then, copy the pattern from `mempalace-session` (richest example):
|
|
|
|
1. Create `bin/<name>` with `#!/usr/bin/env bash` + `chmod +x`.
|
|
2. Implement `--help`, `--dry-run`, `--repair` flags (repair is opt-in; `--no-repair` kept as deprecated alias).
|
|
3. Stage to `~/.cache/<name>/<wing>/` with deterministic filenames.
|
|
4. Invoke `mempalace mine ...` (choose `--mode convos` if input is chat-like).
|
|
5. Do NOT end with `mempalace repair` unless `--repair` was explicitly passed. Repair is a destructive in-place HNSW rebuild and must never run on an unattended schedule.
|
|
6. Update `README.md` with usage + rationale.
|
|
7. Update `install.sh`? No — `bin/*` is auto-linked.
|
|
8. Update `ARCHITECTURE.md` if the wrapper fills a new architectural gap.
|
|
9. Update `SKILL.md` if agents should know when to invoke it.
|
|
|
|
## Adding a new harness extension
|
|
|
|
`extensions/<harness>/` is the home for **bridges** — code that lives
|
|
inside an agent runtime (pi, claude-code, kiro, …) and talks to the
|
|
mempalace MCP server. Currently only `extensions/pi/` exists. If you
|
|
add a second one (e.g. `extensions/claude-code/`), follow the same
|
|
shape:
|
|
|
|
1. **One directory per harness.** Never mix harnesses in one dir.
|
|
2. **Bridge-only scope.** This toolkit owns the mempalace-side wiring;
|
|
harness-generic config (keybindings, env loaders, settings templates)
|
|
belongs in a sibling `<harness>-toolkit` repo, following the pattern
|
|
established by [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit)
|
|
and [`opencode-toolkit`](https://gitea.jordbo.se/joakimp/opencode-toolkit).
|
|
That boundary is load-bearing for `opencode-devbox`'s slim container
|
|
path (mempalace opt-out, ~300 MB saved).
|
|
3. **A `README.md`** covering: what the bridge does, harness-specific
|
|
install path, debug knobs, and any gotchas (e.g. the pi `Type.Unsafe`
|
|
schema passthrough). May also hold the "Deploying <harness> with
|
|
mempalace" recipe since that straddles the two repos.
|
|
4. **Gate `install.sh` steps on the harness being present.** Detect via
|
|
a well-known path (pi uses `~/.pi/agent/extensions/`). Skip silently
|
|
on machines without that harness. Never force-install.
|
|
5. **Symlink the bridge code.** `mempalace.ts` (or equivalent) gets
|
|
symlinked into the harness's extensions directory so edits flow
|
|
through git. Back up any pre-existing real file to
|
|
`<path>.bak.YYYYMMDD-HHMMSS` before linking.
|
|
6. **Probe for the sibling toolkit.** After installing the bridge, check
|
|
whether the harness's own base config is in place (e.g. for pi-toolkit:
|
|
`~/.pi/agent/keybindings.json` symlink, `~/.oh-my-zsh/custom/pi-env.zsh`).
|
|
Warn with a `git clone` pointer if missing. `warn` + `return 0`, never
|
|
halt.
|
|
7. **Mirror in `--uninstall`.** Every symlink this repo creates must have
|
|
a matching removal step guarded by `link_if_into_repo`. Do **not**
|
|
touch sibling-toolkit-owned files — point the user at
|
|
`<harness>-toolkit/install.sh --uninstall` instead.
|
|
8. **Update the root `README.md`** — repo-contents list + Ecosystem
|
|
diagram's "Who owns what" table + Setup section's deploy summary.
|
|
9. **Update this file's Structure block** to list the new
|
|
`extensions/<harness>/` contents.
|
|
|
|
See `extensions/pi/README.md` and the `install_pi_extension` +
|
|
`check_pi_toolkit` functions in `install.sh` for a worked example.
|
|
|
|
## Testing
|
|
|
|
Manual only. Integration-shaped:
|
|
|
|
```bash
|
|
# Smoke test — does it parse args and list what would happen?
|
|
./bin/mempalace-session --help
|
|
./bin/mempalace-session --dry-run
|
|
|
|
# Real test on a single session (safe, deterministic)
|
|
./bin/mempalace-session --session ses_<id> --dry-run
|
|
./bin/mempalace-session --session ses_<id> # file into palace
|
|
mempalace_search "a phrase from that session" # verify visibility
|
|
./bin/mempalace-session --session ses_<id> # re-run → should skip
|
|
```
|
|
|
|
For `mempalace-docs`, test on a small repo (e.g. this one) first:
|
|
|
|
```bash
|
|
./bin/mempalace-docs "$PWD" --dry-run
|
|
```
|
|
|
|
## Gotchas
|
|
|
|
- `install.sh` is idempotent but interactive — use `--yes` in non-interactive contexts.
|
|
- `~/.local/bin` must be on `$PATH`. The installer warns if not.
|
|
- The companion skill lives at `~/.agents/skills/opencode-mempalace-bridge/SKILL.md` and is a **symlink into this repo**. Editing that file edits `SKILL.md` here. To propagate to Claude Code / Kiro, run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils).
|
|
- The opencode DB path defaults to `~/.local/share/opencode/opencode.db`. Override via `$OPENCODE_DB` or `--db`.
|
|
- The mempalace miner **skips symlinks** (as of v3.3.3 — `miner.py` line ~828). That's why the wrappers use `cp -p` / explicit file writes for staging, not symlinks.
|
|
- The convos miner dedups on `source_file` path **and** `mtime`
|
|
(`file_already_mined(..., check_mtime=True)` in upstream `convo_miner.py`).
|
|
A changed/grown transcript is detected, purged, and refiled — it is not
|
|
silently skipped. What must still be stable across runs is the *staged
|
|
path itself*: if a wrapper's staging dir is wiped, dedup has nothing to
|
|
compare against and `mempalace sync` will treat the vanished sources as
|
|
deleted and prune the drawers mined from them. (An earlier version of this
|
|
file claimed "no mtime check" — that was wrong; verified against
|
|
`convo_miner.py` 2026-08.)
|
|
- The docs miner dedups on `source_file` path + `mtime`. That's why staging uses `cp -p` (preserves mtime).
|
|
|
|
## Colocated skill pattern
|
|
|
|
This repo owns an agent skill (`SKILL.md`) that lives alongside the code it documents, rather than in a central skills repo like [`skillset`](https://gitea.jordbo.se/joakimp/skillset). The advantages: the skill moves in lockstep with the wrappers it explains, one `git clone` gets you the full producer-side setup, and retirement (when upstream gaps close) removes skill + code + docs in one commit.
|
|
|
|
The convention for making this coexist cleanly with sibling tooling:
|
|
|
|
1. **`install.sh` creates `~/.agents/skills/<name>/` as a real directory** containing a `SKILL.md` symlink back into this repo. It does **not** create a dir-symlink, because real dirs are the signal that sibling reconcilers (skillset's `deploy-skills.sh`, cli_utils's `agents-sync.zsh`) should leave the dir alone.
|
|
2. **`install.sh` drops a `.skill-source` marker file** at the root of the skill dir:
|
|
```
|
|
# skill-source: mempalace-toolkit
|
|
# repo: <absolute path>
|
|
# url: ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git
|
|
```
|
|
This is a breadcrumb for humans and future tooling — it answers "who owns this skill dir?" at a glance. `deploy-skills.sh` and `agents-sync.zsh` don't read it today (their existing logic already handles external dirs correctly) but may surface it in status reports later.
|
|
3. **`install.sh --uninstall` removes the marker** (only if it still says `mempalace-toolkit`) and the now-empty skill dir.
|
|
|
|
If you add a third colocated skill from a new repo, follow the same convention. The marker format is shared; only the repo name changes.
|
|
|
|
## History
|
|
|
|
Split out from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils) on 2026-04-30. The wrappers originated there but the conceptual fit was weak (`cli_utils` is interactive shell tools; these are agent memory infrastructure). Some older diary entries and KG facts in the palace reference the original paths.
|