Files

T

joakimp fd8b15f570 docs: the convos miner does check mtime — finish a correction that stopped half-way

ARCHITECTURE.md and README.md still carried the claim from 954c3f2 (initial
commit) that the convos miner "keys on source_file path alone (convos miner
doesn't check mtime)", and told the operator to delete the staging dir to force a
re-mine. 29e660e corrected exactly that claim in AGENTS.md and SKILL.md — and
missed these two files, so the repo has been documenting both behaviours at once
ever since. Two files said mtime is checked, two said it is not.

Ground truth, read off the deployed mempalace 3.7.1 rather than inferred:
convo_miner.py:657 calls file_already_mined(..., check_mtime=True) inside
mine_lock(source_file), and palace.py:1430 re-mines when no drawers exist for the
source_file, when the stored normalize_version predates the current schema, or
when the mtime differs. On a mismatch the file's existing drawers are purged
(_source_file_delete_ids -> collection.delete) before refiling, so a changed
transcript is replaced rather than doubled. The docstring names the case outright:
transcripts are not assumed immutable, since a session keeps appending to its own
file while active and /compact or /clear can rewrite one in place.

The stale advice was not merely out of date, it was expensive. "Delete the staging
dir to force a re-mine" is the one gesture that re-keys dedup: the staged path IS
the key, so a stage that is wiped or recreated elsewhere makes the palace refile a
whole wing as duplicates instead of replacing it. The docs now say so, name `touch`
as the non-destructive way to force a single session, and record why staged copies
must carry the source's mtime — with the corollary that an old mtime in a stage or
a remote inbox says nothing about when the file was shipped, so a ship is judged by
the feeder's log instead.

Sample output blocks quoting "(dedup by source_file)" are deliberately left
verbatim: that is what bin/mempalace-session:426 and bin/mempalace-pi-session:857
actually print. Tightening the wrappers' own wording is a separate change, because
the samples have to move with it.

2026-08-18 09:42:40 +02:00

12 KiB

Raw Blame History

AGENTS.md

What this is

Producer-side tooling for MemPalace. Three thin wrappers in bin/ (opencode + pi session feeders, a docs miner), a companion agent skill, and an extensions/ tree with per-harness bridges (currently pi/). Pairs with the consumer-side mempalace skill.

Read ARCHITECTURE.md first — it's the canonical spec for what this repo does and why.

Structure

install.sh              # Idempotent installer — see "What install.sh does" below
ARCHITECTURE.md         # Canonical spec: diagrams, setup recipe, ops notes, upstream roadmap
README.md               # Human-facing quickstart + per-tool usage reference
SKILL.md                # Agent skill (symlinked into ~/.agents/skills/ on install)
docs/
  rfc-001-global-palace.md  # Design: one primary palace + per-machine local fallback (mempalace-edge)
  rfc-002-joiner.md         # Design: joining a machine-local palace into the primary
bin/
  mempalace-docs        # Docs-only MemPalace miner (bash wrapper)
  mempalace-session     # Opencode session → MemPalace bridge (bash + inline Python)
  mempalace-pi-session  # pi session → MemPalace bridge (bash + inline Python)
  mempalace-census      # RFC 002 Phase A: read-only join census of a palace on disk
                        # (classifies mined / diary / agent-authored, self-verifies
                        #  ids, emits --json manifest). Reads, never writes.
contrib/                # systemd / launchd / cron templates for scheduling feeders
extensions/
  pi/                   # pi↔mempalace MCP bridge (bridge-only; pi's own config is in the pi-toolkit repo)
    mempalace.ts        # Symlinked into ~/.pi/agent/extensions/ (MCP <→ pi glue)
    README.md           # Bridge internals, Type.Unsafe gotcha, pi+mempalace deploy recipe

What `install.sh` does

Idempotent, safe to re-run. Always:

Symlinks bin/* into ~/.local/bin/.
Creates ~/.agents/skills/opencode-mempalace-bridge/ with a SKILL.md symlink and a .skill-source marker.

Gated on pi being installed (~/.pi/agent/extensions/ exists):

Symlinks extensions/pi/mempalace.ts into ~/.pi/agent/extensions/. Backs up any real file in the way.

Probes (never halt, warn + return 0):

~/.local/bin is on $PATH.
~/.config/opencode/instructions/mempalace.md exists (opencode wake-up protocol).
mempalace is registered as an MCP server in ~/.config/opencode/opencode.json.
If pi is installed: pi-toolkit artifacts (~/.pi/agent/keybindings.json symlink, ~/.oh-my-zsh/custom/pi-env.zsh) exist. Warns with a git clone ssh://...pi-toolkit.git pointer if missing.

All non-destructive: if something is already in place and points into this repo, prints "already linked" and moves on. If a non-symlink real file is in the way, backs it up with a timestamp.

Not handled here any more (split to pi-toolkit on 2026-05-05):

keybindings.json symlink into ~/.pi/agent/
pi-env.zsh cp into ~/.oh-my-zsh/custom/
settings.example.json template + check_pi_settings probe
check_aws_env probe

Those are pi-generic concerns. This toolkit installs only the pi↔mempalace MCP bridge on top of whatever pi-toolkit set up.

Conventions

Standalone executables in bin/ with #!/usr/bin/env bash shebang, no extension, chmod +x. Must work in non-interactive contexts (agent processes, cron, CI).
Thin wrappers only. Neither tool reimplements the mempalace miner. Both follow the stage-to-cache-then-mine idiom: curate input to ~/.cache/…/<wing>/, then delegate to mempalace mine.
Idempotent + dry-runnable. Every tool supports --dry-run. Second invocation on unchanged input is a no-op (dedup via source_file path + mtime — all three miner modes pass check_mtime=True).
No external Python deps. Stdlib only (sqlite3, json, pathlib). Inline in the bash wrapper via heredoc.
Argument parsing: --help/-h first, then mode flags, then positional args.
Comment sections use # ── Section Name ────── style (matches sibling cli_utils repo).

Adding a new wrapper

Three wrappers live happily as standalone scripts — no shared helper library yet, because each one's stage-to-cache logic differs enough that the common surface is thin (arg parsing + mempalace mine invocation). A fourth wrapper might tip the balance; re-evaluate then. Until then, copy the pattern from mempalace-session (richest example):

Create bin/<name> with #!/usr/bin/env bash + chmod +x.
Implement --help, --dry-run, --repair flags (repair is opt-in; --no-repair kept as deprecated alias).
Stage to ~/.cache/<name>/<wing>/ with deterministic filenames.
Invoke mempalace mine ... (choose --mode convos if input is chat-like).
Do NOT end with mempalace repair unless --repair was explicitly passed. Repair is a destructive in-place HNSW rebuild and must never run on an unattended schedule.
Update README.md with usage + rationale.
Update install.sh? No — bin/* is auto-linked.
Update ARCHITECTURE.md if the wrapper fills a new architectural gap.
Update SKILL.md if agents should know when to invoke it.

Adding a new harness extension

extensions/<harness>/ is the home for bridges — code that lives inside an agent runtime (pi, claude-code, kiro, …) and talks to the mempalace MCP server. Currently only extensions/pi/ exists. If you add a second one (e.g. extensions/claude-code/), follow the same shape:

One directory per harness. Never mix harnesses in one dir.
Bridge-only scope. This toolkit owns the mempalace-side wiring; harness-generic config (keybindings, env loaders, settings templates) belongs in a sibling <harness>-toolkit repo, following the pattern established by pi-toolkit and opencode-toolkit. That boundary is load-bearing for opencode-devbox's slim container path (mempalace opt-out, ~300 MB saved).
A README.md covering: what the bridge does, harness-specific install path, debug knobs, and any gotchas (e.g. the pi Type.Unsafe schema passthrough). May also hold the "Deploying with mempalace" recipe since that straddles the two repos.
Gate install.sh steps on the harness being present. Detect via a well-known path (pi uses ~/.pi/agent/extensions/). Skip silently on machines without that harness. Never force-install.
Symlink the bridge code. mempalace.ts (or equivalent) gets symlinked into the harness's extensions directory so edits flow through git. Back up any pre-existing real file to <path>.bak.YYYYMMDD-HHMMSS before linking.
Probe for the sibling toolkit. After installing the bridge, check whether the harness's own base config is in place (e.g. for pi-toolkit: ~/.pi/agent/keybindings.json symlink, ~/.oh-my-zsh/custom/pi-env.zsh). Warn with a git clone pointer if missing. warn + return 0, never halt.
Mirror in --uninstall. Every symlink this repo creates must have a matching removal step guarded by link_if_into_repo. Do not touch sibling-toolkit-owned files — point the user at <harness>-toolkit/install.sh --uninstall instead.
Update the root README.md — repo-contents list + Ecosystem diagram's "Who owns what" table + Setup section's deploy summary.
Update this file's Structure block to list the new extensions/<harness>/ contents.

See extensions/pi/README.md and the install_pi_extension + check_pi_toolkit functions in install.sh for a worked example.

Testing

Manual only. Integration-shaped:

# Smoke test — does it parse args and list what would happen?
./bin/mempalace-session --help
./bin/mempalace-session --dry-run

# Real test on a single session (safe, deterministic)
./bin/mempalace-session --session ses_<id> --dry-run
./bin/mempalace-session --session ses_<id>     # file into palace
mempalace_search "a phrase from that session"   # verify visibility
./bin/mempalace-session --session ses_<id>     # re-run → should skip

For mempalace-docs, test on a small repo (e.g. this one) first:

./bin/mempalace-docs "$PWD" --dry-run

Gotchas

install.sh is idempotent but interactive — use --yes in non-interactive contexts.
~/.local/bin must be on $PATH. The installer warns if not.
The companion skill lives at ~/.agents/skills/opencode-mempalace-bridge/SKILL.md and is a symlink into this repo. Editing that file edits SKILL.md here. To propagate to Claude Code / Kiro, run agents-sync from cli_utils.
The opencode DB path defaults to ~/.local/share/opencode/opencode.db. Override via $OPENCODE_DB or --db.
The mempalace miner skips symlinks (as of v3.3.3 — miner.py line ~828). That's why the wrappers use cp -p / explicit file writes for staging, not symlinks.
The convos miner dedups on source_file path and mtime (file_already_mined(..., check_mtime=True) in upstream convo_miner.py). A changed/grown transcript is detected, purged, and refiled — it is not silently skipped. What must still be stable across runs is the staged path itself: if a wrapper's staging dir is wiped, dedup has nothing to compare against and mempalace sync will treat the vanished sources as deleted and prune the drawers mined from them. (An earlier version of this file claimed "no mtime check" — that was wrong; verified against convo_miner.py 2026-08.)
The docs miner dedups on source_file path + mtime. That's why staging uses cp -p (preserves mtime).

Colocated skill pattern

This repo owns an agent skill (SKILL.md) that lives alongside the code it documents, rather than in a central skills repo like skillset. The advantages: the skill moves in lockstep with the wrappers it explains, one git clone gets you the full producer-side setup, and retirement (when upstream gaps close) removes skill + code + docs in one commit.

The convention for making this coexist cleanly with sibling tooling:

install.sh creates ~/.agents/skills/<name>/ as a real directory containing a SKILL.md symlink back into this repo. It does not create a dir-symlink, because real dirs are the signal that sibling reconcilers (skillset's deploy-skills.sh, cli_utils's agents-sync.zsh) should leave the dir alone.
install.sh drops a .skill-source marker file at the root of the skill dir:
```
# skill-source: mempalace-toolkit
# repo: <absolute path>
# url: ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git
```
This is a breadcrumb for humans and future tooling — it answers "who owns this skill dir?" at a glance. deploy-skills.sh and agents-sync.zsh don't read it today (their existing logic already handles external dirs correctly) but may surface it in status reports later.
install.sh --uninstall removes the marker (only if it still says mempalace-toolkit) and the now-empty skill dir.

If you add a third colocated skill from a new repo, follow the same convention. The marker format is shared; only the repo name changes.

History

Split out from cli_utils on 2026-04-30. The wrappers originated there but the conceptual fit was weak (cli_utils is interactive shell tools; these are agent memory infrastructure). Some older diary entries and KG facts in the palace reference the original paths.

12 KiB Raw Blame History