Files

T

Joakim Persson f60cf9c732 feat(census): RFC 002 Phase A — read-only join census, and three RFC corrections it found

bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.

The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):

  (a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
      _DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
      but _DELIM is dead code and _delimited_sha256 builds
      "".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
      ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.

  (b) id_recipe is NOT a mined-only marker. It looked like a clean
      discriminator (same 14,586 count as source_file) but the server stamps
      'v3' on content ids too, so classifying on `source_file OR id_recipe`
      swallowed all 60 agent-authored drawers into MINED -- the dangerous
      direction, since Phase C would try to re-mine drawers that have no source
      file and silently drop them. Discriminator is a TRUTHY source_file (the
      writer stores "" rather than omitting the key), cross-checked against the
      miner-only keys source_mtime / normalize_version; disagreement is now a
      first-class warning.

  (c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
      their own id, because update_drawer preserves the id while rewriting and
      re-chunking. So "recompute the content id and skip if present" -- the
      strategy this RFC specified for regime A -- misses every drifted drawer
      and duplicates it. Phase C must key on the STORED id. Flagged as
      edited_since_filing in the manifest so Phase C can be tested on them.

Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.

One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.

2026-08-15 09:06:40 +02:00

12 KiB

Raw Blame History

AGENTS.md

What this is

Producer-side tooling for MemPalace. Three thin wrappers in bin/ (opencode + pi session feeders, a docs miner), a companion agent skill, and an extensions/ tree with per-harness bridges (currently pi/). Pairs with the consumer-side mempalace skill.

Read ARCHITECTURE.md first — it's the canonical spec for what this repo does and why.

Structure

install.sh              # Idempotent installer — see "What install.sh does" below
ARCHITECTURE.md         # Canonical spec: diagrams, setup recipe, ops notes, upstream roadmap
README.md               # Human-facing quickstart + per-tool usage reference
SKILL.md                # Agent skill (symlinked into ~/.agents/skills/ on install)
docs/
  rfc-001-global-palace.md  # Design: one primary palace + per-machine local fallback (mempalace-edge)
  rfc-002-joiner.md         # Design: joining a machine-local palace into the primary
bin/
  mempalace-docs        # Docs-only MemPalace miner (bash wrapper)
  mempalace-session     # Opencode session → MemPalace bridge (bash + inline Python)
  mempalace-pi-session  # pi session → MemPalace bridge (bash + inline Python)
  mempalace-census      # RFC 002 Phase A: read-only join census of a palace on disk
                        # (classifies mined / diary / agent-authored, self-verifies
                        #  ids, emits --json manifest). Reads, never writes.
contrib/                # systemd / launchd / cron templates for scheduling feeders
extensions/
  pi/                   # pi↔mempalace MCP bridge (bridge-only; pi's own config is in the pi-toolkit repo)
    mempalace.ts        # Symlinked into ~/.pi/agent/extensions/ (MCP <→ pi glue)
    README.md           # Bridge internals, Type.Unsafe gotcha, pi+mempalace deploy recipe

What `install.sh` does

Idempotent, safe to re-run. Always:

Symlinks bin/* into ~/.local/bin/.
Creates ~/.agents/skills/opencode-mempalace-bridge/ with a SKILL.md symlink and a .skill-source marker.

Gated on pi being installed (~/.pi/agent/extensions/ exists):

Symlinks extensions/pi/mempalace.ts into ~/.pi/agent/extensions/. Backs up any real file in the way.

Probes (never halt, warn + return 0):

~/.local/bin is on $PATH.
~/.config/opencode/instructions/mempalace.md exists (opencode wake-up protocol).
mempalace is registered as an MCP server in ~/.config/opencode/opencode.json.
If pi is installed: pi-toolkit artifacts (~/.pi/agent/keybindings.json symlink, ~/.oh-my-zsh/custom/pi-env.zsh) exist. Warns with a git clone ssh://...pi-toolkit.git pointer if missing.

All non-destructive: if something is already in place and points into this repo, prints "already linked" and moves on. If a non-symlink real file is in the way, backs it up with a timestamp.

Not handled here any more (split to pi-toolkit on 2026-05-05):

keybindings.json symlink into ~/.pi/agent/
pi-env.zsh cp into ~/.oh-my-zsh/custom/
settings.example.json template + check_pi_settings probe
check_aws_env probe

Those are pi-generic concerns. This toolkit installs only the pi↔mempalace MCP bridge on top of whatever pi-toolkit set up.

Conventions

Standalone executables in bin/ with #!/usr/bin/env bash shebang, no extension, chmod +x. Must work in non-interactive contexts (agent processes, cron, CI).
Thin wrappers only. Neither tool reimplements the mempalace miner. Both follow the stage-to-cache-then-mine idiom: curate input to ~/.cache/…/<wing>/, then delegate to mempalace mine.
Idempotent + dry-runnable. Every tool supports --dry-run. Second invocation on unchanged input is a no-op (dedup via source_file path, optionally + mtime).
No external Python deps. Stdlib only (sqlite3, json, pathlib). Inline in the bash wrapper via heredoc.
Argument parsing: --help/-h first, then mode flags, then positional args.
Comment sections use # ── Section Name ────── style (matches sibling cli_utils repo).

Adding a new wrapper

Three wrappers live happily as standalone scripts — no shared helper library yet, because each one's stage-to-cache logic differs enough that the common surface is thin (arg parsing + mempalace mine invocation). A fourth wrapper might tip the balance; re-evaluate then. Until then, copy the pattern from mempalace-session (richest example):

Create bin/<name> with #!/usr/bin/env bash + chmod +x.
Implement --help, --dry-run, --repair flags (repair is opt-in; --no-repair kept as deprecated alias).
Stage to ~/.cache/<name>/<wing>/ with deterministic filenames.
Invoke mempalace mine ... (choose --mode convos if input is chat-like).
Do NOT end with mempalace repair unless --repair was explicitly passed. Repair is a destructive in-place HNSW rebuild and must never run on an unattended schedule.
Update README.md with usage + rationale.
Update install.sh? No — bin/* is auto-linked.
Update ARCHITECTURE.md if the wrapper fills a new architectural gap.
Update SKILL.md if agents should know when to invoke it.

Adding a new harness extension

extensions/<harness>/ is the home for bridges — code that lives inside an agent runtime (pi, claude-code, kiro, …) and talks to the mempalace MCP server. Currently only extensions/pi/ exists. If you add a second one (e.g. extensions/claude-code/), follow the same shape:

One directory per harness. Never mix harnesses in one dir.
Bridge-only scope. This toolkit owns the mempalace-side wiring; harness-generic config (keybindings, env loaders, settings templates) belongs in a sibling <harness>-toolkit repo, following the pattern established by pi-toolkit and opencode-toolkit. That boundary is load-bearing for opencode-devbox's slim container path (mempalace opt-out, ~300 MB saved).
A README.md covering: what the bridge does, harness-specific install path, debug knobs, and any gotchas (e.g. the pi Type.Unsafe schema passthrough). May also hold the "Deploying with mempalace" recipe since that straddles the two repos.
Gate install.sh steps on the harness being present. Detect via a well-known path (pi uses ~/.pi/agent/extensions/). Skip silently on machines without that harness. Never force-install.
Symlink the bridge code. mempalace.ts (or equivalent) gets symlinked into the harness's extensions directory so edits flow through git. Back up any pre-existing real file to <path>.bak.YYYYMMDD-HHMMSS before linking.
Probe for the sibling toolkit. After installing the bridge, check whether the harness's own base config is in place (e.g. for pi-toolkit: ~/.pi/agent/keybindings.json symlink, ~/.oh-my-zsh/custom/pi-env.zsh). Warn with a git clone pointer if missing. warn + return 0, never halt.
Mirror in --uninstall. Every symlink this repo creates must have a matching removal step guarded by link_if_into_repo. Do not touch sibling-toolkit-owned files — point the user at <harness>-toolkit/install.sh --uninstall instead.
Update the root README.md — repo-contents list + Ecosystem diagram's "Who owns what" table + Setup section's deploy summary.
Update this file's Structure block to list the new extensions/<harness>/ contents.

See extensions/pi/README.md and the install_pi_extension + check_pi_toolkit functions in install.sh for a worked example.

Testing

Manual only. Integration-shaped:

# Smoke test — does it parse args and list what would happen?
./bin/mempalace-session --help
./bin/mempalace-session --dry-run

# Real test on a single session (safe, deterministic)
./bin/mempalace-session --session ses_<id> --dry-run
./bin/mempalace-session --session ses_<id>     # file into palace
mempalace_search "a phrase from that session"   # verify visibility
./bin/mempalace-session --session ses_<id>     # re-run → should skip

For mempalace-docs, test on a small repo (e.g. this one) first:

./bin/mempalace-docs "$PWD" --dry-run

Gotchas

install.sh is idempotent but interactive — use --yes in non-interactive contexts.
~/.local/bin must be on $PATH. The installer warns if not.
The companion skill lives at ~/.agents/skills/opencode-mempalace-bridge/SKILL.md and is a symlink into this repo. Editing that file edits SKILL.md here. To propagate to Claude Code / Kiro, run agents-sync from cli_utils.
The opencode DB path defaults to ~/.local/share/opencode/opencode.db. Override via $OPENCODE_DB or --db.
The mempalace miner skips symlinks (as of v3.3.3 — miner.py line ~828). That's why the wrappers use cp -p / explicit file writes for staging, not symlinks.
The convos miner dedups on source_file path and mtime (file_already_mined(..., check_mtime=True) in upstream convo_miner.py). A changed/grown transcript is detected, purged, and refiled — it is not silently skipped. What must still be stable across runs is the staged path itself: if a wrapper's staging dir is wiped, dedup has nothing to compare against and mempalace sync will treat the vanished sources as deleted and prune the drawers mined from them. (An earlier version of this file claimed "no mtime check" — that was wrong; verified against convo_miner.py 2026-08.)
The docs miner dedups on source_file path + mtime. That's why staging uses cp -p (preserves mtime).

Colocated skill pattern

This repo owns an agent skill (SKILL.md) that lives alongside the code it documents, rather than in a central skills repo like skillset. The advantages: the skill moves in lockstep with the wrappers it explains, one git clone gets you the full producer-side setup, and retirement (when upstream gaps close) removes skill + code + docs in one commit.

The convention for making this coexist cleanly with sibling tooling:

install.sh creates ~/.agents/skills/<name>/ as a real directory containing a SKILL.md symlink back into this repo. It does not create a dir-symlink, because real dirs are the signal that sibling reconcilers (skillset's deploy-skills.sh, cli_utils's agents-sync.zsh) should leave the dir alone.
install.sh drops a .skill-source marker file at the root of the skill dir:
```
# skill-source: mempalace-toolkit
# repo: <absolute path>
# url: ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git
```
This is a breadcrumb for humans and future tooling — it answers "who owns this skill dir?" at a glance. deploy-skills.sh and agents-sync.zsh don't read it today (their existing logic already handles external dirs correctly) but may surface it in status reports later.
install.sh --uninstall removes the marker (only if it still says mempalace-toolkit) and the now-empty skill dir.

If you add a third colocated skill from a new repo, follow the same convention. The marker format is shared; only the repo name changes.

History

Split out from cli_utils on 2026-04-30. The wrappers originated there but the conceptual fit was weak (cli_utils is interactive shell tools; these are agent memory infrastructure). Some older diary entries and KG facts in the palace reference the original paths.

12 KiB Raw Blame History