b609cf5a69f47890bc33cc39fd6c72451b0cee5e
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joakim Persson
|
f60cf9c732 |
feat(census): RFC 002 Phase A — read-only join census, and three RFC corrections it found
bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.
The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):
(a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
_DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
but _DELIM is dead code and _delimited_sha256 builds
"".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.
(b) id_recipe is NOT a mined-only marker. It looked like a clean
discriminator (same 14,586 count as source_file) but the server stamps
'v3' on content ids too, so classifying on `source_file OR id_recipe`
swallowed all 60 agent-authored drawers into MINED -- the dangerous
direction, since Phase C would try to re-mine drawers that have no source
file and silently drop them. Discriminator is a TRUTHY source_file (the
writer stores "" rather than omitting the key), cross-checked against the
miner-only keys source_mtime / normalize_version; disagreement is now a
first-class warning.
(c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
their own id, because update_drawer preserves the id while rewriting and
re-chunking. So "recompute the content id and skip if present" -- the
strategy this RFC specified for regime A -- misses every drifted drawer
and duplicates it. Phase C must key on the STORED id. Flagged as
edited_since_filing in the manifest so Phase C can be tested on them.
Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.
One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.
|
||
|
Joakim Persson
|
29e660e18f |
feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item. |
||
 joakimp
|
35b1e3d81d |
docs: add RFC 001 — global palace with local fallback (mempalace-edge)
Design for moving from one MemPalace per machine per harness to a single primary palace with per-machine offline fallback, plus the source-verified archaeology behind it. Key decisions recorded: - Sync operations (MCP tool calls), not databases. Embeddings are computed client-side and are not portable across architectures; the KG `triples` table has no UNIQUE(subject,predicate,object,valid_from) and its ids embed datetime.now(), so row copies duplicate facts. Replaying tool calls is idempotent where it matters. - Implement as `mempalace-edge`, a local stdio MCP proxy (child mempalace-mcp + HTTPS to the primary + outbox.sqlite), not as per-harness patches. Needs zero mempalace internals, so it serves pi, opencode and the CLI alike and survives mempalace upgrades. - Merged reads (query both, re-sort, dedupe by drawer id) give fleet-wide recall without replication — which is why Phase 3 (pull replication) is deferred: "own writes plus whatever it can reach" is good enough. - Per-wing replication policy: curated/content-addressed wings replicate, mined code/docs stay local (derived, re-mineable, path-dependent ids). Also documents two silently destructive footguns to avoid during rollout: `--palace` vs MEMPALACE_PALACE_PATH (the KG follows the flag only, so `mempalace serve` can start with a silently empty knowledge graph), and `mempalace sync`, which is gitignore-aware drawer deletion rather than replication and would wipe fleet memory when run from a host lacking the repos. Notes that mempalace 3.6.0's `serve` already ships token auth + TLS, making the "unauthenticated, front it with a proxy" notes elsewhere stale. Includes an evidence index mapping each claim to file:line in mempalace 3.6.0, and four upstream candidates (origin_host provenance, per-wing ACL, mempalace_kg_supersede missing from service.py WRITE_TOOLS, and a sync --refuse-shared guard). |
||
|
joakimp
|
90e70fff61 |
docs: add Ecosystem diagram to README + update harness-extension conventions post-split
Two small doc updates consolidating the session's architectural arc:
1. README.md gets a new 'Ecosystem' section (after the contents list,
before 'Why this exists') showing the five-repo composition:
myconfigs -> opencode-toolkit + pi-toolkit -> mempalace-toolkit
Plus an ownership table clarifying which scope lives where. The
diagram makes the opt-out pattern visible \u2014 opencode-devbox's slim
container path skips mempalace-toolkit and still gets a functional
stack.
2. AGENTS.md 'Adding a new harness extension' section was still written
for the pre-split extensions/pi/ which had keybindings.json +
settings.example.json + pi-env.zsh. Rewrote to reflect:
- Bridge-only scope (harness-generic config goes in <harness>-toolkit).
- 'Probe for the sibling toolkit' step replaces the old symlink-keybindings
and template-settings steps.
- Worked example now points at install_pi_extension + check_pi_toolkit
rather than the four functions that moved out.
- Explicitly names the pattern: opencode-toolkit + pi-toolkit as
the two existing examples of the sibling-toolkit convention.
|
||
|
joakimp
|
16915f0e55 |
refactor: split pi-generic config into pi-toolkit repo
Parallel to the opencode-toolkit split earlier today. Pi's own config (keybindings, shell env loader, settings template) moves to a new sibling repo so opencode-devbox's mempalace opt-out can build slim containers that include pi without dragging in chromadb + embedding models (~300 MB). What moved to pi-toolkit (https://gitea.jordbo.se/joakimp/pi-toolkit): - extensions/pi/keybindings.json (mosh/tmux newline fix) - extensions/pi/pi-env.zsh (sources ~/.config/pi/.env) - extensions/pi/settings.example.json (Bedrock template) - install.sh::install_pi_keybindings (symlink step) - install.sh::install_pi_env_loader (cp step + bash fallback) - install.sh::check_pi_settings (probe) - install.sh::check_aws_env (probe) What stays here (this is the pi\u2194mempalace bridge, mempalace-side): - extensions/pi/mempalace.ts (the MCP extension) - install.sh::install_pi_extension (symlink step) - NEW: install.sh::check_pi_toolkit (probe: warns if pi is installed but pi-toolkit's artifacts are missing, with git-clone pointer) install.sh shrank from 520 to 403 lines. Uninstall mirror correctly does NOT touch pi-toolkit-owned files (explicit comment). Docs updated: - extensions/pi/README.md: rewritten as 'pi\u2194MemPalace MCP bridge', recipe becomes 'Deploying pi with mempalace' (pi-toolkit step 3, this repo step 5). - AGENTS.md: Structure block + 'What install.sh does' section reflect the narrower scope and list the four things that moved out. - README.md: repo-contents line + Setup section's deploy summary. Verified on tor-ms22: full install\u2192uninstall\u2192reinstall lifecycle clean. After mempalace-toolkit uninstall, pi-toolkit artifacts (~/.pi/agent/keybindings.json, ~/.oh-my-zsh/custom/pi-env.zsh) remain intact \u2014 correctly untouched. check_pi_toolkit probe fires green when both exist. |
||
|
joakimp
|
118bd20fec |
feat(extensions/pi): ship pi-env.zsh shell loader
The loader that sources ~/.config/pi/.env into every shell was only
living in the myconfigs tor-ms22 backup \u2014 a fresh machine had nowhere
to get it from except copying by hand. Now canonical here.
- extensions/pi/pi-env.zsh: 20-line POSIX-compatible loader
(set -a; source ~/.config/pi/.env; set +a). Works in bash and zsh.
- install.sh install_pi_env_loader:
* oh-my-zsh detected (~/.oh-my-zsh/custom/ exists)
\u2192 cp into that dir (NOT symlink \u2014 that dir is typically part of
a dotfiles backup, and a symlink to mempalace-toolkit would
break when restored on another host).
\u2192 Idempotent: if target content matches repo, says 'already
installed'. If it differs, leaves user edits alone and points
at diff for manual reconcile.
* No oh-my-zsh \u2192 prints source-this-line snippet for ~/.zshrc or
~/.bashrc (derived from $SHELL). Does NOT auto-edit rc files.
- install.sh uninstall: only removes the copy if content still matches
repo. Local edits preserved.
- Docs:
* extensions/pi/README.md Environment setup section rewritten with
both install paths, step 5 of deploy recipe updated.
* AGENTS.md Structure block lists pi-env.zsh.
* Root README repo-contents line mentions it.
Verified on tor-ms22: install fresh \u2192 uninstall (content match \u2192 remove)
\u2192 reinstall \u2192 zsh -ic loads AWS vars correctly. Also tested bash fallback
path via HOME=/tmp/fake-home SHELL=/bin/bash \u2014 prints right .bashrc snippet.
|
||
|
joakimp
|
5d8f523cb3 |
docs(AGENTS): unstale wrapper count + new 'Adding a new harness extension' section
Two stale lines from the 2-wrapper era (we have three now), plus a missing convention doc for future harness extensions: - 'What this is': two wrappers \u2192 three + extensions/ tree - 'Adding a new wrapper': drop the 'third wrapper triggers helper lib' claim since three coexist fine as standalone scripts; keep the threshold at four, re-evaluate then. - New 'Adding a new harness extension' section codifies the extensions/ pattern so a future claude-code or kiro bridge follows the same shape (per-harness dir, symlink-vs-template rules, gated install steps, probe-don't-halt, uninstall mirror, README + Structure updates). |
||
|
joakimp
|
79e0692dac |
docs: update AGENTS.md structure + ARCHITECTURE.md see-also for pi bring-up
AGENTS.md Structure block was stale (listed only 2 bin/ wrappers, no extensions/, no contrib/). Added full tree + a new 'What install.sh does' section enumerating all steps, gates, and probes so maintainers see at a glance what the installer touches. ARCHITECTURE.md is scoped to the producer side (feeding the palace); pi extension is consumer side, so out of scope for the main body. Added a pointer in the See also section so readers can find extensions/pi/README.md. |
||
|
joakimp
|
6352373a1f |
fix(feeders): make post-mine repair opt-in, not default
The three feeder wrappers (mempalace-docs, mempalace-pi-session,
mempalace-session) unconditionally ran 'mempalace repair --yes' after
mining, controllable only via --no-repair opt-out. The contrib launchd
and systemd templates did not pass --no-repair, so every scheduled tick
invoked the destructive in-place HNSW rebuild.
This has bitten us twice:
- 2026-05-04 09:08: a kickstart triggered repair while an MCP
subprocess held the DB open; the live collection was wiped (0
drawers) and had to be restored from the palace.backup snapshot.
- 2026-05-05 10:00: post-mine repair crashed mid-rebuild with
'NotFoundError: Collection [<uuid>] does not exist' - chromadb's
rebuild recreated the collection under a new UUID while the code
still held the old handle. Live DB survived only by luck (crash
hit before the swap).
Fix: flip the default.
- New flag: --repair (opt-in). Prints a warning and sleeps 3s before
invoking 'mempalace repair --yes'.
- --no-repair is retained as a deprecated no-op alias for backward
compatibility with any scripts/units still passing it.
- Default behavior: no repair. Routine ChromaDB add() keeps HNSW
consistent; repair is a recovery op, not a maintenance tick.
Docs updated to match: README, SKILL, ARCHITECTURE, AGENTS,
contrib/README. Scheduling guidance now explicitly warns against
enabling --repair on cron/launchd/systemd-timer runs.
|
||
|
Joakim Persson
|
e49d9285c4 |
install.sh: drop .skill-source marker in deployed skill dir
The skill directory at ~/.agents/skills/opencode-mempalace-bridge/ is a real dir containing a single SKILL.md symlink back into this repo — the 'colocated skill' pattern. Sibling reconcilers (skillset's deploy-skills.sh, cli_utils's agents-sync.zsh) already handle external dirs correctly via their existing 'leave real dirs alone' policies, but a machine-readable marker makes ownership explicit: # skill-source: mempalace-toolkit # repo: <absolute path> # url: ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git The marker is the convention for any external repo that wants to ship a colocated skill. The name is generic (.skill-source, not .managed-by-mempalace-toolkit) so a second colocated skill from a different repo can reuse the same file name; the first line identifies the owner. --uninstall now also removes the marker (only if it still says mempalace-toolkit) and the now-empty skill dir. AGENTS.md + README.md describe the pattern and point at sibling docs in cli_utils/AGENTS-SYNC.md and skillset/README.md that mirror the convention. |
||
|
Joakim Persson
|
954c3f2ebb |
Initial commit — split out from cli_utils
Producer-side MemPalace tooling: two bash wrappers that bridge opencode session history and project documentation into the palace. Originally developed in cli_utils (2026-04-28); split into its own repo on 2026-04-30 because the conceptual fit was weak — cli_utils is interactive shell tooling, while this is agent memory infrastructure with its own architecture, dependency surface, and growth trajectory. Contents: - bin/mempalace-docs — docs-only mining wrapper (originally a2ddcc9 in cli_utils), bridges the gap until MemPalace PR #1213 (exclude_patterns) merges upstream. - bin/mempalace-session — opencode → palace session bridge (originally dacca0e in cli_utils). Reads ~/.local/share/opencode/opencode.db, exports each session to Claude Code JSONL, mines via 'mempalace mine --mode convos'. Bridges the gap until opencode session-stopping hooks + an opencode harness in hooks_cli.py land upstream. - ARCHITECTURE.md — canonical spec: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap. Originally a4cf314 in cli_utils. - SKILL.md — companion agent skill (producer side). Pairs with the consumer-side mempalace skill. Symlinked into ~/.agents/skills/opencode-mempalace-bridge/ by install.sh. - install.sh — idempotent installer, also handles --uninstall. - AGENTS.md — repo conventions. History of the individual files is not preserved in this split; see cli_utils (gitea.jordbo.se/joakimp/cli_utils) commits a2ddcc9, dacca0e, and a4cf314 for the original authorship context. |