ec436ed3adec2c0808efc86eee7621ae2a30cd65
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joakim Persson
|
ec436ed3ad |
docs: reconcile the RFC-001 docs with what is actually deployed
Audit of every doc touching the global-palace rollout against the running
fleet. Each correction below was verified against the filesystem or the host,
not against another doc:
- synlig-primary-runbook: the decommission `rm -rf ~/.mempalace` now carries a
STOP block. That tree holds the fleet palace *and* the only copy of the
bearer token every client authenticates with; the old "empty today" comment
stopped being true when the palace was seeded on 2026-08-14. Adds an ordered
safe decommission, and drops count-based join verification.
- phase-1-exposure-runbook: new S3.8, how to verify a flip actually took --
the procedure that until now existed only in an untracked handover file.
Three claims that fail independently (env var / curl / the palace-path
discriminator) plus an explicit list of checks that produce FALSE POSITIVES:
drawer counts (both sides were seeded from the same palace, and `status`
counts chunks not drawers), write-then-read through the same transport, and
the `mempalace` CLI -- which has no remote support at all, so post-flip it
reads the dead local archive and reports success.
- rfc-001: status Draft -> Phases 0-1 implemented. Records that the join was a
file-level copy, which SIDESTEPPED the S7.6 diary-dedup question rather than
answering it -- so S7.6 remains a hard blocker for the second machine, which
is the one that will actually exercise merge semantics.
- ARCHITECTURE, SKILL, contrib/README, extensions/pi/README all claimed pi
feeds the palace automatically, unconditionally. That is gated on
mempalace-toolkit >=
|
||
|
Joakim Persson
|
2293f1c89b |
docs: the silent-skip fix is base-image-gated, so today's fleet still skips silently
pi-devbox cbd7cf5 makes the remote-palace-without-inbox skip announce itself, but entrypoint-user.sh is COPY'd in Dockerfile.base, so the fix only reaches a container after a base rebuild. Every image running today still skips silently -- recording both behaviours with the cut-off, so the table stays true for whichever image a container is actually on, rather than describing a fix that has not shipped yet. |
||
|
Joakim Persson
|
2e73a9eae8 |
docs: REMOTE_URL includes /mcp, and the entrypoint skips silently where the feeder exits 1
Two details needed before the first client flip. 1. MEMPALACE_REMOTE_URL is the full endpoint including /mcp with no trailing slash. The feeder POSTs to it verbatim (bin/mempalace-pi-session:718), and the server matches `path != "/mcp"` exactly, so a base URL or a trailing slash both 404 -- and a 404 here looks like a routing/proxy fault, not a config typo, which is a bad hour to spend. Matches the existing pi-devbox examples. For this fleet: https://mempalace.jordbo.se/mcp 2. The 3.7 trap has TWO symptoms, not one, and I had only documented the loud one. Direct runs / session-end / cron exit 1 with a clear error (:298-300). But pi-devbox's entrypoint-user.sh:134 checks the same condition and skips *quietly* -- and the skip happens before the subshell that writes ~/.pi/agent/mempalace-catchup.log, so there is not even an empty log to find. A container flipped with REMOTE_URL but no SSH_TARGET therefore contributes nothing to the palace and leaves no artifact explaining why. The skip is correct in itself (there is genuinely no inbox to ship to) but it is indistinguishable from a healthy run with nothing to do. Documented with two commands that tell those apart after a flip. |
||
|
Joakim Persson
|
4cb70ce3e1 |
docs: record the 302 auth-redirect fingerprint and the server's exact HTTP surface
Phase 1 exposure now verified end to end (2026-08-12): /healthz -> ok and an unauthenticated POST /mcp -> 401, both from a client container and from the primary itself. 3.4 and 3.6 marked passed. Two findings from the failure in between, worth more than a checkbox: 1. Leaving Pangolin's resource authentication on does NOT surface as 401 or 403. It is a 302 with `location: https://pangolin.jordbo.se/auth/resource/<uuid> ?redirect=...`, sent *before* the request is proxied -- so the palace never sees it and its journal stays silent, `curl -s` prints an empty body, and an MCP client gets non-JSON. That is 1.1's "Pangolin's HTTP auth is browser-shaped; the clients are not" arriving as a concrete symptom rather than an argument. Now recorded with the exact header shape and the two curl flags that reveal it (-D-, -w '%{redirect_url}'), plus the reading: a 302 is good news, because DNS, TLS and routing all worked and only auth intervened. 2. Read the server's routing to settle whether path-scoped proxy rules are sufficient. The entire HTTP surface is two exact paths (mcp_server.py:5299- 5318): GET /healthz (no token, Host/Origin gated) and POST /mcp (Bearer, compare_digest on the exact string). Everything else is a 404 from the palace itself. So path-scoped rules are tighter than a host-wide proxy and lose nothing. Two client-facing consequences: /mcp is matched exactly, so a trailing slash 404s -- configure clients without one; and there is no GET /mcp, no SSE, no session id, no DELETE, so this is plain JSON-RPC over POST, not MCP streamable-HTTP. A strict client opening with a GET handshake sees 404. Also means the verify step needs no initialize and no Accept: text/event-stream, which the old snippet left ambiguous. Also: run the 200 and 401 from two different networks, not one -- passing from only the primary leaves split-horizon DNS untested. |
||
|
Joakim Persson
|
08e344b047 |
docs: proxy target is http:// not https://; loopback probe refuses, it does not 403
Both corrections come from the first real Phase 1 start on synlig (2026-08-12).
1. The Pangolin resource target was documented as bare `172.17.0.1:8765` with no
scheme, and the obvious guess from that is `https://` -- which cannot work.
`contrib/systemd/mempalace-serve.service` runs `serve --host 172.17.0.1
--port 8765` with no --tls-cert, so the primary speaks plaintext HTTP; TLS
terminates at Pangolin, which is the whole point of the RFC 6.2 decision.
Point a proxy at https:// and it attempts a TLS handshake against a plaintext
listener: 502 from outside, while `curl 172.17.0.1:8765/healthz` on the box
still says ok -- a confusing pair of symptoms. Now spelled `http://` with the
failure mode named, in the runbook and in the unit's comments.
2. `curl -s 127.0.0.1:8765/healthz` was documented as "expect 403". Wrong: the
real run returns empty. With a docker0-only bind nothing is listening on
loopback, so the connection is refused at TCP level before any header is sent
(%{http_code} -> 000, exit 7). The 403 is the *loopback-bind* case verified
2026-08-10 -- server on 127.0.0.1 answering a proxy-forwarded foreign Host.
Two distinct behaviours had been collapsed into one expectation in three
places (both runbooks and the unit).
Worth stating why the correction matters rather than just fixing it: refusal
is the *stronger* signal. A 403 proves only that a request was rejected; a
refused connection proves the loopback and LAN surface is not listening at
all. Someone who expected 403, saw silence, and "fixed" it by rebinding to
0.0.0.0 would have converted a correct configuration into an exposed one.
The docs now also say what to do if it hangs, or if ss shows 0.0.0.0:8765.
|
||
|
Joakim Persson
|
00a95d1a2f |
docs: why the tunnel and the feeder's SSH path are not redundant; newt done
Asked "why do we need Pangolin if you proposed rsync/ssh?", and the runbook did
not actually answer it -- it stated both were needed without saying why neither
substitutes. New section 1.3:
- Pangolin/HTTPS carries the MCP tool surface (search, add_drawer,
diary_write, kg_*) -- every live tool call, from any MCP client.
- SSH/rsync carries transcript *files* only, because mempalace_mine expands
its source path server-side, so the server can only mine its own disk.
HTTPS alone is a palace you can query but cannot feed; SSH alone is files with
no query API. The rsync is not a transport preference, it is a workaround for
where `mine` resolves paths.
Records honestly that `ssh -L 8765:172.17.0.1:8765 synlig` *would* replace the
tunnel for MCP, and why we don't: synlig dials out (reaching for a dial-out
tunnel is itself the evidence inbound was unavailable), MCP clients want a
durable URL rather than a per-session forward, and the forward must be up on
every device before every session.
And the design's weak point, stated instead of glossed: the rsync runs
client -> synlig, so mining needs synlig's SSH reachable *from the client*. Were
that true everywhere, no tunnel would be needed for MCP either. Honest
expectation after Phase 1 is therefore: query/write from anywhere, mine only
from devices that can reach synlig's SSH. Section 4 now carries the upstream ask
that would close the gap -- have the feeder send content over MCP (add_drawer /
diary_write, which it already calls) instead of asking the server to mine a path
it must first rsync there.
New section 3.7, a live trap for the imminent client flip:
MEMPALACE_REMOTE_URL on its own does not degrade to local feeding, it *stops*
feeding. auto mode switches to remote as soon as the URL is set (:286) and
remote mode then exits 1 without MEMPALACE_PI_SSH_TARGET (:298-300), before
anything is staged or filed -- so a cron feeder just starts failing, and the
loudest symptom is silence. Two safe orders given: set all three variables in
one edit, or set URL+token and pin --mode local until the SSH target exists.
newt is installed on synlig and connected to Pangolin (done 2026-08-12), marked
here and in the synlig runbook's item 2; the blocker is now item 3, the one
sudo. Added the follow-up that "connected to Pangolin" only proves newt reached
nyvaken -- reaching the *palace* is a separate claim that fails independently,
so probe 172.17.0.1:8765/healthz from inside newt's namespace.
All seven code citations verified against the source at commit time.
|
||
|
Joakim Persson
|
29e660e18f |
feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item. |
||
|
Joakim Persson
|
3626946013 |
Phase 0 on synlig: install, palace layout, verified Host/Origin policy
synlig is greenfield — no mempalace, no ~/.mempalace at all — so Phase 0 became "provision correctly from birth" rather than "migrate carefully". Nothing is serving; no client config was touched. Done: - mempalace 3.6.0 installed via uv, pinned to the fleet version (the id recipes and idempotency probes this RFC leans on are version-specific). - Embedder pre-warmed. This was the real unknown: the first embed pulls a 79.3 MB ONNX model from the chroma CDN, and an egress-filtered work VM would have failed at the worst moment — the first client write. Pulled at ~20 MB/s, no proxy interference. Done in a throwaway palace so the real one never saw it. - Palace at the stock default ~/.mempalace/palace, so no config file and no MEMPALACE_PALACE_PATH is needed on synlig at all. Two corrections to the RFC, both from provisioning rather than reading: - §7.1 was understated. DEFAULT_PALACE_PATH (~/.mempalace/palace) and DEFAULT_KG_PATH (~/.mempalace/knowledge_graph.sqlite3) already differ with stock defaults, so the KG split is out-of-the-box behaviour, not a consequence of a custom --palace, and it is permanent rather than one-time: serve always passes --palace, any CLI call without it uses the HOME path. A one-time mv does not fix that, it only picks which of the two files gets populated. Fixed instead by converging both rules on one inode via relative symlinks, and verified the load-bearing assumption: a dangling symlink is created on connect, -wal/-shm land next to the target (so the palace dir stays a self-contained backup unit, which is the part that mattered), cross-path read works, same inode. hallways.json deliberately left alone — already palace-derived, HOME path is a warning-only probe. - §6.2 upgraded from "test early" to verified: 11/11 as predicted. The headline is that the safe-sounding reflex is the failure mode — loopback bind + proxy forwarding a public Host is 403, non-loopback is 200. Also confirmed Origin is never relaxed on either bind, and /healthz is Host/Origin-gated but token-free, so it works as the tunnel probe. Recommends binding the docker0 gateway over 0.0.0.0: non-loopback so the pin relaxes, but reachable only from the host and its containers. New docs/synlig-primary-runbook.md carries the discovered facts about the box, the evidence tables, an explicit "deliberately not done" list, and tomorrow's ordered steps. New contrib/systemd/mempalace-serve.service carries the bind rationale inline so nobody "fixes" it back to loopback; staged on synlig with a .staged suffix so systemd cannot pick it up by accident. Flagged for tomorrow: synlig has no newt/tunnel client (docker ps shows only the Gitea runner and digikam), so Pangolin on nyvaken cannot reach it until one is added — easy to miss, because Pangolin will look healthy from its own side. |
||
|
Joakim Persson
|
7e51055c96 |
rfc-001: join protocol, diary non-idempotency, deployment decisions
Second planning round. Three findings from verification, and the decisions that were blocking phasing. Verified in mempalace 3.6.0 and written up: - NEW §7.6 — diary_write has no idempotency guard at all. The id embeds datetime.now() at microsecond resolution and the write is a bare col.add (mcp_server.py:3546) with no col.get probe, in direct contrast to add_drawer's probe 900 lines earlier (:2593-2600). So §3's "log the intent, replay the intent" is false for diaries — replay duplicates. This lands on the critical path because diaries are `replicated` and are exactly what a join replays. - NEW §4.4 — join/bootstrap protocol, which the RFC simply lacked. Not "seed the primary from one palace": every container joins the same way, repeatedly, so a join is idempotent replay and the only question per record type is what dedupes it. Drawers and open KG facts need zero client bookkeeping; closed KG facts and diaries need client-side keys. Join state belongs in the palace dir, not the container (~/.mempalace is not preserved by default), which also makes two containers sharing one host's palace the easy case rather than a double-upload hazard. - §3.2 — the add_triple guard is scoped to `valid_to IS NULL`, so closed historical facts are unguarded. §3 read as unconditionally idempotent. - §7.3 — drawer/diary metadata is exhaustive: no session, PID or conversation field. Two concurrent pi sessions are indistinguishable, and pi-vs-opencode is only accidentally distinguishable because the pi extension never sets added_by (it sets identity for diaries only, mempalace.ts:758). Corrects the §7.3.3 bullet that claimed multi-harness attribution was already solved — the field is the right home, but nothing populates it. Design rule stated: device + agent, never session; provenance granularity equals token granularity. - §6.2 — Host-pinning is coupled to the bind address (enforce_host_pin = _http_is_loopback(host), :5367). The trap is the RFC's own "bind loopback" reflex: behind a proxy that forwards a public Host, a loopback bind 403s, while a non-loopback bind deliberately relaxes the pin. The Origin check is never relaxed. This replaces the warning I first wrote, which had it backwards. - §9.3 — largely resolved: the last-chunk-only probe is deliberate (batched upsert is all-or-nothing, :2586-2592). Only a crash mid-upsert remains untested. - §4.1 — fix shape for opencode's stale-config asymmetry: make the mcp.mempalace subtree env-authoritative, with a fingerprint so hand-edits still win. pi-devbox/entrypoint-user.sh:131-162 already has the jq deep-merge pattern to port; no OPENCODE_CONFIG* layer exists upstream, so the real file must be written. Now Phase 1.5. Decisions (new §8.1): primary = synlig; TLS at Pangolin; single shared token for now (so origin_device stays advisory); diaries replicated. Work/personal is per-wing, not per-device — two primaries split along machine lines is rejected, because pi-devbox/opencode-devbox are simultaneously work and home projects and the device where work happened cannot classify the project. MEMPALACE_REMOTE_URL stays scalar so multi-store remains additive later (new §9.7 keeps the placement question open). |
||
|
Joakim Persson
|
fdcd5871de |
rfc-001: resolve open questions 1 and 6 — both permissive
Recon before planning the implementation, and both blockers dissolved. Q1, does opencode support remote MCP: yes. Its published schema defines McpRemoteConfig (type/url/headers/oauth) as a sibling of McpLocalConfig, with headers as a free string→string map. The RFC had been treating "every sampled config in myconfigs is type:local" as evidence about the schema when it was only evidence about deployments. Consequence: the edge proxy is not the only option for opencode, so nothing in the phasing hangs on this. What opencode still lacks is offline/local-first, which is the honest Phase 2 argument. Q6, how opencode-devbox learns the opt-in: it already does. generate-config.py, run from entrypoint-user.sh:117, registers mempalace as remote+bearer when MEMPALACE_REMOTE_URL is set and local stdio otherwise, and its comment says it deliberately mirrors the mempalace.ts env contract. The image also ships mempalace by default, so R5's premise was wrong and is corrected. Phase 2 there is a third branch in an existing script, not a new mechanism. One new constraint found while confirming this, and it is the sharpest edge in the whole opt-in story: generate-config.py never overwrites an existing config and ~/.config/opencode is a named volume, so flipping the .env on a container that already has a config is a no-op — it only writes an opencode.jsonc.proposed sidecar. pi re-reads env every start; opencode does not. Documented in §4.1 and §9.6, and it applies symmetrically to R6 reversibility. |
||
|
Joakim Persson
|
052dbb8038 |
docs(rfc-001): provenance belongs to the sync boundary, not the agent
Reverses the previous §7.3 on review. It said "stamp added_by everywhere, now, because it cannot be backfilled" and was about to become a skill instruction telling agents to do it. Both halves were wrong. Wrong on ownership: provenance answers "which device asserted this?", so only a party that can verify the answer should write it. An agent must shell out to read env, can forget, and will improvise when the values are absent — the worst possible stamper, and its claim is unverifiable by anyone. Under the §4 design every write reaches the primary over an authenticated channel, including offline ones at outbox-flush time, so the primary can stamp the complete set with no client cooperation. Provenance is a property of the sync channel, not of the record's author. §7.3.2 adds the trust ladder; edge-side stamping is demoted to an advisory interim, because 3.6.0's serve takes a single shared bearer token (mcp_server.py:5291-5293) and the package has zero device/origin concept, so authoritative stamping needs the per-device credentials of §6 — Phase 4, not Phase 0. Wrong on backfill: a solitary devbox is a single-origin store by definition, so origin is a property of the whole palace and can be assigned wholesale at import (one --origin-device flag) at the moment it stops being solitary. Bulk attribution is strictly more reliable than per-record stamping since it cannot be partially applied. Per-record provenance is only needed where origins interleave, which is only the primary. Solitary containers therefore stamp nothing and lose nothing — more R1-compliant than the previous draft, which quietly asked users who had opted out to carry metadata for the feature. Multi-harness-on-one-host stays solved by added_by = agent name. Keeps the verified mechanics (fixed metadata schema, argument whitelisting at mcp_server.py:4777 silently dropping unknown fields, the diary agent_name → wing_pi@host trap, kg_add having no provenance slot, added_by absent from search results) and the identity findings (a container cannot discover its host's identity; hostnames are neither unique nor stable; rename splits one device's history in two). §7.3.5 keeps the fail-closed rule for whichever component does stamp. Phase 0 drops its provenance item accordingly, and §6.2 now specifies server-side stamping from the authenticated credential. Also flags in-document that these notes are a poisoning vector: a future agent reading them out of the palace must not conclude it should hand-stamp. |
||
 joakimp
|
661ee20b39 |
docs(rfc-001): require solitary-first operation, opt-in centralization
Adds §1.1 (R1–R6) and §1.2 as a hard constraint on the design rather than a preference. A shared palace is valuable only for the multi-machine / multi-container pattern; for most users of the published pi-devbox and opencode-devbox images it is useless overhead. Evidence: all three sampled opencode-devbox deployments contain zero mempalace references. Requirements: solitary operation stays the default and stays byte-identical (no extra process, no outbox, no network calls); opt-in via docker-compose.yml + .env only, never an image rebuild; credentials only in .env, never in compose, the image, a command line, or a log; no new required services (the primary stays a separate standalone compose project); degrade-not-fail where mempalace is absent; fully reversible. §1.2 extends the convention pi-devbox already ships (.env.example:12-23, "local by default" with commented MEMPALACE_REMOTE_URL/_TOKEN) into a three-state ladder — local stdio (default, unchanged) / direct remote (unchanged) / edge (new, MEMPALACE_EDGE=1) — instead of inventing a new mechanism. Notes that the devbox-palace volume coupling reverses under edge mode: the local palace holds the outbox, so persisting it becomes required rather than irrelevant. Consequence recorded in §4.1: mempalace-edge must be selected at registration time, not left always-in-path to decide by env at runtime, since that would insert a process and a failure mode into every solitary user's setup. pi branches in createClient(); opencode needs its static MCP JSON templated at container start, which is new open question §9.6. Also: adds an R1 acceptance test, marks pi-devbox/.env.example:21 as stale (advertises mempalace-mcp --transport http rather than mempalace serve --token/--tls-cert), and extends the evidence index. |
||
|
joakimp
|
35b1e3d81d |
docs: add RFC 001 — global palace with local fallback (mempalace-edge)
Design for moving from one MemPalace per machine per harness to a single primary palace with per-machine offline fallback, plus the source-verified archaeology behind it. Key decisions recorded: - Sync operations (MCP tool calls), not databases. Embeddings are computed client-side and are not portable across architectures; the KG `triples` table has no UNIQUE(subject,predicate,object,valid_from) and its ids embed datetime.now(), so row copies duplicate facts. Replaying tool calls is idempotent where it matters. - Implement as `mempalace-edge`, a local stdio MCP proxy (child mempalace-mcp + HTTPS to the primary + outbox.sqlite), not as per-harness patches. Needs zero mempalace internals, so it serves pi, opencode and the CLI alike and survives mempalace upgrades. - Merged reads (query both, re-sort, dedupe by drawer id) give fleet-wide recall without replication — which is why Phase 3 (pull replication) is deferred: "own writes plus whatever it can reach" is good enough. - Per-wing replication policy: curated/content-addressed wings replicate, mined code/docs stay local (derived, re-mineable, path-dependent ids). Also documents two silently destructive footguns to avoid during rollout: `--palace` vs MEMPALACE_PALACE_PATH (the KG follows the flag only, so `mempalace serve` can start with a silently empty knowledge graph), and `mempalace sync`, which is gitignore-aware drawer deletion rather than replication and would wipe fleet memory when run from a host lacking the repos. Notes that mempalace 3.6.0's `serve` already ships token auth + TLS, making the "unauthenticated, front it with a proxy" notes elsewhere stale. Includes an evidence index mapping each claim to file:line in mempalace 3.6.0, and four upstream candidates (origin_host provenance, per-wing ACL, mempalace_kg_supersede missing from service.py WRITE_TOOLS, and a sync --refuse-shared guard). |