synlig is greenfield — no mempalace, no ~/.mempalace at all — so Phase 0
became "provision correctly from birth" rather than "migrate carefully".
Nothing is serving; no client config was touched.
Done:
- mempalace 3.6.0 installed via uv, pinned to the fleet version (the id
recipes and idempotency probes this RFC leans on are version-specific).
- Embedder pre-warmed. This was the real unknown: the first embed pulls a
79.3 MB ONNX model from the chroma CDN, and an egress-filtered work VM
would have failed at the worst moment — the first client write. Pulled
at ~20 MB/s, no proxy interference. Done in a throwaway palace so the
real one never saw it.
- Palace at the stock default ~/.mempalace/palace, so no config file and
no MEMPALACE_PALACE_PATH is needed on synlig at all.
Two corrections to the RFC, both from provisioning rather than reading:
- §7.1 was understated. DEFAULT_PALACE_PATH (~/.mempalace/palace) and
DEFAULT_KG_PATH (~/.mempalace/knowledge_graph.sqlite3) already differ
with stock defaults, so the KG split is out-of-the-box behaviour, not a
consequence of a custom --palace, and it is permanent rather than
one-time: serve always passes --palace, any CLI call without it uses
the HOME path. A one-time mv does not fix that, it only picks which of
the two files gets populated. Fixed instead by converging both rules on
one inode via relative symlinks, and verified the load-bearing
assumption: a dangling symlink is created on connect, -wal/-shm land
next to the target (so the palace dir stays a self-contained backup
unit, which is the part that mattered), cross-path read works, same
inode. hallways.json deliberately left alone — already palace-derived,
HOME path is a warning-only probe.
- §6.2 upgraded from "test early" to verified: 11/11 as predicted. The
headline is that the safe-sounding reflex is the failure mode — loopback
bind + proxy forwarding a public Host is 403, non-loopback is 200. Also
confirmed Origin is never relaxed on either bind, and /healthz is
Host/Origin-gated but token-free, so it works as the tunnel probe.
Recommends binding the docker0 gateway over 0.0.0.0: non-loopback so
the pin relaxes, but reachable only from the host and its containers.
New docs/synlig-primary-runbook.md carries the discovered facts about the
box, the evidence tables, an explicit "deliberately not done" list, and
tomorrow's ordered steps. New contrib/systemd/mempalace-serve.service
carries the bind rationale inline so nobody "fixes" it back to loopback;
staged on synlig with a .staged suffix so systemd cannot pick it up by
accident.
Flagged for tomorrow: synlig has no newt/tunnel client (docker ps shows
only the Gitea runner and digikam), so Pangolin on nyvaken cannot reach
it until one is added — easy to miss, because Pangolin will look healthy
from its own side.
The three feeder wrappers (mempalace-docs, mempalace-pi-session,
mempalace-session) unconditionally ran 'mempalace repair --yes' after
mining, controllable only via --no-repair opt-out. The contrib launchd
and systemd templates did not pass --no-repair, so every scheduled tick
invoked the destructive in-place HNSW rebuild.
This has bitten us twice:
- 2026-05-04 09:08: a kickstart triggered repair while an MCP
subprocess held the DB open; the live collection was wiped (0
drawers) and had to be restored from the palace.backup snapshot.
- 2026-05-05 10:00: post-mine repair crashed mid-rebuild with
'NotFoundError: Collection [<uuid>] does not exist' - chromadb's
rebuild recreated the collection under a new UUID while the code
still held the old handle. Live DB survived only by luck (crash
hit before the swap).
Fix: flip the default.
- New flag: --repair (opt-in). Prints a warning and sleeps 3s before
invoking 'mempalace repair --yes'.
- --no-repair is retained as a deprecated no-op alias for backward
compatibility with any scripts/units still passing it.
- Default behavior: no repair. Routine ChromaDB add() keeps HNSW
consistent; repair is a recovery op, not a maintenance tick.
Docs updated to match: README, SKILL, ARCHITECTURE, AGENTS,
contrib/README. Scheduling guidance now explicitly warns against
enabling --repair on cron/launchd/systemd-timer runs.
Drop-in equivalents of the opencode templates for each scheduler
mechanism:
systemd/mempalace-pi-session.{service,timer}
launchd/se.jordbo.mempalace-pi-session.plist
cron/mempalace-pi-session.cron
Schedule is staggered from the opencode jobs (Mon 03:00 -> Tue 03:00)
so machines running both don't race each other on the post-mine HNSW
repair step. Service unit uses ConditionPathExists=%h/.pi/agent/sessions
to no-op silently on machines that haven't used pi, matching the
opencode template's guard on ~/.local/share/opencode/opencode.db.
contrib/README.md grows a 'Templates at a glance' table so the set is
discoverable without reading the whole doc.
Minimal Debian/Ubuntu hosts (and most base container images) don't
ship cron by default. `crontab: command not found` is the first
thing a user hits if they try the cron path without installing it.
Previous caveats block covered semantics (no Persistent, mail-drop
stderr) but silently assumed cron was present. Add an explicit
"check command -v crontab, apt install cron, or pick systemd"
preflight to the caveats so the error is surfaced before the
user runs into it.
Caught during 2026-04-30 Phase 4 runtime validation on a Debian
trixie host: `crontab -T` lint failed because cron wasn't
installed, even though the underlying docker-exec shell command
(the actual workload) ran fine.
Previous wording claimed opencode-devbox "bakes it in via
mempalace-toolkit" as if that were always true, but until
opencode-devbox v1.14.30b the image only shipped the mempalace
Python package, not the toolkit wrappers. Users following the
*-devbox scheduler docs on earlier images would hit
"mempalace-session: command not found" inside the container.
Rewrite the precondition to:
- Name the version where bake-in starts (v1.14.30b).
- Link to the upstream INSTALL_MEMPALACE_TOOLKIT build arg.
- Document the escape hatch for custom/older containers
(./install.sh --yes) and flag its ephemeral nature, so nobody
leans on the manual install as a long-term solution.
Caught during 2026-04-30 runtime validation of the *-devbox
systemd unit on a freshly-rebuilt container.
On hosts running a long-lived opencode-devbox (or equivalent)
container, mempalace-session lives INSIDE the container, not on
the host. The existing contrib/* templates install a scheduler on
the machine that runs the tool; for the devbox case the scheduler
has to live on the host and reach into the container via
'docker exec'. This was noted in passing in contrib/README.md but
no templates were actually shipped for it.
Adds parallel *-devbox templates for systemd and cron:
contrib/systemd/mempalace-session-devbox.service
- Type=oneshot, same 2h TimeoutStartSec + low Nice as the direct
variant.
- Two Environment knobs (CONTAINER, CONTAINER_USER) default to
opencode-devbox/developer, overrideable via
'systemctl --user edit'.
- ExecCondition checks 'docker ps --filter name= --filter
status=running' so the unit no-ops cleanly when the container
is currently down. systemd reports this as a successful
'condition failed' state — no alert noise across dev cycles
of teardown/rebuild.
- ExecStart is plain /usr/bin/docker exec with no shell; systemd
does the env-var expansion.
- Stdout/stderr go to journalctl --user -u <unit> (nothing to
redirect, since docker exec surfaces container output to the
calling process).
contrib/systemd/mempalace-session-devbox.timer
- Mon 03:00 Persistent=true RandomizedDelaySec=30m, mirrors the
direct timer.
contrib/cron/mempalace-session-devbox.cron
- Equivalent shell-wrapped form for hosts using cron instead of
systemd. 'docker ps | grep -q .' short-circuits if the container
isn't running. Log goes to $HOME/.cache/mempalace-session/
cron-devbox.log on the HOST (outside the container) so it's
inspectable without dropping into the devbox.
contrib/README.md:
- Replaces the two-paragraph 'Running inside a container' note
with a proper section: preconditions, install recipes for both
the systemd and cron devbox variants, verify/uninstall commands,
customization via 'systemctl --user edit', behaviour when the
container is down.
- Chooser table gains a dedicated row pointing devbox users at
the *-devbox templates, and mentions the systemd vs cron pick
for that case.
- New 'When to pick devbox variants vs direct ones' table covers
the rare both-installed case (host mempalace AND in-container
mempalace see separate palaces — they don't cross-pollinate).
Top-level README.md 'Keeping it fresh' subsection gains a quick-start
block for the devbox variant alongside the existing Linux/macOS
quick-starts.
Tested: all four systemd units parse cleanly as INI via
configparser (sections + key=value pairs); validated file sizes
and locations match the layout described in docs. Runtime
validation (systemctl --user enable; actual docker exec) requires
a host with docker + an opencode-devbox container up — deferred
to the user's Mac/Linux boxes.
Ship a launchd user agent plist alongside the existing systemd and
cron templates so macOS users can schedule mempalace-session without
falling back to cron. launchd is the macOS-native equivalent of a
systemd user timer: same scheduling model, same log conventions, same
single-instance guarantees.
- contrib/launchd/se.jordbo.mempalace-session.plist:
- Label uses reverse-DNS from the jordbo.se domain for consistency
with other user-installed launchd jobs; fork the prefix if reusing
this template in a different org.
- ProgramArguments points at /Users/USER/.local/bin/mempalace-session
(USER is substituted at install time, same pattern as
contrib/cron/).
- EnvironmentVariables.PATH covers ~/.local/bin, Apple Silicon
Homebrew, Intel Homebrew, and system defaults — launchd agents
get a minimal PATH by default and the wrapper needs to find
mempalace + python3.
- StartCalendarInterval matches systemd unit's schedule: Monday
03:00 local.
- RunAtLoad=false — load shouldn't trigger a run; schedule does.
- ProcessType=Background + LowPriorityIO=true + Nice=10 mirror
the systemd unit's Nice=10 + IOSchedulingClass=idle. macOS's
automatic App Nap and resource throttling for Background jobs
yields to interactive work cleanly.
- ExitTimeOut=7200 matches systemd's TimeoutStartSec=7200.
- StandardOut/ErrorPath under ~/Library/Logs/ so Console.app
surfaces them.
- contrib/README.md gains a full launchd section:
- Caveat table comparing to systemd (Persistent=true isn't quite
matched; RandomizedDelaySec has no equivalent; overlap prevention
is automatic).
- Install recipe using launchctl bootstrap (modern) with a fallback
note for legacy launchctl load -w on older macOS.
- Verify section shows launchctl list, launchctl print, log tails,
and launchctl kickstart for manual testing.
- Uninstall via launchctl bootout.
- Chooser table updated: macOS now explicitly points at launchd,
not cron.
- ARCHITECTURE.md §5, SKILL.md Quick automation pitch, and README.md
Keeping it fresh section all updated to mention the three scheduler
options and give per-platform quick-starts.
Plist XML validated with plistlib.
Until opencode session-stopping hooks land upstream, mempalace-session
is the entire mechanism that gets opencode conversations into the
palace — skip it and session history stays trapped in a local SQLite
DB, invisible to semantic search. Previous docs covered setup well
but were thin on when and how often to run it.
- ARCHITECTURE.md §5: replace the one-line 'When to re-mine' note with
a full Operational Routine section — triggers, cadence, relationship
to the session lifecycle, automation pointers, verification.
- SKILL.md: add an Operational Routine section aimed at agents —
when to suggest invoking the tool, cadence guidance, how to
distinguish this producer-side tool from the consumer-side
mempalace skill's in-session habits.
- README.md: add 'Keeping it fresh' subsection pointing at contrib/
and the full docs.
contrib/ ships three ready-to-use templates:
- systemd/mempalace-session.{service,timer} — user units with weekly
Mon 03:00 schedule, Persistent=true catch-up, RandomizedDelaySec for
fleet-wide jitter, ConditionPathExists guard for opencode-less boxes,
Nice+IOSchedulingClass=idle so it never fights interactive work.
- cron/mempalace-session.cron — sample crontab entry with log
redirection and clear USER-substitution instructions.
- README.md with install/verify/uninstall recipes for both, a chooser
table (systemd vs cron), container/devbox caveats, and tuning notes
(daily vs weekly vs monthly trade-offs).
The user's LATER-list item 'wrap mempalace-session in cron/systemd
timer for true auto-save coverage' is now actionable: a single
systemctl --user enable --now command stands it up.
Joakim Persson
joakimp