From 36845e14b237e86d9c314dd535a7c06697edaaa7 Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Thu, 30 Apr 2026 06:29:55 +0000 Subject: [PATCH] Document the operational routine + ship automation templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Until opencode session-stopping hooks land upstream, mempalace-session is the entire mechanism that gets opencode conversations into the palace — skip it and session history stays trapped in a local SQLite DB, invisible to semantic search. Previous docs covered setup well but were thin on when and how often to run it. - ARCHITECTURE.md §5: replace the one-line 'When to re-mine' note with a full Operational Routine section — triggers, cadence, relationship to the session lifecycle, automation pointers, verification. - SKILL.md: add an Operational Routine section aimed at agents — when to suggest invoking the tool, cadence guidance, how to distinguish this producer-side tool from the consumer-side mempalace skill's in-session habits. - README.md: add 'Keeping it fresh' subsection pointing at contrib/ and the full docs. contrib/ ships three ready-to-use templates: - systemd/mempalace-session.{service,timer} — user units with weekly Mon 03:00 schedule, Persistent=true catch-up, RandomizedDelaySec for fleet-wide jitter, ConditionPathExists guard for opencode-less boxes, Nice+IOSchedulingClass=idle so it never fights interactive work. - cron/mempalace-session.cron — sample crontab entry with log redirection and clear USER-substitution instructions. - README.md with install/verify/uninstall recipes for both, a chooser table (systemd vs cron), container/devbox caveats, and tuning notes (daily vs weekly vs monthly trade-offs). The user's LATER-list item 'wrap mempalace-session in cron/systemd timer for true auto-save coverage' is now actionable: a single systemctl --user enable --now command stands it up. --- ARCHITECTURE.md | 86 +++++++++++- README.md | 19 +++ SKILL.md | 51 +++++++ contrib/README.md | 154 ++++++++++++++++++++++ contrib/cron/mempalace-session.cron | 19 +++ contrib/systemd/mempalace-session.service | 23 ++++ contrib/systemd/mempalace-session.timer | 16 +++ 7 files changed, 367 insertions(+), 1 deletion(-) create mode 100644 contrib/README.md create mode 100644 contrib/cron/mempalace-session.cron create mode 100644 contrib/systemd/mempalace-session.service create mode 100644 contrib/systemd/mempalace-session.timer diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f3a27d6..fb2e4f0 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -188,7 +188,91 @@ Both wrappers dedup via `mempalace mine`'s built-in key: ### When to re-mine - `mempalace-docs`: after significant doc changes in a project. -- `mempalace-session`: opportunistically. Every few days catches new opencode sessions. Or wire to cron / systemd timer for true auto-save coverage (not yet done). +- `mempalace-session`: see the full **Operational Routine** below. + +### Operational Routine (the `mempalace-session` workflow) + +Until opencode grows session hooks and `hooks_cli.py` grows an opencode harness (see §6), **`mempalace-session` is the entire mechanism that gets opencode conversations into the palace.** Skip it and your session history exists only inside `~/.local/share/opencode/opencode.db` — a local SQLite file that's invisible to `mempalace_search`, vulnerable to volume wipes, and lost if the devbox is replaced. + +That makes the routine worth codifying: + +#### Triggers (when to run) + +| Trigger | What to run | Why | +|---|---|---| +| **Substantive session you want preserved past `/exit`** | `mempalace-session --session ` | Targeted save before destructive action; see `~/.local/share/opencode/opencode.db` `session` table for the ID. | +| **Before a container recreate** | `mempalace-session` | The opencode DB lives in a named volume (`devbox-data`) so it normally survives, but a full mine right before is cheap insurance. | +| **Fresh machine, first provisioning** | `mempalace-session --dry-run` then `mempalace-session` | Backfills the whole corpus. Expect ~20 min / 60 sessions. | +| **Periodic sweep** | `mempalace-session` | Weekly catches anything you didn't explicitly save. Dedup is free, so running more often only costs the ~5 min repair. | +| **After upstream mempalace upgrade** | `mempalace-session` + `mempalace repair` | If the miner changed normalization or chunking, re-mine ensures the palace reflects current logic. Rare. | + +#### Cadence (how often) + +**Default: weekly.** Dedup is free on unchanged sessions, and `wing_conversations` growth is roughly linear in user activity. Weekly is frequent enough that searches almost always include recent context, and infrequent enough that the cost is negligible. + +**Daily** is fine but wasteful — you'll pay the post-mine `repair` cost seven times more often than you need. If you want daily runs, add `--no-repair` and schedule a separate weekly repair. + +**Monthly** is too infrequent. You'll search for "that thing we discussed last Tuesday" and miss it. + +#### Relationship to the session lifecycle + +`mempalace-session` is **offline, inter-session maintenance** — it runs between agent sessions, not during them. It does not replace the in-session habits from the consumer-side `mempalace` skill: + +| Habit | When | Who | +|---|---|---| +| Wake-up search (load recent diary) | Agent session start | Agent, during session | +| Wind-down diary write | Agent session end | Agent, during session | +| `mempalace-session` mine | Between sessions (manual or scheduled) | Operator or automation | + +The first two are live; the third is batched. They're complementary, not alternatives. A machine doing only wake-up/wind-down keeps a diary but loses the actual conversation turns. A machine doing only `mempalace-session` captures the raw turns but not the curated summaries. Do both. + +#### Automation + +Pick one: + +1. **systemd user timer** (recommended on modern Linux). Survives reboots, optional `Persistent=true` catch-up, logs to `journalctl`, background I/O priority. Templates in [`contrib/systemd/`](contrib/systemd/). +2. **cron** (simpler, works anywhere). Templates in [`contrib/cron/`](contrib/cron/). +3. **Manual** — run `mempalace-session` opportunistically. Fine on machines where you're in and out frequently; less fine on long-running devboxes. + +Install recipes, verification commands, and uninstall steps for all three are in [`contrib/README.md`](contrib/README.md). + +Quick-start (systemd user timer): + +```bash +mkdir -p ~/.config/systemd/user +cp contrib/systemd/*.{service,timer} ~/.config/systemd/user/ +systemctl --user daemon-reload +systemctl --user enable --now mempalace-session.timer +# Optional on headless boxes: keep timer running when logged out +sudo loginctl enable-linger "$USER" +systemctl --user list-timers mempalace-session.timer +``` + +Quick-start (cron): + +```bash +sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron \ + | (crontab -l 2>/dev/null; cat) | crontab - +mkdir -p ~/.cache/mempalace-session +``` + +#### Verification + +After any run (manual or scheduled), confirm the palace grew: + +```bash +mempalace-session --dry-run # should list sessions +# Or from inside a live MCP client: +# mempalace_status — wing_conversations count +# mempalace_reconnect — refresh index after mine +``` + +A healthy run produces one of: +- **First run on fresh corpus**: several hundred to several thousand new drawers. +- **Incremental run**: zero to a few dozen new drawers (whatever grew since last run). +- **Rerun with no new activity**: zero new drawers, only the repair step runs. + +A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `~/.cache/mempalace-session//` modification times. ### Cost profile (reference) diff --git a/README.md b/README.md index 07987a9..3524ea5 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,25 @@ mempalace-docs /workspace/my_project --dry-run mempalace-docs /workspace/my_project ``` +### Keeping it fresh (automation) + +Manual invocation is fine while you're actively driving the machine, but long-running devboxes benefit from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install templates: + +- **systemd user timer** (recommended on modern Linux): survives reboots, catches missed runs, logs to `journalctl`. +- **cron**: simpler, works everywhere, no user-unit awareness needed. + +Quick-start (systemd, weekly Mon 03:00 local): + +```bash +mkdir -p ~/.config/systemd/user +cp contrib/systemd/*.{service,timer} ~/.config/systemd/user/ +systemctl --user daemon-reload +systemctl --user enable --now mempalace-session.timer +sudo loginctl enable-linger "$USER" # optional, for headless boxes +``` + +See [`contrib/README.md`](contrib/README.md) for full install/verify/uninstall recipes, tuning, and devbox/container caveats. The full operational routine (triggers, cadence, verification) is in [`ARCHITECTURE.md`](ARCHITECTURE.md) §5. + ### Containerized (devbox) notes On a Docker-based devbox, the palace and opencode DB should live on named volumes so they survive container recreate: diff --git a/SKILL.md b/SKILL.md index 28d88b1..7f0bcdc 100644 --- a/SKILL.md +++ b/SKILL.md @@ -104,6 +104,57 @@ mempalace-session # stages + mines fresh Staging is ephemeral by design; the palace is the source of truth. +## Operational Routine (when to invoke) + +Until upstream opencode session hooks land, **`mempalace-session` is the entire mechanism** that gets opencode conversations into the palace. If the user's opencode history isn't showing up in `mempalace_search`, the most likely cause is "`mempalace-session` hasn't been run recently". + +### Agent-level triggers + +Suggest invoking the tool when any of these apply: + +- User asks *"why can't you find our conversation from earlier?"* / *"you should remember this from last week"* → the palace may not have the opencode session yet. Check `mempalace_list_wings` for `wing_conversations`; if the last drawer there is older than the referenced conversation, run `mempalace-session`. +- User mentions an upcoming **container recreate / system migration / OS reinstall** → suggest a full mine first as a backup checkpoint. The opencode DB normally survives via named volumes, but an explicit mine is cheap insurance. +- User has just **set up a new machine** and asks about mempalace → part of the recipe is the initial backfill. Run `mempalace-session --dry-run` first to show scope, then the real mine. +- User completed a **substantive session they want preserved past `/exit`** → offer a targeted `mempalace-session --session ` right then. +- User asks *"how do I keep this up to date?"* → point at [`contrib/`](../../contrib/) (systemd timer or cron recipes). + +### Cadence guidance + +| Situation | Suggested cadence | +|---|---| +| Active devbox, daily opencode use | Weekly automated (systemd timer or cron) | +| Occasional opencode user | Monthly manual or weekly automated | +| Fresh machine / first setup | One-shot full backfill, then schedule | +| "I'm about to rebuild the container" | Run now, as a checkpoint | +| Automated daily mines | Pass `--no-repair` + schedule weekly repair separately | + +Don't suggest running more often than daily — the post-mine HNSW repair (~5 min on 5k drawers) dominates cost, and session growth is slow enough that daily is already overkill. + +### Relationship to the `mempalace` skill + +The consumer-side `mempalace` skill covers in-session habits (wake-up search, wind-down diary). `mempalace-session` is **between-session maintenance** — it runs when no agent session is active, typically automated. Both are needed: + +- Wake-up/wind-down → curated summaries (what mattered, what was decided) +- `mempalace-session` → raw turns (the actual words exchanged) + +A machine running only one of these has half a memory. Agents loading the `mempalace` skill on an opencode machine should be aware this tool exists and point users at it if they ask about persistence. + +### Quick automation pitch + +If the user has a devbox and hasn't set up automation, offer it: + +```bash +# One-time setup (modern Linux, systemd user timer) +cd ~/mempalace-toolkit +mkdir -p ~/.config/systemd/user +cp contrib/systemd/*.{service,timer} ~/.config/systemd/user/ +systemctl --user daemon-reload +systemctl --user enable --now mempalace-session.timer +sudo loginctl enable-linger "$USER" # optional, for headless boxes +``` + +Full install/verify/uninstall recipes for both systemd and cron are in [`contrib/README.md`](../../contrib/README.md). The quick-start above defaults to weekly runs on Monday 03:00 local time with a ≤30 min randomized delay. + ## Failure Modes & Fixes | Symptom | Cause | Fix | diff --git a/contrib/README.md b/contrib/README.md new file mode 100644 index 0000000..5e406be --- /dev/null +++ b/contrib/README.md @@ -0,0 +1,154 @@ +# contrib/ — automation recipes for `mempalace-session` + +Manual invocation of `mempalace-session` is fine on a machine you actively drive. For long-running devboxes, a weekly automated mine keeps the palace fresh without thinking about it. This directory ships ready-to-use templates for two common scheduling mechanisms. + +> **Before using either**: confirm the toolkit is installed and the wrapper works — +> `mempalace-session --dry-run` should list qualifying sessions. If that errors, fix the install before scheduling. + +Pick **one**. Running both would double-mine (harmless — dedup skips everything on the second run — but wastes wall time on the HNSW repair). + +--- + +## systemd user timer (recommended on modern Linux) + +**Why:** runs without the user logged in (with `loginctl enable-linger`), survives reboots, logs to `journalctl`, Persistent=true catches missed runs after the machine was off. No root required — it's a *user* unit. + +**Install:** + +```bash +mkdir -p ~/.config/systemd/user +cp contrib/systemd/mempalace-session.service ~/.config/systemd/user/ +cp contrib/systemd/mempalace-session.timer ~/.config/systemd/user/ +systemctl --user daemon-reload +systemctl --user enable --now mempalace-session.timer + +# Optional: keep the timer running when you log out (needed on headless servers) +sudo loginctl enable-linger "$USER" +``` + +**Verify:** + +```bash +# Is the timer active and when will it next fire? +systemctl --user list-timers mempalace-session.timer + +# Last run status + log tail +systemctl --user status mempalace-session.service + +# Full run log (since today) +journalctl --user -u mempalace-session --since today + +# Force a run right now (outside the schedule), for testing +systemctl --user start mempalace-session.service +``` + +**Uninstall:** + +```bash +systemctl --user disable --now mempalace-session.timer +rm ~/.config/systemd/user/mempalace-session.{service,timer} +systemctl --user daemon-reload +``` + +### What the service does + +- `Type=oneshot` — runs to completion, not a long-lived daemon. +- `ConditionPathExists=%h/.local/share/opencode/opencode.db` — skips silently on machines that haven't used opencode (no wasted boot-time runs). +- `ConditionPathExists=!%t/mempalace-session.lock` + `ExecStartPre/ExecStopPost` — soft mutual exclusion between overlapping runs. +- `Nice=10` + `IOSchedulingClass=idle` — background priority; won't interfere with interactive work. +- `TimeoutStartSec=7200` — 2 hour ceiling. The reference 60-session mine takes ~21 min; this is headroom for large corpora + slow disks. + +### What the timer does + +- `OnCalendar=Mon 03:00` — weekly, Monday 03:00 local time. Edit to taste (see `man systemd.time` for syntax). +- `Persistent=true` — if the machine was off at the scheduled time, run on next boot. +- `RandomizedDelaySec=30m` — jitters up to 30 minutes to avoid thundering-herd across a fleet. + +--- + +## cron + +**Why:** simpler, ubiquitous, works on any UNIX. No `loginctl enable-linger` dance, no user-units awareness required. + +**Caveats:** no "persistent" semantics (a missed run while the machine was off stays missed); default cron output goes to mail or is silently dropped if no MTA. + +**Install:** + +```bash +# Edit the template first — replace USER with your actual username +sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron > /tmp/mempalace-session.cron + +# Append to your existing crontab (preserves any entries you already have) +(crontab -l 2>/dev/null; cat /tmp/mempalace-session.cron) | crontab - +rm /tmp/mempalace-session.cron + +# Verify +crontab -l | grep mempalace +``` + +Ensure `~/.cache/mempalace-session/` exists so the log file can be written: + +```bash +mkdir -p ~/.cache/mempalace-session +``` + +**Verify a run is happening:** + +```bash +# Tail the log the cron entry writes to +tail -f ~/.cache/mempalace-session/cron.log + +# Or force a run manually to prove the command is well-formed +mempalace-session +``` + +**Uninstall:** + +```bash +crontab -e # remove the mempalace-session line by hand +``` + +--- + +## Which should I pick? + +| Situation | Pick | +|---|---| +| Desktop / laptop, modern systemd-based distro | systemd user timer | +| Long-running devbox or server, wants "Persistent=true" catch-up | systemd user timer | +| macOS, BSD, or distro without systemd | cron | +| You already have a cron-based job scheduler on the box | cron | +| You want logs in `journalctl` rather than a file | systemd user timer | + +If you're not sure, pick systemd. `Persistent=true` alone is worth it on any box that ever sleeps or reboots. + +--- + +## Running inside a container (devbox) + +Inside a Docker-based devbox, neither systemd nor cron typically runs by default. Two options: + +1. **Schedule on the host, not the container** — have the host run `docker exec -u mempalace-session` on a timer. The container must be long-running (not per-invocation) for this to work. +2. **Run a systemd-in-container setup** — viable but usually not worth the complexity for this alone. + +For most devbox users, a simple weekly manual run via `mempalace-session` (or a host-side cron that shells into the container) is the pragmatic choice. The tool is cheap enough that skipping a week costs nothing — dedup will catch up on the next run. + +--- + +## Tuning + +**Frequency.** Weekly is the default because: +- New sessions you care about are typically a handful per week per user. +- Dedup is free on unchanged sessions, so there's no cost to running daily other than the ~5 min post-mine repair. +- Weekly keeps the palace fresh enough that searches almost always return current context. + +**Daily or more:** edit `OnCalendar=` or the cron DOW field. On a daily schedule, add `--no-repair` to the wrapper invocation and let a separate weekly unit handle repair — otherwise you repair 7× more often than you need. + +**Monthly:** probably too infrequent. You'll search for "that thing we discussed last Tuesday" and miss it. + +--- + +## See also + +- [`../../ARCHITECTURE.md`](../../ARCHITECTURE.md) §5 — operational routine (triggers, cadence) in full context. +- [`../../SKILL.md`](../../SKILL.md) — the agent-side Operational Routine section for when an AI agent should suggest running this. diff --git a/contrib/cron/mempalace-session.cron b/contrib/cron/mempalace-session.cron new file mode 100644 index 0000000..b47809d --- /dev/null +++ b/contrib/cron/mempalace-session.cron @@ -0,0 +1,19 @@ +# Sample crontab entry for mempalace-session. +# +# Runs a full opencode → MemPalace mine weekly (Mondays at 03:00 local). +# Dedup is free on subsequent runs (unchanged sessions skipped); only the +# post-mine HNSW repair has cost, so frequent runs are cheap. +# +# To install: +# (crontab -l 2>/dev/null; cat contrib/cron/mempalace-session.cron) | crontab - +# +# To remove, edit your crontab: +# crontab -e +# +# Adjust PATH below if ~/.local/bin is not the binary location on your box. +# Replace USER with your actual username, or use `$(whoami)` in-shell. + +PATH=/home/USER/.local/bin:/usr/local/bin:/usr/bin:/bin + +# m h dom mon dow command +0 3 * * 1 mempalace-session >> /home/USER/.cache/mempalace-session/cron.log 2>&1 diff --git a/contrib/systemd/mempalace-session.service b/contrib/systemd/mempalace-session.service new file mode 100644 index 0000000..43c3275 --- /dev/null +++ b/contrib/systemd/mempalace-session.service @@ -0,0 +1,23 @@ +[Unit] +Description=Mine opencode session history into MemPalace +Documentation=https://gitea.jordbo.se/joakimp/mempalace-toolkit +# Only run if opencode has actually been used (avoids noise on idle machines) +ConditionPathExists=%h/.local/share/opencode/opencode.db +# Don't start if a previous run is still going +ConditionPathExists=!%t/mempalace-session.lock + +[Service] +Type=oneshot +# The wrapper writes to ~/.cache/mempalace-session/ and the palace. +# Keep stdout/stderr in the journal — inspect with: +# journalctl --user -u mempalace-session --since today +ExecStart=%h/.local/bin/mempalace-session +# Belt-and-braces lock so two overlapping runs can't corrupt staging +ExecStartPre=/bin/sh -c 'touch %t/mempalace-session.lock' +ExecStopPost=/bin/sh -c 'rm -f %t/mempalace-session.lock' +# Protect against runaway runs (the 60-session reference mine takes ~21 min; +# give 2h of headroom for larger corpora and slow disks) +TimeoutStartSec=7200 +# Low priority — this is background maintenance +Nice=10 +IOSchedulingClass=idle diff --git a/contrib/systemd/mempalace-session.timer b/contrib/systemd/mempalace-session.timer new file mode 100644 index 0000000..023a38e --- /dev/null +++ b/contrib/systemd/mempalace-session.timer @@ -0,0 +1,16 @@ +[Unit] +Description=Weekly opencode → MemPalace session mine +Documentation=https://gitea.jordbo.se/joakimp/mempalace-toolkit + +[Timer] +# Every Monday at 03:00 local time. Adjust to taste. +# Use `systemctl --user list-timers mempalace-session.timer` to see next run. +OnCalendar=Mon 03:00 +# If the machine was off at the scheduled time, run at next boot. +Persistent=true +# Randomize up to 30 minutes to avoid thundering-herd across machines. +RandomizedDelaySec=30m +AccuracySec=1m + +[Install] +WantedBy=timers.target