Files
mempalace-toolkit/contrib/README.md
T
joakimp 6352373a1f fix(feeders): make post-mine repair opt-in, not default
The three feeder wrappers (mempalace-docs, mempalace-pi-session,
mempalace-session) unconditionally ran 'mempalace repair --yes' after
mining, controllable only via --no-repair opt-out. The contrib launchd
and systemd templates did not pass --no-repair, so every scheduled tick
invoked the destructive in-place HNSW rebuild.

This has bitten us twice:
  - 2026-05-04 09:08: a kickstart triggered repair while an MCP
    subprocess held the DB open; the live collection was wiped (0
    drawers) and had to be restored from the palace.backup snapshot.
  - 2026-05-05 10:00: post-mine repair crashed mid-rebuild with
    'NotFoundError: Collection [<uuid>] does not exist' - chromadb's
    rebuild recreated the collection under a new UUID while the code
    still held the old handle. Live DB survived only by luck (crash
    hit before the swap).

Fix: flip the default.
  - New flag: --repair (opt-in). Prints a warning and sleeps 3s before
    invoking 'mempalace repair --yes'.
  - --no-repair is retained as a deprecated no-op alias for backward
    compatibility with any scripts/units still passing it.
  - Default behavior: no repair. Routine ChromaDB add() keeps HNSW
    consistent; repair is a recovery op, not a maintenance tick.

Docs updated to match: README, SKILL, ARCHITECTURE, AGENTS,
contrib/README. Scheduling guidance now explicitly warns against
enabling --repair on cron/launchd/systemd-timer runs.
2026-05-05 12:35:04 +02:00

331 lines
16 KiB
Markdown

# contrib/ — automation recipes for `mempalace-session` and `mempalace-pi-session`
Manual invocation of the session-mining wrappers is fine on a machine you actively drive. For long-running devboxes, a weekly automated mine keeps the palace fresh without thinking about it. This directory ships ready-to-use templates for two common scheduling mechanisms, for each wrapper.
> **Before using either**: confirm the toolkit is installed and the wrapper works —
> `mempalace-session --dry-run` (and/or `mempalace-pi-session --dry-run`) should list qualifying sessions. If that errors, fix the install before scheduling.
Pick **one scheduler** (systemd *or* launchd *or* cron). The opencode and pi jobs can be installed side by side and staggered — templates ship with Mon 03:00 for opencode, Tue 03:00 for pi to avoid racing the post-mine HNSW repair.
## Templates at a glance
| File | What it schedules | When |
|---|---|---|
| `systemd/mempalace-session.{service,timer}` | opencode → palace | Mon 03:00 |
| `systemd/mempalace-pi-session.{service,timer}` | pi → palace | Tue 03:00 |
| `systemd/mempalace-session-devbox.{service,timer}` | opencode (inside a devbox container) → palace | Mon 03:00 |
| `launchd/se.jordbo.mempalace-session.plist` | opencode → palace (macOS) | Mon 03:00 |
| `launchd/se.jordbo.mempalace-pi-session.plist` | pi → palace (macOS) | Tue 03:00 |
| `cron/mempalace-session.cron` | opencode → palace | Mon 03:00 |
| `cron/mempalace-pi-session.cron` | pi → palace | Tue 03:00 |
| `cron/mempalace-session-devbox.cron` | opencode (devbox) → palace | Mon 03:00 |
The pi variants are drop-in copies of the opencode variants with script name and schedule updated; the install recipes below apply equally — just swap `mempalace-session` for `mempalace-pi-session` and the schedule day.
---
## systemd user timer (recommended on modern Linux)
**Why:** runs without the user logged in (with `loginctl enable-linger`), survives reboots, logs to `journalctl`, Persistent=true catches missed runs after the machine was off. No root required — it's a *user* unit.
**Install:**
```bash
mkdir -p ~/.config/systemd/user
cp contrib/systemd/mempalace-session.service ~/.config/systemd/user/
cp contrib/systemd/mempalace-session.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now mempalace-session.timer
# Optional: keep the timer running when you log out (needed on headless servers)
sudo loginctl enable-linger "$USER"
```
**Verify:**
```bash
# Is the timer active and when will it next fire?
systemctl --user list-timers mempalace-session.timer
# Last run status + log tail
systemctl --user status mempalace-session.service
# Full run log (since today)
journalctl --user -u mempalace-session --since today
# Force a run right now (outside the schedule), for testing
systemctl --user start mempalace-session.service
```
**Uninstall:**
```bash
systemctl --user disable --now mempalace-session.timer
rm ~/.config/systemd/user/mempalace-session.{service,timer}
systemctl --user daemon-reload
```
### What the service does
- `Type=oneshot` — runs to completion, not a long-lived daemon.
- `ConditionPathExists=%h/.local/share/opencode/opencode.db` — skips silently on machines that haven't used opencode (no wasted boot-time runs).
- `ConditionPathExists=!%t/mempalace-session.lock` + `ExecStartPre/ExecStopPost` — soft mutual exclusion between overlapping runs.
- `Nice=10` + `IOSchedulingClass=idle` — background priority; won't interfere with interactive work.
- `TimeoutStartSec=7200` — 2 hour ceiling. The reference 60-session mine takes ~21 min; this is headroom for large corpora + slow disks.
### What the timer does
- `OnCalendar=Mon 03:00` — weekly, Monday 03:00 local time. Edit to taste (see `man systemd.time` for syntax).
- `Persistent=true` — if the machine was off at the scheduled time, run on next boot.
- `RandomizedDelaySec=30m` — jitters up to 30 minutes to avoid thundering-herd across a fleet.
---
## launchd user agent (macOS)
**Why:** the macOS-native equivalent of a systemd user timer. Runs without a Terminal window open, logs to `~/Library/Logs/`, single-instance guarantees baked in, background-priority scheduling via `ProcessType=Background`. No Homebrew or third-party scheduler required.
**Caveats vs. systemd:**
| Systemd feature | launchd equivalent | Notes |
|---|---|---|
| `Persistent=true` catches missed runs | Partial — `StartCalendarInterval` fires on next system-awake time | If the Mac is fully off at scheduled time, the run is skipped. Sleep-at-schedule → fires on wake. |
| `RandomizedDelaySec=30m` | None native | Single-user machines rarely need jitter; add a `sleep $((RANDOM % 1800))` wrapper if you do. |
| `ConditionPathExists` | None native | `mempalace-session` exits cleanly when the opencode DB is missing, so no guard is strictly needed. |
| Lock file for overlap prevention | Automatic | launchd refuses to start a second instance of the same `Label` while one is running. |
**Install:**
```bash
# Substitute your username into the template
sed "s|USER|$USER|g" contrib/launchd/se.jordbo.mempalace-session.plist \
> ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
# Ensure the log directory exists
mkdir -p ~/Library/Logs
# Modern load (macOS 10.11+). "gui/$(id -u)" targets your login session.
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
launchctl enable "gui/$(id -u)/se.jordbo.mempalace-session"
```
> On older macOS or if you hit permissions errors with `bootstrap`, fall back to the legacy form:
> `launchctl load -w ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist`
**Verify:**
```bash
# Quick check — is the job registered?
launchctl list | grep mempalace-session
# Detailed state (next run time, last exit code, throttling)
launchctl print "gui/$(id -u)/se.jordbo.mempalace-session"
# Run log tail
tail -f ~/Library/Logs/mempalace-session.log
tail -f ~/Library/Logs/mempalace-session.err.log
# Force a run right now (outside the schedule), for testing
launchctl kickstart -p "gui/$(id -u)/se.jordbo.mempalace-session"
```
**Uninstall:**
```bash
launchctl bootout "gui/$(id -u)" ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
rm ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
# Optional — keep the old logs for post-mortem, or delete them:
# rm ~/Library/Logs/mempalace-session{,.err}.log
```
### What the plist does
- `Label=se.jordbo.mempalace-session` — reverse-DNS label; shows up in `launchctl list` and Console.app. Change the prefix if you're forking this for a different org.
- `ProgramArguments` — absolute path to `mempalace-session`. Template uses `/Users/USER/.local/bin/mempalace-session`; the install `sed` substitutes your actual username.
- `EnvironmentVariables.PATH` — covers `~/.local/bin`, Apple Silicon Homebrew (`/opt/homebrew/bin`), Intel Homebrew (`/usr/local/bin`), and system defaults. launchd agents get a minimal PATH by default, and `mempalace-session` needs to find `mempalace` + `python3`.
- `StartCalendarInterval``Weekday=1, Hour=3, Minute=0` = Monday 03:00. Omit any key to match "any" (e.g. drop `Weekday` for daily).
- `RunAtLoad=false` — don't run on load/reboot, only on schedule. Flip to `true` if you want a run at every boot.
- `ProcessType=Background` + `LowPriorityIO=true` + `Nice=10` — macOS throttles this job's CPU and I/O so it yields to interactive work.
- `ExitTimeOut=7200` — 2h ceiling, matches the systemd unit.
- `StandardOut/ErrorPath``~/Library/Logs/` is the macOS convention; Console.app picks these up automatically.
---
## cron
**Why:** simpler, ubiquitous, works on any UNIX. No `loginctl enable-linger` dance, no user-units awareness required.
**Caveats:** no "persistent" semantics (a missed run while the machine was off stays missed); default cron output goes to mail or is silently dropped if no MTA. Also: **cron is not installed by default on minimal Debian/Ubuntu hosts** (nor on most container images). Check with `command -v crontab` — if absent, `sudo apt install cron` (or equivalent) first, or use the systemd timer instead.
**Install:**
```bash
# Edit the template first — replace USER with your actual username
sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron > /tmp/mempalace-session.cron
# Append to your existing crontab (preserves any entries you already have)
(crontab -l 2>/dev/null; cat /tmp/mempalace-session.cron) | crontab -
rm /tmp/mempalace-session.cron
# Verify
crontab -l | grep mempalace
```
Ensure `~/.cache/mempalace-session/` exists so the log file can be written:
```bash
mkdir -p ~/.cache/mempalace-session
```
**Verify a run is happening:**
```bash
# Tail the log the cron entry writes to
tail -f ~/.cache/mempalace-session/cron.log
# Or force a run manually to prove the command is well-formed
mempalace-session
```
**Uninstall:**
```bash
crontab -e # remove the mempalace-session line by hand
```
---
## Which should I pick?
| Situation | Pick |
|---|---|
| Desktop / laptop, mempalace installed directly on the host, modern systemd-based Linux | `systemd/mempalace-session.{service,timer}` |
| macOS (any recent version), mempalace on the host | `launchd/se.jordbo.mempalace-session.plist` |
| Long-running Linux devbox or server, mempalace on the host | `systemd/mempalace-session.{service,timer}` |
| **opencode-devbox (or similar) container with mempalace *inside* it** | **`systemd/mempalace-session-devbox.{service,timer}`** (preferred) or `cron/mempalace-session-devbox.cron` (simpler) |
| BSD, Alpine, or Linux distro without systemd | `cron/mempalace-session.cron` |
| You already have a cron-based job scheduler on the box | any `cron/*.cron` template |
| You want logs in `journalctl` (Linux) or Console.app (macOS) rather than a file | systemd user timer / launchd |
If you're not sure: **systemd on Linux, launchd on macOS, cron only when neither is available. Use the `-devbox` variant when mempalace lives inside a long-running container rather than on the host.** All templates wrap the same `mempalace-session` command — the difference is purely in *where* the scheduler lives and whether it needs to `docker exec` into a container to reach the tool.
---
## Running inside a container (devbox)
If you run opencode inside a long-lived container like [opencode-devbox](https://gitea.jordbo.se/joakimp/opencode-devbox), neither systemd nor cron is running inside that container — they're host-level services. The correct pattern is **host-side scheduling that `docker exec`s into the running container**. The `*-devbox` templates in `contrib/systemd/` and `contrib/cron/` implement exactly this.
Preconditions:
- **Long-lived container.** `docker compose up -d` with `restart: unless-stopped` (or equivalent). If the container is ephemeral (per-invocation), this pattern doesn't apply.
- **`mempalace-session` is already installed inside the container.** opencode-devbox bakes it in from `v1.14.30b` onward (see the [`INSTALL_MEMPALACE_TOOLKIT`](https://gitea.jordbo.se/joakimp/opencode-devbox#build-args) arg). For earlier images or custom containers without the toolkit installed, run `./install.sh --yes` from a `mempalace-toolkit` checkout inside the container first — the installer is idempotent and safe to re-run. Note that a manual install lives in the container's ephemeral layer and is lost on `docker compose up --force-recreate`, so the bake-in approach is the durable solution.
- **Host user can talk to docker.** Member of the `docker` group on Linux, or Docker Desktop running under the current login session on macOS.
- **Canonical container name is `opencode-devbox`.** If you renamed it via `container_name:` or docker-compose project naming, adjust `CONTAINER` in the template.
Both devbox templates guard against "container currently stopped" — they no-op silently if `docker ps` shows no running container with the expected name. That makes the timer safe to leave enabled across dev cycles where you tear the container down and bring it back up.
### systemd user timer (host-side, devbox variant)
```bash
# Install
mkdir -p ~/.config/systemd/user
cp contrib/systemd/mempalace-session-devbox.service ~/.config/systemd/user/
cp contrib/systemd/mempalace-session-devbox.timer ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now mempalace-session-devbox.timer
# Keep the timer running across logout (typical on dev hosts)
sudo loginctl enable-linger "$USER"
```
**Customize container name / user** (if you don't use the defaults):
```bash
systemctl --user edit mempalace-session-devbox.service
# In the override that opens, set:
# [Service]
# Environment=CONTAINER=my-devbox-name
# Environment=CONTAINER_USER=my-user
```
Or edit the shipped service file in place before copying.
**Verify:**
```bash
systemctl --user list-timers mempalace-session-devbox.timer
systemctl --user status mempalace-session-devbox.service
journalctl --user -u mempalace-session-devbox --since today
# Force a run right now (while the container is up)
systemctl --user start mempalace-session-devbox.service
```
**Behaviour when the container is down:** `ExecCondition` fails, service is marked "condition failed" (considered a successful no-op by systemd), no alert noise. Bring the container back up and the next scheduled fire will run normally.
**Uninstall:**
```bash
systemctl --user disable --now mempalace-session-devbox.timer
rm ~/.config/systemd/user/mempalace-session-devbox.{service,timer}
systemctl --user daemon-reload
```
### cron (host-side, devbox variant)
```bash
# Read the template — it has CONTAINER / CONTAINER_USER at the top.
# Adjust if your setup differs from opencode-devbox defaults.
cat contrib/cron/mempalace-session-devbox.cron
# Install (preserves existing crontab entries)
(crontab -l 2>/dev/null; cat contrib/cron/mempalace-session-devbox.cron) | crontab -
# Ensure the log directory exists
mkdir -p ~/.cache/mempalace-session
# Verify
crontab -l | grep mempalace-session-devbox
tail -f ~/.cache/mempalace-session/cron-devbox.log
```
**Uninstall:**
```bash
crontab -e # remove the mempalace-session-devbox line by hand
```
### When to pick devbox variants vs direct ones
| Setup | Templates to use |
|---|---|
| mempalace installed directly on the host (no devbox) | `mempalace-session.{service,timer}`, `mempalace-session.cron`, `se.jordbo.mempalace-session.plist` |
| opencode-devbox container is where opencode + mempalace live | `mempalace-session-devbox.{service,timer}`, `mempalace-session-devbox.cron` |
| Both (rare — mempalace on host AND inside a separate devbox) | Install both; they write to separate palaces. |
The in-container mempalace sees only the container's opencode.db and palace (via named volumes). The host's mempalace, if installed, sees only the host's. Two parallel palaces don't cross-pollinate — decide where you want the source of truth to live and schedule accordingly.
### Alternative not documented here
"Run systemd inside the container" is technically viable (systemd-in-docker images exist) but adds non-trivial complexity for the sake of a once-weekly batch job. The host-scheduled approach above is equivalent in outcome and much simpler. Skip systemd-in-container unless you already have other reasons to need it.
---
## Tuning
**Frequency.** Weekly is the default because:
- New sessions you care about are typically a handful per week per user.
- Dedup is free on unchanged sessions, so there's no cost to running daily other than the ~5 min post-mine repair.
- Weekly keeps the palace fresh enough that searches almost always return current context.
**Scheduling cadence:** edit `OnCalendar=` or the cron DOW field. Post-mine repair is now **opt-in** (`--repair`) and should NOT be added to unattended schedules — the in-place HNSW rebuild has wiped live palaces on past runs. Run `mempalace repair` manually from a quiet interactive session if you ever need it.
**Monthly:** probably too infrequent. You'll search for "that thing we discussed last Tuesday" and miss it.
---
## See also
- [`../../ARCHITECTURE.md`](../../ARCHITECTURE.md) §5 — operational routine (triggers, cadence) in full context.
- [`../../SKILL.md`](../../SKILL.md) — the agent-side Operational Routine section for when an AI agent should suggest running this.