diff --git a/AGENTS.md b/AGENTS.md index 31288e7..b68cf73 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,7 +64,7 @@ Those are pi-generic concerns. This toolkit installs **only** the pi↔mempalace - **Standalone executables** in `bin/` with `#!/usr/bin/env bash` shebang, no extension, `chmod +x`. Must work in non-interactive contexts (agent processes, cron, CI). - **Thin wrappers only.** Neither tool reimplements the mempalace miner. Both follow the **stage-to-cache-then-mine** idiom: curate input to `~/.cache/…//`, then delegate to `mempalace mine`. -- **Idempotent + dry-runnable.** Every tool supports `--dry-run`. Second invocation on unchanged input is a no-op (dedup via `source_file` path, optionally + `mtime`). +- **Idempotent + dry-runnable.** Every tool supports `--dry-run`. Second invocation on unchanged input is a no-op (dedup via `source_file` path + `mtime` — all three miner modes pass `check_mtime=True`). - **No external Python deps.** Stdlib only (`sqlite3`, `json`, `pathlib`). Inline in the bash wrapper via heredoc. - Argument parsing: `--help`/`-h` first, then mode flags, then positional args. - Comment sections use `# ── Section Name ──────` style (matches sibling `cli_utils` repo). diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 2ab2f9b..d3ae417 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -91,7 +91,7 @@ Neither wrapper reimplements the mempalace miner. They each: 1. Curate input (filter / transform / rename). 2. Write it to a deterministic path under `~/.cache/…//` with `mtime` preserved (via `cp -p` or explicit `os.utime`). -3. Delegate actual embedding + filing to `mempalace mine`, which already dedups on `source_file` path. +3. Delegate actual embedding + filing to `mempalace mine`, which already dedups on `source_file` path + `mtime`. This keeps the wrappers thin. A third wrapper following the same idiom would justify factoring a shared helper library — two does not. @@ -264,7 +264,11 @@ best, and lets a scoped `mempalace sync` prune every drawer mined from it at wor Both wrappers dedup via `mempalace mine`'s built-in key: - `mempalace-docs`: keys on `source_file` path + `mtime` → edit a doc, it re-mines; unchanged files are skipped. -- `mempalace-session`: keys on `source_file` path alone (convos miner doesn't check mtime) → a session's JSONL filename is `_.jsonl`, stable per session, so re-runs skip already-filed sessions. To force re-mining, delete the staging dir. +- `mempalace-session` / `mempalace-pi-session`: the convos miner keys on `source_file` path + `mtime` as well (`file_already_mined(..., check_mtime=True)` in upstream `convo_miner.py`) → a session's JSONL filename is `_.jsonl`, stable per session, so a re-run on unchanged content is a no-op, while a session that has grown or been rewritten in place is **purged and refiled** rather than skipped. Transcripts are not immutable — an agent appends to its own transcript while the session is live, and `/compact` or `/clear` rewrites one — so this is the common case, not an edge case. + +Because the key includes `mtime`, a staged copy must carry the *source's* mtime, not the copy's: `mempalace-docs` stages with `cp -p`, and the session feeders stamp the source transcript's mtime onto the staged file. Side effect worth knowing when inspecting a stage or a remote inbox: staged files show the original session's timestamp, so an old mtime there is expected and says nothing about when the file was shipped. Judge a ship by the feeder's log, never by inbox timestamps. + +**Do not delete a staging dir to force a re-mine.** Changed content re-mines by itself, and the staged *path* is the dedup key: if a stage is wiped or recreated somewhere else, the palace no longer recognises the old drawers as belonging to those sources, so the wing lands a second time instead of being replaced — and a scoped `mempalace sync` will prune drawers whose staged source has vanished. The non-destructive way to force one session is to `touch` its staged JSONL, which takes the purge-and-refile path without moving the key. **Verified:** a second full `mempalace-session` run immediately after the first produces 0 new drawers. The only cost is the post-mine `repair` step (index rebuild — ~5 min on 5k drawers). diff --git a/README.md b/README.md index 59a6726..148fc83 100644 --- a/README.md +++ b/README.md @@ -460,7 +460,7 @@ mempalace-session --help - Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer. - `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix. -**Dedup:** staging at `/opencode-stage//` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`_.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir. +**Dedup:** staging at `/opencode-stage//` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`_.jsonl`). The convos miner keys on `source_file` **and** `mtime`, so re-runs skip unchanged sessions, while a session that has grown or been rewritten in place is purged and refiled automatically — there is nothing to force. What must stay stable is the staged *path* itself: wiping the staging dir re-keys dedup rather than refreshing it, so those sources land as duplicates instead of replacing the drawers they already have. **`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count: diff --git a/SKILL.md b/SKILL.md index 9a6f315..b1d6a18 100644 --- a/SKILL.md +++ b/SKILL.md @@ -129,6 +129,18 @@ mempalace-session --session ses_abc123 # one specific session ### Force re-mine +You almost never need this. The convos miner keys on `source_file` + `mtime`, so a +session that grew or was rewritten in place re-mines by itself on the next run, and a +normalizer-schema bump re-mines what it affects. The non-destructive force for a single +session is to change its staged mtime: + +```bash +touch /opencode-stage//_.jsonl +mempalace-session # purges that session's drawers, refiles +``` + +Nuking the stage is a last resort, not the normal route: + ```bash rm -rf /opencode-stage// # nukes staging dir mempalace-session # stages + mines fresh @@ -136,6 +148,10 @@ mempalace-session # stages + mines fresh # a scoped sync prunes the drawers mined from it instead of refiling them. ``` +The staged path *is* the dedup key, so this is a destructive way to get a refile: if the +stage comes back at a different path, the palace cannot match the new sources to the old +drawers and the whole wing lands a second time instead of being replaced. + Staging is ephemeral by design; the palace is the source of truth. ## Operational Routine (when to invoke) @@ -245,7 +261,7 @@ Budget **~20 minutes per 60-session batch**. Scales roughly linearly with messag - **Don't run `mempalace mine` directly on a project.** Use `mempalace-docs` — otherwise source code floods the palace. - **Don't try to point `mempalace mine --mode convos` at `opencode.db` directly.** The convos miner reads files (txt/md/json/jsonl) only — no SQLite support. Use `mempalace-session` to export first. -- **Don't delete staging dirs unnecessarily.** They're dedup anchors; deleting means a forced re-mine of everything in that wing. +- **Don't delete staging dirs unnecessarily.** They're dedup anchors, not caches: deleting forces a re-mine of everything in that wing, and if the dir returns at a different path the old drawers are duplicated rather than replaced. To refile one session, `touch` its staged JSONL instead. - **Don't forget `mempalace_reconnect`** after a mine from inside a live MCP session — otherwise search hits the stale index. - **Don't mine with `--min-messages 0` or `1`** — 78 out of 140 sessions in reference corpus were throwaway `/exit`'d sessions that would flood the palace with noise. Default 3 is sensible.