docs: the convos miner does check mtime — finish a correction that stopped half-way

ARCHITECTURE.md and README.md still carried the claim from 954c3f2 (initial
commit) that the convos miner "keys on source_file path alone (convos miner
doesn't check mtime)", and told the operator to delete the staging dir to force a
re-mine. 29e660e corrected exactly that claim in AGENTS.md and SKILL.md — and
missed these two files, so the repo has been documenting both behaviours at once
ever since. Two files said mtime is checked, two said it is not.

Ground truth, read off the deployed mempalace 3.7.1 rather than inferred:
convo_miner.py:657 calls file_already_mined(..., check_mtime=True) inside
mine_lock(source_file), and palace.py:1430 re-mines when no drawers exist for the
source_file, when the stored normalize_version predates the current schema, or
when the mtime differs. On a mismatch the file's existing drawers are purged
(_source_file_delete_ids -> collection.delete) before refiling, so a changed
transcript is replaced rather than doubled. The docstring names the case outright:
transcripts are not assumed immutable, since a session keeps appending to its own
file while active and /compact or /clear can rewrite one in place.

The stale advice was not merely out of date, it was expensive. "Delete the staging
dir to force a re-mine" is the one gesture that re-keys dedup: the staged path IS
the key, so a stage that is wiped or recreated elsewhere makes the palace refile a
whole wing as duplicates instead of replacing it. The docs now say so, name `touch`
as the non-destructive way to force a single session, and record why staged copies
must carry the source's mtime — with the corollary that an old mtime in a stage or
a remote inbox says nothing about when the file was shipped, so a ship is judged by
the feeder's log instead.

Sample output blocks quoting "(dedup by source_file)" are deliberately left
verbatim: that is what bin/mempalace-session:426 and bin/mempalace-pi-session:857
actually print. Tightening the wrappers' own wording is a separate change, because
the samples have to move with it.
This commit is contained in:
2026-08-18 09:42:40 +02:00
parent 947604b25d
commit fd8b15f570
4 changed files with 25 additions and 5 deletions
+1 -1
View File
@@ -64,7 +64,7 @@ Those are pi-generic concerns. This toolkit installs **only** the pi↔mempalace
- **Standalone executables** in `bin/` with `#!/usr/bin/env bash` shebang, no extension, `chmod +x`. Must work in non-interactive contexts (agent processes, cron, CI). - **Standalone executables** in `bin/` with `#!/usr/bin/env bash` shebang, no extension, `chmod +x`. Must work in non-interactive contexts (agent processes, cron, CI).
- **Thin wrappers only.** Neither tool reimplements the mempalace miner. Both follow the **stage-to-cache-then-mine** idiom: curate input to `~/.cache/…/<wing>/`, then delegate to `mempalace mine`. - **Thin wrappers only.** Neither tool reimplements the mempalace miner. Both follow the **stage-to-cache-then-mine** idiom: curate input to `~/.cache/…/<wing>/`, then delegate to `mempalace mine`.
- **Idempotent + dry-runnable.** Every tool supports `--dry-run`. Second invocation on unchanged input is a no-op (dedup via `source_file` path, optionally + `mtime`). - **Idempotent + dry-runnable.** Every tool supports `--dry-run`. Second invocation on unchanged input is a no-op (dedup via `source_file` path + `mtime` — all three miner modes pass `check_mtime=True`).
- **No external Python deps.** Stdlib only (`sqlite3`, `json`, `pathlib`). Inline in the bash wrapper via heredoc. - **No external Python deps.** Stdlib only (`sqlite3`, `json`, `pathlib`). Inline in the bash wrapper via heredoc.
- Argument parsing: `--help`/`-h` first, then mode flags, then positional args. - Argument parsing: `--help`/`-h` first, then mode flags, then positional args.
- Comment sections use `# ── Section Name ──────` style (matches sibling `cli_utils` repo). - Comment sections use `# ── Section Name ──────` style (matches sibling `cli_utils` repo).
+6 -2
View File
@@ -91,7 +91,7 @@ Neither wrapper reimplements the mempalace miner. They each:
1. Curate input (filter / transform / rename). 1. Curate input (filter / transform / rename).
2. Write it to a deterministic path under `~/.cache/…/<wing>/` with `mtime` preserved (via `cp -p` or explicit `os.utime`). 2. Write it to a deterministic path under `~/.cache/…/<wing>/` with `mtime` preserved (via `cp -p` or explicit `os.utime`).
3. Delegate actual embedding + filing to `mempalace mine`, which already dedups on `source_file` path. 3. Delegate actual embedding + filing to `mempalace mine`, which already dedups on `source_file` path + `mtime`.
This keeps the wrappers thin. A third wrapper following the same idiom would justify factoring a shared helper library — two does not. This keeps the wrappers thin. A third wrapper following the same idiom would justify factoring a shared helper library — two does not.
@@ -264,7 +264,11 @@ best, and lets a scoped `mempalace sync` prune every drawer mined from it at wor
Both wrappers dedup via `mempalace mine`'s built-in key: Both wrappers dedup via `mempalace mine`'s built-in key:
- `mempalace-docs`: keys on `source_file` path + `mtime` → edit a doc, it re-mines; unchanged files are skipped. - `mempalace-docs`: keys on `source_file` path + `mtime` → edit a doc, it re-mines; unchanged files are skipped.
- `mempalace-session`: keys on `source_file` path alone (convos miner doesn't check mtime) → a session's JSONL filename is `<slug>_<id>.jsonl`, stable per session, so re-runs skip already-filed sessions. To force re-mining, delete the staging dir. - `mempalace-session` / `mempalace-pi-session`: the convos miner keys on `source_file` path + `mtime` as well (`file_already_mined(..., check_mtime=True)` in upstream `convo_miner.py`) → a session's JSONL filename is `<slug>_<id>.jsonl`, stable per session, so a re-run on unchanged content is a no-op, while a session that has grown or been rewritten in place is **purged and refiled** rather than skipped. Transcripts are not immutable — an agent appends to its own transcript while the session is live, and `/compact` or `/clear` rewrites one — so this is the common case, not an edge case.
Because the key includes `mtime`, a staged copy must carry the *source's* mtime, not the copy's: `mempalace-docs` stages with `cp -p`, and the session feeders stamp the source transcript's mtime onto the staged file. Side effect worth knowing when inspecting a stage or a remote inbox: staged files show the original session's timestamp, so an old mtime there is expected and says nothing about when the file was shipped. Judge a ship by the feeder's log, never by inbox timestamps.
**Do not delete a staging dir to force a re-mine.** Changed content re-mines by itself, and the staged *path* is the dedup key: if a stage is wiped or recreated somewhere else, the palace no longer recognises the old drawers as belonging to those sources, so the wing lands a second time instead of being replaced — and a scoped `mempalace sync` will prune drawers whose staged source has vanished. The non-destructive way to force one session is to `touch` its staged JSONL, which takes the purge-and-refile path without moving the key.
**Verified:** a second full `mempalace-session` run immediately after the first produces 0 new drawers. The only cost is the post-mine `repair` step (index rebuild — ~5 min on 5k drawers). **Verified:** a second full `mempalace-session` run immediately after the first produces 0 new drawers. The only cost is the post-mine `repair` step (index rebuild — ~5 min on 5k drawers).
+1 -1
View File
@@ -460,7 +460,7 @@ mempalace-session --help
- Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer. - Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer.
- `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix. - `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix.
**Dedup:** staging at `<palace-root>/opencode-stage/<wing>/` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir. **Dedup:** staging at `<palace-root>/opencode-stage/<wing>/` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file` **and** `mtime`, so re-runs skip unchanged sessions, while a session that has grown or been rewritten in place is purged and refiled automatically — there is nothing to force. What must stay stable is the staged *path* itself: wiping the staging dir re-keys dedup rather than refreshing it, so those sources land as duplicates instead of replacing the drawers they already have.
**`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count: **`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count:
+17 -1
View File
@@ -129,6 +129,18 @@ mempalace-session --session ses_abc123 # one specific session
### Force re-mine ### Force re-mine
You almost never need this. The convos miner keys on `source_file` + `mtime`, so a
session that grew or was rewritten in place re-mines by itself on the next run, and a
normalizer-schema bump re-mines what it affects. The non-destructive force for a single
session is to change its staged mtime:
```bash
touch <palace-root>/opencode-stage/<wing>/<slug>_<id>.jsonl
mempalace-session # purges that session's drawers, refiles
```
Nuking the stage is a last resort, not the normal route:
```bash ```bash
rm -rf <palace-root>/opencode-stage/<wing>/ # nukes staging dir rm -rf <palace-root>/opencode-stage/<wing>/ # nukes staging dir
mempalace-session # stages + mines fresh mempalace-session # stages + mines fresh
@@ -136,6 +148,10 @@ mempalace-session # stages + mines fresh
# a scoped sync prunes the drawers mined from it instead of refiling them. # a scoped sync prunes the drawers mined from it instead of refiling them.
``` ```
The staged path *is* the dedup key, so this is a destructive way to get a refile: if the
stage comes back at a different path, the palace cannot match the new sources to the old
drawers and the whole wing lands a second time instead of being replaced.
Staging is ephemeral by design; the palace is the source of truth. Staging is ephemeral by design; the palace is the source of truth.
## Operational Routine (when to invoke) ## Operational Routine (when to invoke)
@@ -245,7 +261,7 @@ Budget **~20 minutes per 60-session batch**. Scales roughly linearly with messag
- **Don't run `mempalace mine` directly on a project.** Use `mempalace-docs` — otherwise source code floods the palace. - **Don't run `mempalace mine` directly on a project.** Use `mempalace-docs` — otherwise source code floods the palace.
- **Don't try to point `mempalace mine --mode convos` at `opencode.db` directly.** The convos miner reads files (txt/md/json/jsonl) only — no SQLite support. Use `mempalace-session` to export first. - **Don't try to point `mempalace mine --mode convos` at `opencode.db` directly.** The convos miner reads files (txt/md/json/jsonl) only — no SQLite support. Use `mempalace-session` to export first.
- **Don't delete staging dirs unnecessarily.** They're dedup anchors; deleting means a forced re-mine of everything in that wing. - **Don't delete staging dirs unnecessarily.** They're dedup anchors, not caches: deleting forces a re-mine of everything in that wing, and if the dir returns at a different path the old drawers are duplicated rather than replaced. To refile one session, `touch` its staged JSONL instead.
- **Don't forget `mempalace_reconnect`** after a mine from inside a live MCP session — otherwise search hits the stale index. - **Don't forget `mempalace_reconnect`** after a mine from inside a live MCP session — otherwise search hits the stale index.
- **Don't mine with `--min-messages 0` or `1`** — 78 out of 140 sessions in reference corpus were throwaway `/exit`'d sessions that would flood the palace with noise. Default 3 is sensible. - **Don't mine with `--min-messages 0` or `1`** — 78 out of 140 sessions in reference corpus were throwaway `/exit`'d sessions that would flood the palace with noise. Default 3 is sensible.