diff --git a/AGENTS.md b/AGENTS.md index 1a1cf3d..eb32b09 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,6 @@ extensions/ notify.ts # Native terminal notification when agent finishes ext-toggle.ts # /ext slash command — list & toggle extensions at runtime todo.ts # `todo` tool for the agent + /todos for the user (copy of upstream example) - mcp-loader.ts # Generic MCP server loader + /mcp slash command task.ts # `task` tool — pi-task (isolated child, PASS/FAIL envelope, boundary diff) as a tool fork-gate.ts # tool_call hook: block forks whose brief needs `task`; reason = the task(...) to make aws-sso.ts # /aws-sso: renew the AWS SSO session when it ends (local: host browser; remote: device code + ntfy push) @@ -68,7 +67,14 @@ LICENSE # MIT. Creates `~/.pi/agent/extensions/` proactively. 2. Calls `build_install_set` to resolve which extensions to install based on `--only` / `--skip` flags (default: all `.ts` files in `extensions/`). -3. For each selected extension: symlinks `extensions/.ts` → +3. `prune_stale_links` (after the confirmation prompt) — removes links in + `~/.pi/agent/extensions/` (`*.ts` and `*.ts.off`) that point into THIS repo + and whose target no longer exists. That is how a retired extension leaves a + preserved pi-devbox volume: the link outlives the file in every new image. + Foreign links and real files are never touched. `warn_retired_mcp_block` + (before the prompt, read-only) warns if `settings.json` still has the + retired loader's top-level `mcp` key. +4. For each selected extension: symlinks `extensions/.ts` → `~/.pi/agent/extensions/.ts`. Backs up any pre-existing real file or foreign symlink. @@ -345,97 +351,35 @@ cp /opt/homebrew/Cellar/pi-coding-agent/*/libexec/lib/node_modules/@earendil-wor # review diff, then commit ``` -### `mcp-loader.ts` +### `mcp-loader.ts` — RETIRED 2026-10-12 -Generic MCP client — reads an `mcp` block from `~/.pi/agent/settings.json` -and connects to each declared server (stdio subprocess or streamable-HTTP -endpoint), exposing the tools as namespaced pi tools. +Removed in favour of pi's built-in MCP support (`builtin:mcp`, pi ≥ 0.99.0: +`~/.pi/agent/mcp.json`, `pi mcp add|list|login`, the `/mcp` manager, OAuth, +per-tool `exposure`, resources). Reasons, measured on pi 1.1.0 before removal +(README "MCP servers" has the numbers): -**Why this exists:** pi has no built-in MCP loader (unlike opencode and Claude -Desktop). Adding each new MCP server as a hand-rolled extension (the way -`mempalace.ts` does it) doesn't scale past 2-3 servers. This loader is the -config-driven generalization — one extension, any number of servers, two -transports. +- **The two could not coexist.** pi treats an extension that registers `/mcp` + as a *replacement* for the built-in, so with the loader installed `mcp.json` + was never read and every start printed `… registers command /mcp, so + built-in extension mcp was not loaded`. +- **No cost to switching.** Time-to-ready is the same within noise (0.283 s + vs 0.272 s), and with no servers configured the built-in adds zero tokens + to a request. +- **The fleet had no `mcp` block anywhere** (checked on tor-ms22; all pi-devbox + devices share one configuration), so nothing needed migrating. -**Key design decisions:** +What survives elsewhere: `mempalace-toolkit/extensions/pi/mempalace.ts` +carries a vendored copy of the loader's `RemoteMcpClient` (streamable-HTTP). +That copy is now the only one, so there is no longer a sync partner for its +`MCP-STREAMABLE-HTTP-CLIENT-SYNC` token. The design notes below the old +loader's header — tool-name sanitisation for Bedrock, fail-soft per server, +404 session re-init — live on in git history (`git show a88fa99:extensions/mcp-loader.ts`). -- **Settings.json shape matches opencode / Claude Desktop verbatim.** A user - can copy-paste their `mcp` block from one harness's config to the other. -- **Tool names are prefixed with the server name** to avoid collisions when - multiple servers expose tools with the same short name. Tools that already - start with `_` (the convention some servers like mempalace - follow internally) skip the prefix to avoid double-prefixing. -- **Tool names are sanitized to `^[A-Za-z][A-Za-z0-9_]{0,63}$`** before - registering with pi. This is the strictest regex we have to satisfy: - Anthropic's Messages API allows hyphens, but AWS Bedrock's Anthropic shim - rejects them outright — a single hyphenated tool name causes the whole - turn to 4xx silently, manifesting as "no output at all" once the offending - server is enabled. context7 surfaces this because its tools are named - `resolve-library-id` and `query-docs`. We replace any non-`[A-Za-z0-9_]` - char with `_`, prepend `t_` if the result doesn't start with a letter, and - truncate to 64 chars. The original MCP-side name is kept in the closure - used by `client.callTool`, so the sanitization is purely pi-facing. -- **Fail-soft per server.** A server that won't start (binary missing, init - handshake fails, remote 4xx/5xx) logs one stderr line and is skipped. - Other servers continue. Pi keeps working. -- **Two transports behind one interface.** `IMcpClient` (`start`, `tools`, - `callTool`, `stop`) is implemented by `StdioMcpClient` (subprocess + - newline-delimited JSON-RPC) and `RemoteMcpClient` (HTTP POST + JSON or - SSE response). The extension entry dispatches on `cfg.type` and the - rest of the code is transport-agnostic. -- **Does not replace `mempalace.ts`.** The mempalace bridge has bespoke - agent-identity injection from `$MEMPALACE_AGENT_NAME` that's worth - preserving. Loader and bridge coexist. Listing `mempalace` in the `mcp` - block would produce duplicate tool registrations — don't. -- **No stdio reconnect on death.** If a stdio subprocess crashes - mid-session, its tools become permanently unavailable until pi `/reload`s. - Same limitation as the mempalace bridge today. -- **Remote sessions self-heal on 404.** Per spec 2025-11-25 §2.2, a server - that has terminated a session MUST respond with HTTP 404 to requests - carrying the stale `Mcp-Session-Id`. `RemoteMcpClient` catches this - condition (404 + sessionAtStart was non-null), drops the id, runs a fresh - `initialize` + `notifications/initialized`, and retries the original - request once. Persistent 404s after refresh surface as errors. +Do **not** reintroduce an extension that registers `/mcp`, `codemode` or +`tool_search` without meaning to replace the built-in: pi then drops the +built-in of the same name, with nothing but one startup warning to show for it. -**API used:** - -- `pi.registerTool({ name, label, description, parameters, execute })` for - each MCP tool exposed by each connected server. -- `pi.on("session_shutdown", ...)` to SIGTERM stdio subprocesses and DELETE - remote sessions cleanly. -- `Type.Unsafe<...>(inputSchema)` from typebox to pass the MCP server's - JSON schema through to pi without conversion (TypeBox schemas are - plain JSON Schema at runtime). - -**Internal MCP clients:** - -- `class StdioMcpClient` — spawn subprocess, write newline-delimited - JSON-RPC, parse newline-delimited responses, match by request `id`. - Stderr drained silently unless `PI_MCP_LOADER_DEBUG=1`. -- `class RemoteMcpClient` — streamable-HTTP per MCP spec 2025-03-26. - POST JSON-RPC body to a single URL with `Accept: application/json, - text/event-stream`. Server replies either with one JSON response or an - SSE stream containing `event: message` / `data: ` blocks; we - consume the stream until the response with our `id` arrives, then - cancel the reader. Optional `Mcp-Session-Id` header is captured on - initialize and round-tripped on every subsequent request, then DELETEd - on stop. Per-server `headers` config (e.g. `Authorization`) is merged - into every request. No GET-stream subscription — server-initiated - notifications are not consumed. -- Both share `interface IMcpClient { serverName, tools, start, callTool, stop }`. - Both send `initialize` handshake with `protocolVersion: MCP_PROTOCOL_VERSION` - (currently `2025-11-25`, per the constant at the top of the file), - then `notifications/initialized`, then `tools/list` to discover tools. - -**Future extensions:** - -- OAuth flow for remote servers that require it (today: pre-issued bearer - tokens via `headers` only). -- Per-tool enable/disable in settings.json (e.g. - `"mcp.searxng.tools": ["web_search"]` to expose only a subset). -- Reconnect-on-crash with exponential backoff. -- Schema sanitization for MCP servers that emit malformed `inputSchema` - (some return `{type: "object", properties: {...}}` without `required`). +## Testing No framework. Manual: diff --git a/README.md b/README.md index b00bced..cee57df 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ chmod +x install.sh ./install.sh ``` -Each `.ts` file in `extensions/` is symlinked into `~/.pi/agent/extensions/`. Existing real files are backed up with a timestamp. Re-runs are idempotent. +Each `.ts` file in `extensions/` is symlinked into `~/.pi/agent/extensions/`. Existing real files are backed up with a timestamp. Re-runs are idempotent. A link into this repo whose file is gone (an extension that was retired, like `mcp-loader.ts`) is removed. ## Bundled skill @@ -335,72 +335,42 @@ State lives in the session's tool result `details`, not an external file. So: This is a verbatim copy of the upstream `examples/extensions/todo.ts` shipped with `pi-coding-agent`. Refresh from upstream when desired (see `AGENTS.md`). -### `mcp-loader.ts` +### MCP servers — use pi's built-in support (`mcp-loader.ts` is retired) -Generic MCP server loader. Reads an `mcp` block from `~/.pi/agent/settings.json` (same shape as opencode and Claude Desktop) and connects to each declared server, exposing all of their tools to pi as native tools — namespaced as `_` to avoid collisions, with non-`[A-Za-z0-9_]` characters replaced by `_` so the names pass the strictest provider tool-name regex (e.g. AWS Bedrock). +`mcp-loader.ts` was removed on 2026-10-12. pi has shipped MCP support as a built-in extension since 0.99.0, and it covers everything the loader did and more: -**Settings.json shape:** +| | `mcp-loader.ts` (retired) | pi built-in (`builtin:mcp`) | +|---|---|---| +| Config | `mcp` block in `settings.json` | `~/.pi/agent/mcp.json` (`mcpServers`, Claude Desktop shape); per-project `.pi/mcp.json` after trust | +| Manage | `/mcp` toggle overlay | `/mcp` manager (inspect tools, reconnect, sign in, enable/disable, exposure), `pi mcp add/remove/list/login/logout` from a shell | +| Auth | pre-issued bearer token in `headers` | headers with `${ENV}` / `!command` values, **and OAuth** (dynamic registration, CIMD, refresh) | +| Startup | awaited every server's `tools/list` while extensions loaded | connects in the background at session start; the first prompt waits (up to 10 s) only for servers with `direct` tools | +| Tools | every tool declared to the model as `_` | `mcp____`; per-server/per-tool `exposure`: `codemode` (default), `deferred`, `direct`, `hidden` | +| Also | — | resources (`list_mcp_resources`, `read_mcp_resource`), per-request `timeout`, reconnect on next call, `mcp.log` | -```jsonc +Measured before retiring it (pi 1.1.0, pi-devbox v1.11.0, 7 interleaved runs each, time to pi's `(auto)` footer in a real pty): `-ne` 0.276 s, `-ne -e mcp-loader.ts` 0.283 s, `-ne -e builtin:mcp` 0.272 s. With no servers configured, the built-in adds nothing to a request: the same prompt cost 79 input + 7680 cached tokens with either loader, and with neither. A stdio server (`npx -y @modelcontextprotocol/server-everything`) connected in 3 s via `pi mcp list` and its `echo` tool was called end to end from `pi -p`. + +**Why remove it rather than keep both:** an extension that registers `/mcp` *replaces* the built-in, so with the loader installed pi never read `mcp.json` and printed `Extension … registers command /mcp, so built-in extension mcp was not loaded` on every start. + +**Moving an old `mcp` block** to `~/.pi/agent/mcp.json` (pi `docs/mcp.md`, "Migrate configuration from another client"): + +```json { - // … existing pi settings … - "mcp": { - "searxng": { - "type": "local", - "command": ["uvx", "mcp-searxng"], - "env": { "SEARXNG_URL": "https://searxng.your-host.lan" } - }, - "gitea": { - "type": "local", - "command": ["gitea-mcp", "-t", "stdio"], - "enabled": false, - "env": { "GITEA_ACCESS_TOKEN": "...", "GITEA_HOST": "https://gitea.example.com" } - }, - "context7": { - "type": "remote", - "url": "https://mcp.context7.com/mcp" - } + "mcpServers": { + "searxng": { "command": "uvx", "args": ["mcp-searxng"], "env": { "SEARXNG_URL": "https://searxng.your-host.lan" }, "exposure": "direct" }, + "context7": { "url": "https://mcp.context7.com/mcp", "headers": { "Authorization": "Bearer ${CONTEXT7_TOKEN}" } } } } ``` -**Per-server keys:** +- `"type": "local"` + `"command": [exe, ...args]` becomes `"command": exe` + `"args": [...]`. `"type": "remote"` becomes a `url` entry. +- Set `"exposure": "direct"` to keep the loader's behaviour of declaring every tool to the model. The built-in default, `codemode`, makes tools reachable only from `codemode` scripts. +- Tool names change from `_` to `mcp____`. Non-`[A-Za-z0-9_]` characters still become `_`, so the names stay valid for AWS Bedrock. +- `pi mcp list` validates the file and connects every enabled server. -| Key | Description | -|---|---| -| `type` | `"local"` (stdio subprocess) or `"remote"` (streamable-http). Default `"local"`. | -| `command` | Argv array. First element is the executable, rest are args. Local servers only. | -| `url` | Remote MCP endpoint URL. Remote servers only. | -| `headers` | Optional object of HTTP headers (e.g. `Authorization`, `X-API-Key`) sent with every request. Remote servers only. | -| `enabled` | Default `true`. Set `false` to disable a server without removing the entry. | -| `env` | Optional object of env vars injected into the subprocess. Inherits parent env first, then overlays these keys. Local servers only. | +`install.sh` now removes the stale `~/.pi/agent/extensions/mcp-loader.ts` link (and any other link into this repo whose file is gone). It warns once if `settings.json` still has an `mcp` block. -**Transports:** - -- `local` — stdio JSON-RPC subprocess (mcp-searxng, gitea-mcp, mcp-server-time…). -- `remote` — streamable-HTTP per MCP spec 2025-03-26: POST JSON-RPC, server replies either `application/json` or `text/event-stream`. Optional `Mcp-Session-Id` round-trip if the server issues one. No GET subscription stream (server-initiated notifications are not consumed). - -**Limitations:** - -- **No stdio reconnect** if a subprocess dies mid-session — those tools become unavailable until `/reload` (same as `mempalace.ts`). -- **Remote sessions self-heal on 404.** If a streamable-HTTP server forgets our session id (e.g. server restart), the client transparently re-initializes and retries the request once. -- **No OAuth flow.** Remote servers requiring OAuth must be accessed with a pre-issued bearer token via `headers`. -- **Coexists with `mempalace.ts`** but does not replace it. The mempalace bridge has bespoke handling (agent identity injection) that's worth keeping. Don't list `mempalace` in the `mcp` block too — you'd get duplicate tool registrations. - -**Debug:** set `PI_MCP_LOADER_DEBUG=1` in the environment to surface per-server stderr and connection logs. - -**Slash command:** `/mcp` opens a multi-toggle overlay listing every server in the `mcp` block with its runtime status: - -- `running · N tools` — connected, tools registered -- `failed: ` — start handshake threw -- `disabled in settings` — `enabled: false` -- `invalid: ` — malformed config (read-only row) - -UX matches `/ext`: **space** stages a toggle, **enter** writes back to `settings.json` and reloads pi, **esc** cancels. Toggling re-enables a previously-disabled server by removing the explicit `enabled` key (the default is `true`). - -Each extension is a TypeScript module loaded by [jiti](https://github.com/unjs/jiti) — no compilation step. See the [pi extensions docs](https://github.com/earendil-works/pi/blob/main/docs/extensions.md) and the [built-in examples](https://github.com/earendil-works/pi/tree/main/examples/extensions) for the API surface. - ---- +**MemPalace is unaffected.** `mempalace.ts` (mempalace-toolkit) runs its own MCP client and registers `mempalace_*` tools directly, so `exposure` never applies to them. Don't add MemPalace to `mcp.json`: you would get a second, `mcp__mempalace__*`, copy of every tool. ### `aws-sso.ts` diff --git a/extensions/mcp-loader.ts b/extensions/mcp-loader.ts deleted file mode 100644 index 6df6ab8..0000000 --- a/extensions/mcp-loader.ts +++ /dev/null @@ -1,842 +0,0 @@ -/** - * MCP Loader Extension - * - * Reads an `mcp` block from pi's settings.json (same shape as opencode and - * Claude Desktop) and connects to each declared server, exposing all of - * their tools to pi as native tools. - * - * Settings.json shape: - * - * { - * "mcp": { - * "searxng": { - * "type": "local", - * "command": ["uvx", "mcp-searxng"], - * "env": { "SEARXNG_URL": "https://searxng.your-host.lan" } - * }, - * "gitea": { - * "type": "local", - * "command": ["gitea-mcp", "-t", "stdio"], - * "enabled": false, - * "env": { "GITEA_ACCESS_TOKEN": "...", "GITEA_HOST": "..." } - * }, - * "context7": { - * "type": "remote", - * "url": "https://mcp.context7.com/mcp", - * "headers": { "Authorization": "Bearer …" } - * } - * } - * } - * - * Per-server keys: - * type "local" (stdio subprocess) | "remote" (streamable-http) - * command argv array — first element is the executable, rest are args. - * For local servers only. - * url remote MCP endpoint URL. For remote servers only. - * headers optional object of HTTP headers to send with every request to - * a remote server (e.g. Authorization, X-API-Key). Remote only. - * enabled default true. Set false to disable a server without - * removing it from settings.json. - * env optional object of environment variables to inject into - * the spawned subprocess. Inherits parent env first, then - * overlays these keys. Local only. - * - * Tool naming - * - * Each MCP tool is registered with pi as `_` to - * avoid collisions across servers. If the tool name already begins with - * `_` (e.g. mempalace's tools are `mempalace_search` etc.) - * the prefix is not duplicated. - * - * Lifecycle - * - * • Servers are connected at extension load time (pi startup or /reload). - * • Each server's `tools/list` is awaited before pi finishes registering. - * • Local subprocess stderr is silenced unless PI_MCP_LOADER_DEBUG=1. - * • Local subprocesses receive SIGTERM at session_shutdown; remote - * servers receive an MCP DELETE if a session id was issued. - * • A server that fails to start (binary missing, init handshake error, - * remote 4xx/5xx) logs a single line to stderr and is skipped. Other - * servers continue. Pi keeps working without that server's tools. - * - * Transports - * - * local — stdio JSON-RPC subprocess (mcp-searxng, gitea-mcp, …). - * remote — streamable-HTTP per MCP spec 2025-03-26: POST JSON-RPC, server - * replies either application/json or text/event-stream. Optional - * Mcp-Session-Id header round-tripped if the server issues one. - * No GET-stream subscription (we don't consume server-initiated - * notifications). - * - * Limitations - * - * • No stdio reconnect: if a stdio subprocess dies mid-session, its tools - * become unavailable until pi is reloaded. The mempalace.ts extension - * has the same limitation today. - * • Remote sessions self-heal on 404: if a streamable-HTTP server - * forgets our session id (e.g. server restart), the next request gets - * HTTP 404; the client transparently re-initializes and retries the - * request once. Persistent 404s after refresh surface as errors. - * • No OAuth flow: remote servers requiring OAuth must be accessed with - * a pre-issued bearer token via `headers`. - * - * Coexistence with mempalace.ts - * - * The mempalace.ts extension already spawns mempalace-mcp directly with - * bespoke handling (agent identity injection from $MEMPALACE_AGENT_NAME, - * wake-up context). This loader does NOT touch mempalace — leave the - * bespoke extension in place. If you also list mempalace under the `mcp` - * block in settings.json the loader will register a parallel set of - * tools, which is harmless but redundant. Don't. - * - * Slash command - * - * `/mcp` lists configured MCP servers with their runtime status - * (running / failed / disabled / invalid) and lets - * you toggle the `enabled` flag in settings.json. Same UX as `/ext`: - * space stages, enter applies + reloads, esc cancels. - */ - -import { type ChildProcessWithoutNullStreams, spawn } from "node:child_process"; -import * as fs from "node:fs"; -import * as os from "node:os"; -import * as path from "node:path"; -import { - type ExtensionAPI, - getSettingsListTheme, -} from "@earendil-works/pi-coding-agent"; -import { - Container, - Key, - matchesKey, - type SettingItem, - SettingsList, -} from "@earendil-works/pi-tui"; -import { Type } from "typebox"; - -// ── MCP types ──────────────────────────────────────────────────────────────── - -type McpTool = { - name: string; - description?: string; - inputSchema?: Record; -}; - -type LocalServerConfig = { - type?: "local"; - command: string[]; - env?: Record; - enabled?: boolean; -}; - -type RemoteServerConfig = { - type: "remote"; - url: string; - headers?: Record; - enabled?: boolean; -}; - -type ServerConfig = LocalServerConfig | RemoteServerConfig; - -const SETTINGS_PATH = path.join(os.homedir(), ".pi", "agent", "settings.json"); -const DEBUG = process.env.PI_MCP_LOADER_DEBUG === "1"; - -// MCP protocol version we advertise. Servers we've tested: -// - searxng (mcp-searxng) accepts both 2024-11-05 and 2025-11-25 -// - context7 accepts both -// - mempalace bridge: not touched by this loader -// Bump as the spec evolves; both stdio and remote clients use this. -const MCP_PROTOCOL_VERSION = "2025-11-25"; - -function dlog(msg: string) { - if (DEBUG) process.stderr.write(`[mcp-loader] ${msg}\n`); -} -function warn(msg: string) { - process.stderr.write(`[mcp-loader] ${msg}\n`); -} - -// ── Settings.json reader ───────────────────────────────────────────────────── - -function readMcpServers(): Record { - if (!fs.existsSync(SETTINGS_PATH)) { - dlog(`no settings.json at ${SETTINGS_PATH}`); - return {}; - } - let raw: string; - try { - raw = fs.readFileSync(SETTINGS_PATH, "utf8"); - } catch (err) { - warn(`could not read ${SETTINGS_PATH}: ${(err as Error).message}`); - return {}; - } - let parsed: any; - try { - parsed = JSON.parse(raw); - } catch (err) { - warn(`settings.json parse error: ${(err as Error).message}`); - return {}; - } - const block = parsed?.mcp; - if (!block || typeof block !== "object") { - dlog("no `mcp` block in settings.json"); - return {}; - } - return block as Record; -} - -// ── Stdio MCP client ───────────────────────────────────────────────────────── - -interface IMcpClient { - readonly serverName: string; - tools: McpTool[]; - start(): Promise; - callTool(name: string, args: Record): Promise; - stop(): void | Promise; -} - -class StdioMcpClient implements IMcpClient { - private proc: ChildProcessWithoutNullStreams | null = null; - private nextId = 1; - private pending = new Map void; reject: (e: Error) => void }>(); - private stdoutBuf = ""; - public readonly serverName: string; - private command: string; - private args: string[]; - private env: Record | undefined; - public tools: McpTool[] = []; - - constructor(serverName: string, command: string, args: string[], env?: Record) { - this.serverName = serverName; - this.command = command; - this.args = args; - this.env = env; - } - - async start(): Promise { - const childEnv = { ...process.env, ...(this.env ?? {}) }; - this.proc = spawn(this.command, this.args, { stdio: ["pipe", "pipe", "pipe"], env: childEnv }); - - this.proc.on("error", (err) => this.failAll(err)); - this.proc.on("exit", (code, signal) => { - this.failAll(new Error(`${this.serverName}: subprocess exited (code=${code}, signal=${signal})`)); - }); - - this.proc.stdout.setEncoding("utf8"); - this.proc.stdout.on("data", (chunk: string) => this.onStdout(chunk)); - - this.proc.stderr.setEncoding("utf8"); - if (DEBUG) { - this.proc.stderr.on("data", (chunk: string) => { - process.stderr.write(`[${this.serverName} stderr] ${chunk}`); - }); - } else { - this.proc.stderr.resume(); // drain without logging - } - - // MCP initialize handshake - await this.request("initialize", { - protocolVersion: MCP_PROTOCOL_VERSION, - capabilities: {}, - clientInfo: { name: "pi-mcp-loader", version: "0.2.0" }, - }); - this.notify("notifications/initialized", {}); - - const listed = await this.request("tools/list", {}); - this.tools = (listed?.tools as McpTool[]) ?? []; - } - - private onStdout(chunk: string) { - this.stdoutBuf += chunk; - let nl: number; - while ((nl = this.stdoutBuf.indexOf("\n")) !== -1) { - const line = this.stdoutBuf.slice(0, nl).trim(); - this.stdoutBuf = this.stdoutBuf.slice(nl + 1); - if (!line) continue; - let msg: any; - try { - msg = JSON.parse(line); - } catch { - continue; - } - if (typeof msg.id === "number" && this.pending.has(msg.id)) { - const { resolve, reject } = this.pending.get(msg.id)!; - this.pending.delete(msg.id); - if (msg.error) reject(new Error(msg.error.message ?? "MCP error")); - else resolve(msg.result); - } - // notifications (no id) ignored. - } - } - - private failAll(err: Error) { - for (const { reject } of this.pending.values()) reject(err); - this.pending.clear(); - } - - private write(obj: unknown) { - if (!this.proc) throw new Error(`${this.serverName}: not started`); - this.proc.stdin.write(`${JSON.stringify(obj)}\n`); - } - - private notify(method: string, params: unknown) { - this.write({ jsonrpc: "2.0", method, params }); - } - - request(method: string, params: unknown): Promise { - const id = this.nextId++; - return new Promise((resolve, reject) => { - this.pending.set(id, { resolve, reject }); - this.write({ jsonrpc: "2.0", id, method, params }); - }); - } - - async callTool(name: string, args: Record): Promise { - return this.request("tools/call", { name, arguments: args }); - } - - stop() { - if (this.proc) { - try { - this.proc.kill("SIGTERM"); - } catch {} - this.proc = null; - } - } -} - -// ── Streamable-HTTP MCP client (remote) ────────────────────────────────────── -// -// MCP-STREAMABLE-HTTP-CLIENT-SYNC: v1 -// Canonical copy. This class is VENDORED into -// mempalace-toolkit/extensions/pi/mempalace.ts (adapted: pinned protocolVersion, -// AbortController timeouts, alive/ensureAlive). When you change the -// streamable-HTTP protocol handling here, bump the SYNC token in BOTH files. -// mempalace-toolkit/scripts/check-mcp-client-sync.sh enforces the match. -// -// Per MCP spec 2025-03-26: a single endpoint URL accepts POST with a -// JSON-RPC body. The server responds either with application/json (a -// single response) or text/event-stream (an SSE stream containing the -// response, possibly preceded by notifications, terminated by stream -// close). We do NOT open a separate GET subscription stream — we don't -// consume server-initiated notifications today. -// -// Session id: if the server returns an Mcp-Session-Id header on -// initialize, we round-trip it on every subsequent request and DELETE -// it on stop(). - -class RemoteMcpClient implements IMcpClient { - public readonly serverName: string; - private url: string; - private extraHeaders: Record; - private sessionId: string | null = null; - private nextId = 1; - public tools: McpTool[] = []; - - constructor(serverName: string, url: string, headers?: Record) { - this.serverName = serverName; - this.url = url; - this.extraHeaders = headers ?? {}; - } - - async start(): Promise { - await this.request("initialize", { - protocolVersion: MCP_PROTOCOL_VERSION, - capabilities: {}, - clientInfo: { name: "pi-mcp-loader", version: "0.2.0" }, - }); - await this.notify("notifications/initialized", {}); - const listed = await this.request("tools/list", {}); - this.tools = (listed?.tools as McpTool[]) ?? []; - } - - async callTool(name: string, args: Record): Promise { - return this.request("tools/call", { name, arguments: args }); - } - - async stop(): Promise { - if (!this.sessionId) return; - try { - await fetch(this.url, { method: "DELETE", headers: this.buildHeaders() }); - } catch {} - } - - private buildHeaders(): Record { - const h: Record = { - "Content-Type": "application/json", - Accept: "application/json, text/event-stream", - "MCP-Protocol-Version": MCP_PROTOCOL_VERSION, - ...this.extraHeaders, - }; - if (this.sessionId) h["Mcp-Session-Id"] = this.sessionId; - return h; - } - - /** - * Re-establish a session after the server returned 404 to a request that - * carried our `Mcp-Session-Id`. Per spec 2025-11-25, the client MUST start - * a new session by sending a fresh InitializeRequest without a session id. - * - * Caller must clear `this.sessionId` BEFORE invoking this so the new - * initialize POST goes out without the stale id. - */ - private async reinitialize(): Promise { - dlog(`${this.serverName}: session lost (404) — reinitializing`); - await this.request("initialize", { - protocolVersion: MCP_PROTOCOL_VERSION, - capabilities: {}, - clientInfo: { name: "pi-mcp-loader", version: "0.2.0" }, - }, /*allowReinitOn404=*/ false); - await this.notify("notifications/initialized", {}); - } - - private async request(method: string, params: unknown, allowReinitOn404 = true): Promise { - const id = this.nextId++; - const body = JSON.stringify({ jsonrpc: "2.0", id, method, params }); - const sessionAtStart = this.sessionId; - let res: Response; - try { - res = await fetch(this.url, { method: "POST", headers: this.buildHeaders(), body }); - } catch (err) { - throw new Error(`${this.serverName}: fetch failed on ${method}: ${(err as Error).message}`); - } - - const sid = res.headers.get("Mcp-Session-Id") ?? res.headers.get("mcp-session-id"); - if (sid) this.sessionId = sid; - - // 404 + we sent a session id → server forgot us. Drop our id, re-init, - // retry once. `allowReinitOn404=false` is passed by the recovery path to - // prevent infinite recursion if the server is permanently broken. - if (res.status === 404 && sessionAtStart && allowReinitOn404) { - try { await res.arrayBuffer(); } catch {} - this.sessionId = null; - await this.reinitialize(); - return this.request(method, params, /*allowReinitOn404=*/ false); - } - - if (!res.ok) { - let detail = ""; - try { detail = (await res.text()).slice(0, 200); } catch {} - throw new Error(`${this.serverName}: HTTP ${res.status} on ${method}${detail ? `: ${detail}` : ""}`); - } - - const ct = (res.headers.get("content-type") ?? "").toLowerCase(); - if (ct.includes("text/event-stream")) { - return await this.readSseForResponse(res, id); - } - if (ct.includes("application/json")) { - const json: any = await res.json(); - if (json.error) throw new Error(json.error.message ?? "MCP error"); - return json.result; - } - if (res.status === 202) return undefined; - throw new Error(`${this.serverName}: unexpected content-type "${ct}" on ${method}`); - } - - private async notify(method: string, params: unknown): Promise { - const body = JSON.stringify({ jsonrpc: "2.0", method, params }); - let res: Response; - try { - res = await fetch(this.url, { method: "POST", headers: this.buildHeaders(), body }); - } catch (err) { - throw new Error(`${this.serverName}: fetch failed on notify ${method}: ${(err as Error).message}`); - } - try { await res.arrayBuffer(); } catch {} - if (!res.ok && res.status !== 202) { - throw new Error(`${this.serverName}: HTTP ${res.status} on notify ${method}`); - } - } - - private async readSseForResponse(res: Response, expectedId: number): Promise { - if (!res.body) throw new Error(`${this.serverName}: SSE response had no body`); - const reader = res.body.getReader(); - const decoder = new TextDecoder(); - let buf = ""; - let dataLines: string[] = []; - - const tryDispatch = (): { matched: boolean; result?: any } => { - if (dataLines.length === 0) return { matched: false }; - const data = dataLines.join("\n"); - dataLines = []; - let msg: any; - try { - msg = JSON.parse(data); - } catch { - return { matched: false }; - } - if (typeof msg.id === "number" && msg.id === expectedId) { - if (msg.error) throw new Error(msg.error.message ?? "MCP error"); - return { matched: true, result: msg.result }; - } - return { matched: false }; - }; - - try { - while (true) { - const { done, value } = await reader.read(); - if (done) break; - buf += decoder.decode(value, { stream: true }); - let nl: number; - while ((nl = buf.indexOf("\n")) !== -1) { - const rawLine = buf.slice(0, nl); - buf = buf.slice(nl + 1); - const line = rawLine.endsWith("\r") ? rawLine.slice(0, -1) : rawLine; - if (line === "") { - const out = tryDispatch(); - if (out.matched) return out.result; - } else if (line.startsWith(":")) { - // SSE comment — ignore (often a keepalive) - } else if (line.startsWith("data:")) { - dataLines.push(line.slice(5).replace(/^ /, "")); - } - // event:, id:, retry: — ignored; only `data` matters for our use - } - } - const out = tryDispatch(); - if (out.matched) return out.result; - } finally { - try { reader.cancel(); } catch {} - } - throw new Error(`${this.serverName}: SSE stream closed without response for id=${expectedId}`); - } -} - -// ── MCP result → pi tool result helpers ────────────────────────────────────── - -function extractText(mcpResult: any): string { - if (!mcpResult) return ""; - const content = mcpResult.content; - if (!Array.isArray(content)) return JSON.stringify(mcpResult); - return content - .map((c: any) => { - if (c?.type === "text" && typeof c.text === "string") return c.text; - return JSON.stringify(c); - }) - .join("\n"); -} - -function namespacedToolName(serverName: string, toolName: string): string { - // Avoid double-prefix: if the MCP server already names its tools - // `_<...>`, leave them alone. - const prefix = `${serverName}_`; - const raw = toolName.startsWith(prefix) ? toolName : `${prefix}${toolName}`; - // Sanitize for the strictest tool-name regex we have to satisfy. - // Anthropic's Messages API allows `^[a-zA-Z0-9_-]{1,64}$` (hyphens OK), - // but AWS Bedrock's Anthropic shim rejects names containing `-` outright - // — it accepts `^[a-zA-Z][a-zA-Z0-9_]{0,63}$`. Hyphenated MCP tool names - // (context7's `resolve-library-id`, `query-docs`) caused entire turns to - // 4xx silently on Bedrock, manifesting as no output at all once the - // server was enabled. Replace any non-[A-Za-z0-9_] with `_`, then prefix - // a leading underscore with `t` if the result starts with a non-letter. - // Truncate to 64 to also satisfy Anthropic's length cap. - let sanitized = raw.replace(/[^A-Za-z0-9_]/g, "_"); - if (!/^[A-Za-z]/.test(sanitized)) sanitized = `t_${sanitized}`; - if (sanitized.length > 64) sanitized = sanitized.slice(0, 64); - return sanitized; -} - -// ── Settings.json writer ───────────────────────────────────────────────────── - -/** - * Flip the `enabled` flag for a server in settings.json and write back. - * Preserves all other keys verbatim by round-tripping through JSON with - * 2-space indent (matches the file's existing style). - */ -function setServerEnabled(serverName: string, enabled: boolean): void { - const raw = fs.readFileSync(SETTINGS_PATH, "utf8"); - const parsed = JSON.parse(raw); - if (!parsed.mcp || !parsed.mcp[serverName]) { - throw new Error(`server '${serverName}' not in settings.json`); - } - if (enabled) { - // Default is true — clear the explicit flag if present, keep the file tidy. - delete parsed.mcp[serverName].enabled; - } else { - parsed.mcp[serverName].enabled = false; - } - fs.writeFileSync(SETTINGS_PATH, `${JSON.stringify(parsed, null, 2)}\n`, "utf8"); -} - -// ── Per-server runtime state (for /mcp display) ────────────────────────────── - -type ServerRuntimeState = - | { kind: "running"; toolCount: number } - | { kind: "failed"; message: string } - | { kind: "disabled" } - | { kind: "invalid"; message: string }; - -// ── Extension entry ────────────────────────────────────────────────────────── - -export default async function mcpLoaderExtension(pi: ExtensionAPI) { - const servers = readMcpServers(); - const runtime = new Map(); - const clients: IMcpClient[] = []; - - // Always register /mcp — even when no servers are configured, so users - // discover the surface. Handler reads settings.json fresh on each invoke. - registerMcpCommand(pi, runtime); - - if (Object.keys(servers).length === 0) { - dlog("no MCP servers configured — nothing to load"); - return; - } - - for (const [name, cfg] of Object.entries(servers)) { - if (cfg.enabled === false) { - dlog(`${name}: disabled in settings.json, skipping`); - runtime.set(name, { kind: "disabled" }); - continue; - } - - let client: IMcpClient; - if (cfg.type === "remote") { - const remote = cfg as RemoteServerConfig; - if (typeof remote.url !== "string" || remote.url.length === 0) { - warn(`${name}: remote server missing \`url\`, skipping`); - runtime.set(name, { kind: "invalid", message: "remote server missing url" }); - continue; - } - client = new RemoteMcpClient(name, remote.url, remote.headers); - } else { - const local = cfg as LocalServerConfig; - if (!Array.isArray(local.command) || local.command.length === 0) { - warn(`${name}: invalid \`command\` (must be non-empty array), skipping`); - runtime.set(name, { kind: "invalid", message: "command must be non-empty array" }); - continue; - } - const [bin, ...args] = local.command; - client = new StdioMcpClient(name, bin, args, local.env); - } - - try { - await client.start(); - } catch (err) { - const message = (err as Error).message; - warn(`${name}: failed to start — ${message}`); - runtime.set(name, { kind: "failed", message }); - try { await client.stop(); } catch {} - continue; - } - - dlog(`${name}: connected, ${client.tools.length} tools`); - runtime.set(name, { kind: "running", toolCount: client.tools.length }); - clients.push(client); - - // Register each MCP tool as a pi tool. - for (const tool of client.tools) { - const piName = namespacedToolName(name, tool.name); - const schema = - tool.inputSchema && typeof tool.inputSchema === "object" - ? (Type.Unsafe>( - tool.inputSchema as object, - ) as unknown as ReturnType) - : Type.Object({}, { additionalProperties: true }); - - pi.registerTool({ - name: piName, - label: piName, - description: tool.description ?? `MCP tool from ${name}`, - parameters: schema, - async execute(_toolCallId, params) { - try { - const result = await client.callTool(tool.name, params as Record); - const text = extractText(result); - return { - content: [{ type: "text", text }], - details: { server: name, tool: tool.name, raw: result }, - }; - } catch (err) { - return { - content: [ - { - type: "text", - text: `MCP call failed (${name}/${tool.name}): ${(err as Error).message}`, - }, - ], - details: { server: name, tool: tool.name, error: (err as Error).message }, - isError: true, - }; - } - }, - }); - } - } - - // Tear down clients (stdio subprocesses, remote sessions) on session shutdown. - pi.on("session_shutdown", async () => { - await Promise.allSettled(clients.map((c) => Promise.resolve(c.stop()))); - }); -} - -// ── /mcp slash command ────────────────────────────────────────────────────── - -const VAL_ON = "● enabled"; -const VAL_OFF = "○ disabled"; -const VAL_RO = "[read-only]"; - -function statusLabel(state: ServerRuntimeState | undefined, configEnabled: boolean): string { - if (!state) return configEnabled ? "unknown" : "disabled"; - switch (state.kind) { - case "running": - return `running · ${state.toolCount} tool${state.toolCount === 1 ? "" : "s"}`; - case "failed": - return `failed: ${state.message}`; - case "disabled": - return "disabled in settings"; - case "invalid": - return `invalid: ${state.message}`; - } -} - -function registerMcpCommand(pi: ExtensionAPI, runtime: Map) { - pi.registerCommand("mcp", { - description: "List and toggle MCP servers configured in ~/.pi/agent/settings.json", - handler: async (_args, ctx) => { - const servers = readMcpServers(); - const names = Object.keys(servers); - if (names.length === 0) { - ctx.ui.notify( - `No MCP servers configured. Add an \`mcp\` block to ${SETTINGS_PATH}.`, - "info", - ); - return; - } - - // Snapshot config-level enabled state (default true if unset). - const configEnabled = new Map(); - for (const [n, cfg] of Object.entries(servers)) { - configEnabled.set(n, cfg.enabled !== false); - } - - // Staged state: name → enabled (initialised from configEnabled). - const staged = new Map(configEnabled); - - await ctx.ui.custom((tui, theme, _kb, done) => { - let statusText = ""; - - const items: SettingItem[] = names.map((n) => { - const cfg = servers[n]; - const cfgEn = configEnabled.get(n) ?? true; - const state = runtime.get(n); - const isRemote = cfg.type === "remote"; - const isInvalid = state?.kind === "invalid"; - const transport = isRemote ? "remote" : "local"; - const desc = `${transport} · ${statusLabel(state, cfgEn)}`; - - if (isInvalid) { - return { - id: n, - label: n, - currentValue: VAL_RO, - description: desc, - }; - } - return { - id: n, - label: n, - currentValue: cfgEn ? VAL_ON : VAL_OFF, - values: [VAL_ON, VAL_OFF], - description: desc, - }; - }); - - const container = new Container(); - - container.addChild({ - render(_w: number) { - return [ - theme.fg("accent", theme.bold("MCP servers — settings.json `mcp` block")), - theme.fg( - "muted", - " space: toggle (stage) · enter: apply + reload · esc: cancel", - ), - "", - ]; - }, - invalidate() {}, - }); - - const settingsList = new SettingsList( - items, - Math.min(items.length + 2, 20), - getSettingsListTheme(), - (id, newValue) => { - const isInvalid = runtime.get(id)?.kind === "invalid"; - if (isInvalid) { - settingsList.updateValue(id, VAL_RO); - statusText = `⊘ ${id}: invalid config — fix in settings.json`; - tui.requestRender(); - return; - } - const stagingEnabled = newValue === VAL_ON; - staged.set(id, stagingEnabled); - - const drift: string[] = []; - for (const n of names) { - const s = staged.get(n); - const c = configEnabled.get(n); - if (s !== undefined && s !== c) drift.push(`${n}→${s ? "on" : "off"}`); - } - statusText = drift.length ? `pending: ${drift.join(", ")}` : ""; - tui.requestRender(); - }, - () => { - done(undefined); - }, - ); - - container.addChild(settingsList); - - container.addChild({ - render(_w: number) { - if (!statusText) return [""]; - const colored = statusText.startsWith("⊘") - ? theme.fg("warning", statusText) - : theme.fg("muted", statusText); - return ["", colored]; - }, - invalidate() {}, - }); - - return { - render(width: number) { - return container.render(width); - }, - invalidate() { - container.invalidate(); - }, - handleInput(data: string) { - if (matchesKey(data, Key.enter)) { - const errors: string[] = []; - let applied = 0; - for (const n of names) { - const s = staged.get(n); - const c = configEnabled.get(n); - if (s === undefined || s === c) continue; - try { - setServerEnabled(n, s); - applied++; - } catch (err) { - const msg = err instanceof Error ? err.message : String(err); - errors.push(`${n}: ${msg}`); - } - } - done(undefined); - if (errors.length) { - ctx.ui.notify(`mcp: ${errors.join(" | ")}`, "error"); - } else if (applied > 0) { - ctx.ui.notify(`mcp: applied ${applied} change(s); reloading…`, "info"); - ctx.reload().catch((err) => { - const msg = err instanceof Error ? err.message : String(err); - ctx.ui.notify(`reload failed: ${msg}`, "error"); - }); - } - return; - } - settingsList.handleInput?.(data); - tui.requestRender(); - }, - }; - }); - }, - }); -} diff --git a/install.sh b/install.sh index e0feffe..d19c643 100755 --- a/install.sh +++ b/install.sh @@ -120,6 +120,41 @@ build_install_set() { fi } +# ── stale links ────────────────────────────────────── +# An extension removed from this repo leaves its symlink behind in +# ~/.pi/agent/extensions/ — on a pi-devbox volume that link survives every +# recreate, pointing into an image that no longer has the file. Remove links +# that point into THIS repo and whose target is gone (.ts and the /ext-disabled +# .ts.off alike). Links into other repos and real files are never touched. +prune_stale_links() { + local dest pattern + for pattern in "*.ts" "*.ts.off"; do + for dest in "${EXTENSIONS_DEST}"/$pattern; do + [[ -L "$dest" ]] || continue + [[ -e "$dest" ]] && continue # target still exists + link_into_repo "$dest" || continue # someone else's link + rm "$dest" + ok "Removed stale $(basename "$dest") (no longer in this repo)" + done + done +} + +# mcp-loader.ts was retired in favour of pi's built-in MCP support. Its config +# lived in settings.json under a top-level `mcp` key, which pi ignores; say so +# once instead of letting those servers disappear silently. +warn_retired_mcp_block() { + local settings="${PI_AGENT_DIR}/settings.json" has="" + [[ -f "$settings" ]] || return 0 + if command -v jq >/dev/null 2>&1; then + has=$(jq -r 'if type == "object" and has("mcp") then "yes" else "" end' "$settings" 2>/dev/null || true) + elif command -v python3 >/dev/null 2>&1; then + has=$(python3 -c 'import json,sys; d=json.load(open(sys.argv[1])); print("yes" if isinstance(d, dict) and "mcp" in d else "")' "$settings" 2>/dev/null || true) + fi + [[ "$has" == "yes" ]] || return 0 + warn "${settings} has an 'mcp' block, but mcp-loader.ts is retired and pi ignores it." + warn " Move those servers to ~/.pi/agent/mcp.json (pi's built-in MCP; see README 'MCP servers')." +} + # ── install ────────────────────────────────────────── do_install() { echo @@ -128,6 +163,7 @@ do_install() { echo require_pi_installed + warn_retired_mcp_block build_install_set if [[ -z "$INSTALL_SET" ]]; then @@ -143,6 +179,8 @@ do_install() { confirm || { echo "Aborted."; exit 0; } echo + prune_stale_links + for src in "${EXTENSIONS_SRC}"/*.ts; do [[ -e "$src" ]] || continue local name diff --git a/skill/SKILL.md b/skill/SKILL.md index 7ec1750..b7b4ca7 100644 --- a/skill/SKILL.md +++ b/skill/SKILL.md @@ -23,11 +23,13 @@ Pi has **two distinct extension locations** and it's easy to look in the wrong o | Location | Mechanism | Examples | |---|---|---| -| `~/.pi/agent/extensions/*.ts` (or `.ts.off`) | **Local extensions** — TypeScript files, usually symlinks into `/opt/pi-extensions/extensions/` or similar. Toggled via `/ext` slash command. | `ssh-controlmaster`, `git-checkpoint`, `notify`, `todo`, `mempalace`, `mcp-loader`, `ext-toggle`, `confirm-destructive` | +| `~/.pi/agent/extensions/*.ts` (or `.ts.off`) | **Local extensions** — TypeScript files, usually symlinks into `/opt/pi-extensions/extensions/` or similar. Toggled via `/ext` slash command. | `ssh-controlmaster`, `git-checkpoint`, `notify`, `todo`, `mempalace`, `ext-toggle`, `task`, `fork-gate`, `secret-guard`, `aws-sso`, `confirm-destructive` | | `~/.pi/agent/git////` | **Package extensions (git-installed)** — git-cloned npm packages registered via the `packages` array in `~/.pi/agent/settings.json`. | `pi-fork` (`github.com/elpapi42/pi-fork`), `pi-observational-memory` (`github.com/elpapi42/pi-observational-memory`, **default branch `master`** — a `main` branch does not exist, so `pi install git:...` resolves against `master`) | | `~/.pi/agent/npm/node_modules//` | **Package extensions (npm-installed)** — `pi install npm:`; recorded in `packages[]` as `npm:`. | `pi-atelier` (status rail + sidebar TUI) | | `/opt//` — **pi-devbox containers only** | **Vendored package extensions** — cloned into an image layer at build time with `node_modules` baked, then registered at container start by `entrypoint-user.sh` via `pi install /opt/`. Recorded in `packages[]` as a **relative** path (`../../../../opt/pi-fork`) that resolves out of `~/.pi/agent` into the image layer, so it survives volume recreate. | `/opt/pi-fork`, `/opt/pi-observational-memory`, `/opt/pi-studio` | +**MCP servers are not an extension here.** pi's built-in MCP support (`builtin:mcp`) reads `~/.pi/agent/mcp.json`; manage it with `/mcp` or `pi mcp add|list`. Its tools are named `mcp____` and default to `codemode` exposure (not declared to the model) unless the server sets `"exposure": "direct"`. The `mempalace_*` tools are different: `mempalace.ts` registers them itself, so exposure never applies. The old `mcp-loader.ts` (an `mcp` block in `settings.json`) was retired on 2026-10-12, because an extension that registers `/mcp` replaces the built-in. + When the user asks how to use "the X extension", **check all of these** — `find ~/.pi/agent -maxdepth 4 -name "*X*"` covers the first three, and `ls -d /opt/*X*` the fourth. The `/ext` slash command shows the local-extensions list with enable/disable state. There is also a distinct skill-bundled-script category (e.g. `ci-release-watcher`'s `ssh-control-master-setup.sh`) which is **not** a pi extension at all — it's a helper script inside a skill. Don't conflate the three. **In a pi-devbox container, do not conclude "pi-fork isn't installed" because `~/.pi/agent/git/` is empty.** It is deliberately absent: `Dockerfile.variant` vendors to `/opt` and installs by local path, because a build-time `pi install git:...` would write into `~/.pi/agent`, which the named volume then shadows on first run.