diff --git a/learn/09-ui-architecture.md b/learn/09-ui-architecture.md index 0ecd055..ae2ea5c 100644 --- a/learn/09-ui-architecture.md +++ b/learn/09-ui-architecture.md @@ -106,4 +106,4 @@ Custom hooks encapsulate complex logic: --- -**Previous:** [← API Client](./08-api-client.md) +**Previous:** [<- API Client](./08-api-client.md) | **Next:** [Configuration and System Prompt ->](./10-configuration-and-system-prompt.md) diff --git a/learn/10-configuration-and-system-prompt.md b/learn/10-configuration-and-system-prompt.md new file mode 100644 index 0000000..4f7aaea --- /dev/null +++ b/learn/10-configuration-and-system-prompt.md @@ -0,0 +1,253 @@ +# 10. Configuration and System Prompt + +> How settings cascade from enterprise policy down to project CLAUDE.md, and how the system prompt is assembled. + +--- + +## Configuration Hierarchy + +Claude Code loads settings from **5 layers**, each with strict priority. Higher layers override lower ones. + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#4a9eff', 'primaryBorderColor': '#4a9eff'}}}%% +flowchart TD + subgraph Sources["Configuration Sources — Highest to Lowest Priority"] + direction TB + ENT["Enterprise / MDM Policy
managed-settings.json
Org-level, cannot override"]:::ent + USER["User Settings
~/.claude/settings.json
Per-user global defaults"]:::user + PROJ["Project Settings
.claude/settings.json
Per-repo configuration"]:::proj + MCP_CFG[".mcp.json
Project MCP servers
Committed to repo"]:::proj + CLAUDE_MD["CLAUDE.md files
Instructions, rules,
memory for the model"]:::md + end + + ENT --> MERGE + USER --> MERGE + PROJ --> MERGE + MCP_CFG --> MERGE + CLAUDE_MD --> MERGE + + MERGE["getInitialSettings
Merge all layers
Higher priority wins"]:::merge + + APPSTATE["AppState.settings
Runtime configuration"]:::state + + MERGE --> APPSTATE + + classDef ent fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px + classDef user fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef proj fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef md fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef merge fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef state fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +### Enterprise / MDM Policy + +Highest priority. Set by organization admins via Mobile Device Management (macOS profiles). Stored at a managed file path. Users **cannot** override these settings. Controls things like: +- Allowed/denied MCP servers +- Permission mode restrictions +- Feature availability +- Analytics opt-out + +### User Settings (`~/.claude/settings.json`) + +Per-user defaults. Controls preferences like allowed tools, custom deny rules, and hooks. + +### Project Settings (`.claude/settings.json`) + +Per-repository settings. Committed to the repo so all collaborators share the same configuration. + +### `.mcp.json` + +MCP server definitions for the project. Lives at the repo root. Defines which MCP servers are available. Validated via Zod schema (`McpServerConfigSchema`). + +### CLAUDE.md + +Markdown instruction files that the model reads as part of its system prompt. These are the "memory" files that tell Claude about this specific project. + +--- + +## CLAUDE.md Discovery + +CLAUDE.md files are discovered from multiple locations and merged: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#6f42c1', 'primaryBorderColor': '#6f42c1'}}}%% +flowchart TD + subgraph Discovery["CLAUDE.md Discovery — getMemoryFiles"] + GLOBAL["~/.claude/CLAUDE.md
Global instructions"]:::global + CWD_WALK["Walk from cwd up to git root
Check each dir for CLAUDE.md
and .claude/CLAUDE.md"]:::walk + ADDITIONAL["--add-dir paths
Extra directories to scan"]:::additional + end + + FILTER["filterInjectedMemoryFiles
Remove duplicates,
check setting sources"]:::filter + + PARSE["getClaudeMds
Read and concatenate
all discovered files"]:::parse + + CACHE["setCachedClaudeMdContent
Cache for auto-mode
classifier to read"]:::cache + + INJECT["Injected into getUserContext
becomes part of system prompt"]:::inject + + Discovery --> FILTER --> PARSE --> CACHE --> INJECT + + classDef global fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef walk fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef additional fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef filter fill:#3d2b00,stroke:#fd7e14,color:#e0e0e0,stroke-width:2px + classDef parse fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef cache fill:#333,stroke:#888,color:#e0e0e0,stroke-width:1px + classDef inject fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +The `--bare` flag skips auto-discovery but still honors explicit `--add-dir` paths. + +--- + +## System Prompt Assembly + +The system prompt is what the model "sees" before any conversation messages. It's assembled from multiple pieces: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#e83e8c', 'primaryBorderColor': '#e83e8c'}}}%% +flowchart TD + subgraph Priority["System Prompt Priority — buildEffectiveSystemPrompt"] + OVERRIDE["1. Override System Prompt
Set via loop mode
REPLACES everything"]:::override + COORD["2. Coordinator System Prompt
If coordinator mode active"]:::coord + AGENT["3. Agent System Prompt
If mainThreadAgentDefinition set
In proactive mode: APPENDED
Otherwise: REPLACES default"]:::agent + CUSTOM["4. Custom System Prompt
Via --system-prompt flag"]:::custom + DEFAULT["5. Default System Prompt
The standard Claude Code prompt"]:::default + end + + APPEND["appendSystemPrompt
Always appended at end
except when override is set"]:::append + + EFFECTIVE["Effective System Prompt
Sent as system param
in API request"]:::result + + OVERRIDE -->|"if set"| EFFECTIVE + COORD -->|"if coordinator"| EFFECTIVE + AGENT -->|"if agent defined"| EFFECTIVE + CUSTOM -->|"if --system-prompt"| EFFECTIVE + DEFAULT -->|"fallback"| EFFECTIVE + APPEND --> EFFECTIVE + + classDef override fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px + classDef coord fill:#3d2b00,stroke:#fd7e14,color:#e0e0e0,stroke-width:2px + classDef agent fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef custom fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef default fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef append fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef result fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +### System Prompt Sections + +Individual sections of the system prompt are defined via `systemPromptSection()`: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#4a9eff', 'primaryBorderColor': '#4a9eff'}}}%% +flowchart LR + subgraph Sections["System Prompt Sections"] + CACHED["Cached Sections
Computed once at start
Reused every turn
Cleared on /clear or /compact"]:::cached + VOLATILE["DANGEROUS Uncached Sections
Recomputed every turn
BREAKS prompt cache"]:::volatile + end + + RESOLVE["resolveSystemPromptSections
Check cache, compute if needed"]:::resolve + + PROMPT["Final system prompt string"]:::result + + CACHED --> RESOLVE + VOLATILE --> RESOLVE + RESOLVE --> PROMPT + + classDef cached fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef volatile fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px + classDef resolve fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef result fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px +``` + +Most sections are **cached** (computed once, reused every turn) to preserve prompt cache hits. Volatile sections that recompute every turn are explicitly named `DANGEROUS_uncachedSystemPromptSection` as a warning. + +--- + +## Context Injection + +Beyond the system prompt, two context objects are injected into every conversation: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#28a745', 'primaryBorderColor': '#28a745'}}}%% +flowchart LR + subgraph SystemCtx["getSystemContext — memoized"] + GIT["Git status snapshot
branch, status, log,
default branch, user"] + INJECT_SYS["System prompt injection
internal debugging only"] + end + + subgraph UserCtx["getUserContext — memoized"] + CMD["CLAUDE.md content
all discovered files
concatenated"] + DATE["Current date
ISO format"] + end + + PREPEND["prependUserContext
Prepended to messages
before API call"]:::merge + + SystemCtx --> PREPEND + UserCtx --> PREPEND + + classDef merge fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +Both are **memoized** and cached for the duration of the conversation. They're cleared on `/clear` and `/compact`. + +--- + +## Markdown Config Discovery + +Commands, agents, skills, and workflows are all loaded via `markdownConfigLoader.ts`: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#fd7e14', 'primaryBorderColor': '#fd7e14'}}}%% +flowchart TD + subgraph Dirs["Search Directories — Priority Order"] + MANAGED["Managed (enterprise)
Policy-level configs"]:::ent + USER_DIR["User (~/.claude/)
Personal configs"]:::user + PROJ_DIRS["Project (.claude/)
Walk cwd up to git root
Check each ancestor"]:::proj + end + + LOADER["loadMarkdownFilesForSubdir
Discover + parse YAML frontmatter
+ markdown content"]:::loader + + DEDUP["Deduplicate by inode
Handle symlinks gracefully"]:::dedup + + FILES["MarkdownFile array
filePath, frontmatter,
content, source"]:::result + + MANAGED --> LOADER + USER_DIR --> LOADER + PROJ_DIRS --> LOADER + LOADER --> DEDUP --> FILES + + classDef ent fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px + classDef user fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef proj fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef loader fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef dedup fill:#3d2b00,stroke:#fd7e14,color:#e0e0e0,stroke-width:2px + classDef result fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +This same loader powers: +- Slash commands (`.claude/commands/`) +- Agents (`.claude/agents/`) +- Skills (`.claude/skills/`) +- Workflows (`.claude/workflows/`) +- Output styles (`.claude/output-styles/`) + +Each entry is parsed with YAML frontmatter for metadata (description, tools, etc.) and the markdown body becomes the instructions. + +--- + +## Key Files + +- `src/utils/systemPrompt.ts` — `buildEffectiveSystemPrompt()` logic +- `src/constants/systemPromptSections.ts` — Section caching system +- `src/context.ts` — `getSystemContext()`, `getUserContext()`, git status +- `src/utils/markdownConfigLoader.ts` — CLAUDE.md and `.claude/` directory discovery +- `src/utils/claudemd.ts` — Memory file reading and concatenation +- `src/utils/settings/settings.ts` — Settings merge logic + +--- + +**Previous:** [<- UI Architecture](./09-ui-architecture.md) | **Next:** [MCP Deep Dive ->](./11-mcp-deep-dive.md) diff --git a/learn/11-mcp-deep-dive.md b/learn/11-mcp-deep-dive.md new file mode 100644 index 0000000..0b5cfd2 --- /dev/null +++ b/learn/11-mcp-deep-dive.md @@ -0,0 +1,290 @@ +# 11. MCP Deep Dive + +> The Model Context Protocol subsystem -- transports, tool wrapping, authentication, and server lifecycle. + +--- + +## What is MCP? + +MCP (Model Context Protocol) is an open protocol for connecting AI models to external tools and data sources. Claude Code is both an **MCP client** (connecting to external MCP servers) and can act as an **MCP server** (exposing its own tools to other systems). + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#4a9eff', 'primaryBorderColor': '#4a9eff'}}}%% +graph TB + subgraph ClaudeCode["Claude Code"] + CLIENT["MCP Client
Connects to external servers"]:::client + SERVER["MCP Server Mode
Exposes tools to other systems"]:::server + TOOLS["42 Built-in Tools
+ MCP tools merged in"]:::tools + end + + subgraph External["External MCP Servers"] + STDIO_S["stdio servers
Local processes"]:::external + SSE_S["SSE/HTTP servers
Remote endpoints"]:::external + WS_S["WebSocket servers
Persistent connections"]:::external + CLAUDE_AI["claude.ai proxy
Managed connectors"]:::external + IDE_S["IDE servers
Editor integration"]:::external + end + + CLIENT --> STDIO_S + CLIENT --> SSE_S + CLIENT --> WS_S + CLIENT --> CLAUDE_AI + CLIENT --> IDE_S + + STDIO_S --> TOOLS + SSE_S --> TOOLS + WS_S --> TOOLS + CLAUDE_AI --> TOOLS + IDE_S --> TOOLS + + classDef client fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef server fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef tools fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef external fill:#333,stroke:#888,color:#aaa,stroke-width:1px,stroke-dasharray: 5 5 +``` + +--- + +## Transport Types + +Claude Code supports **6 transport types** for connecting to MCP servers: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#17a2b8', 'primaryBorderColor': '#17a2b8'}}}%% +flowchart TD + subgraph Local["Local Transports"] + STDIO["stdio
Spawn local process
Communicate via stdin/stdout
Most common type"]:::local + SDK_T["sdk
In-process transport
SDK-managed placeholder
Tool calls route to SDK"]:::local + end + + subgraph Remote["Remote Transports"] + SSE["sse
Server-Sent Events
Legacy remote protocol
OAuth authentication"]:::remote + HTTP["http
Streamable HTTP
Modern remote protocol
Per-request timeouts"]:::remote + WS["ws
WebSocket
Persistent bidirectional
TLS/proxy support"]:::remote + CLAUDEAI["claudeai-proxy
Via claude.ai OAuth
Managed connectors
Auto token refresh"]:::remote + end + + subgraph IDE["IDE Transports"] + SSE_IDE["sse-ide
IDE SSE connection
No authentication"]:::ide + WS_IDE["ws-ide
IDE WebSocket
Auth token in header"]:::ide + end + + classDef local fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef remote fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef ide fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px +``` + +--- + +## Connection Lifecycle + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#28a745', 'actorTextColor': '#e0e0e0', 'actorBorder': '#28a745', 'signalColor': '#28a745', 'noteBkgColor': '#16213e', 'noteTextColor': '#e0e0e0', 'activationBkgColor': '#1b3a1b', 'activationBorderColor': '#28a745'}}}%% +sequenceDiagram + participant H as useManageMCPConnections + participant C as connectToServer (memoized) + participant T as Transport Layer + participant S as External Server + + H->>H: getAllMcpConfigs - merge all sources + H->>H: Batch servers (local=3, remote=20) + + loop For each server + H->>C: connectToServer(name, config) + activate C + + alt Auth cached as needed + C-->>H: return needs-auth + else Server disabled + C-->>H: return disabled + else Policy blocked + C-->>H: return not allowed + end + + C->>T: Create transport (stdio/sse/http/ws) + T->>S: Connect + activate S + + alt auth required (401/403) + S-->>C: Auth error + C-->>H: return needs-auth + else connection timeout + S-->>C: Timeout (30s default) + C-->>H: return error + else success + S-->>T: Connected + deactivate S + C->>S: tools/list + S-->>C: Available tools + C->>C: Wrap as MCPTool objects + C-->>H: return connected + tools + end + deactivate C + end + + H->>H: Update AppState.mcp +``` + +### Batched Connections + +Servers are connected in parallel batches to avoid overwhelming the system: +- **Local (stdio) servers**: batch size of 3 (spawning processes is heavy) +- **Remote servers**: batch size of 20 (network connections are lighter) + +The batch sizes are configurable via `MCP_SERVER_CONNECTION_BATCH_SIZE` and `MCP_REMOTE_SERVER_CONNECTION_BATCH_SIZE` environment variables. + +--- + +## Tool Wrapping + +Each MCP tool is wrapped in a `MCPTool` object that implements the standard `Tool` interface: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#ffc107', 'primaryBorderColor': '#ffc107'}}}%% +flowchart LR + MCP_TOOL["MCP Server Tool
name, inputSchema,
description"]:::raw + + WRAP["MCPTool wrapper
Implements Tool interface
name: mcp__server__tool"]:::wrap + + subgraph Lifecycle["Tool Call Lifecycle"] + VALIDATE["Validate input
via JSON Schema"]:::step + PERM["Check permissions
Same system as built-in"]:::step + CALL["Forward to MCP server
tools/call RPC"]:::step + RESULT["Process result
Truncate if needed
Handle images"]:::step + end + + MCP_TOOL --> WRAP --> VALIDATE --> PERM --> CALL --> RESULT + + classDef raw fill:#333,stroke:#888,color:#e0e0e0,stroke-width:1px + classDef wrap fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef step fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +### Naming Convention + +MCP tools are named `mcp____`: +- `mcp__slack__send_message` +- `mcp__github__create_issue` +- `mcp__ide__getDiagnostics` + +The double-underscore separators prevent ambiguity. In SDK no-prefix mode, tools keep their original names. + +### Description Capping + +MCP tool descriptions are capped at **2,048 characters**. OpenAPI-generated servers have been observed dumping 15-60KB of endpoint docs into descriptions, which wastes context tokens. + +--- + +## Configuration Sources + +MCP server definitions come from multiple sources, merged with priority rules: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#dc3545', 'primaryBorderColor': '#dc3545'}}}%% +flowchart TD + subgraph Sources["MCP Config Sources"] + ENT["Enterprise managed-mcp.json
Exclusive control if present"]:::ent + GLOBAL["Global ~/.claude/settings.json
mcpServers field"]:::user + LOCAL[".claude/settings.local.json
Local project overrides"]:::local + PROJECT[".mcp.json at project root
Committed to repo"]:::proj + PLUGIN["Plugin MCP servers
From installed plugins"]:::plugin + CLAUDEAI["claude.ai connectors
Fetched at startup"]:::claudeai + CLI["--mcp-config flag
Runtime override"]:::cli + SDK_CFG["SDK mcp_set_servers
Programmatic control"]:::sdk + end + + MERGE["getAllMcpConfigs
Merge + dedup + policy filter"]:::merge + + POLICY{"Enterprise policy
allowedMcpServers?
deniedMcpServers?"}:::check + + ALLOWED["Allowed servers
proceed to connect"]:::allow + BLOCKED["Blocked servers
silently dropped"]:::deny + + Sources --> MERGE --> POLICY + POLICY -->|"allowed"| ALLOWED + POLICY -->|"denied"| BLOCKED + + classDef ent fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px + classDef user fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef local fill:#3d2b00,stroke:#fd7e14,color:#e0e0e0,stroke-width:2px + classDef proj fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef plugin fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef claudeai fill:#1a1a4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef cli fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef sdk fill:#333,stroke:#888,color:#e0e0e0,stroke-width:1px + classDef merge fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef check fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef allow fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef deny fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px +``` + +### Deduplication + +Plugin and claude.ai connector servers are deduplicated against manually-configured servers: +- **Signature-based**: Same `command+args` (stdio) or same `url` (remote) = same server +- **Manual wins**: If a user manually configured a server, the plugin/connector duplicate is suppressed +- **First-loaded wins**: Between plugins, the first one loaded takes priority + +--- + +## Authentication + +Remote MCP servers may require OAuth authentication: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#dc3545', 'primaryBorderColor': '#dc3545'}}}%% +flowchart TD + CONNECT["Connect to remote server"]:::start + + AUTH{"Auth required?"}:::check + + PROVIDER["ClaudeAuthProvider
OAuth flow per server"]:::auth + TOKENS["Retrieve/refresh tokens"]:::auth + + OK["Connection established"]:::success + NEEDS["Status: needs-auth
Cached for 15 min"]:::needsauth + + CALL["MCP tool call"]:::start + CALL_AUTH{"401 on call?"}:::check + + RETRY["Refresh token
retry once"]:::auth + MCE["McpAuthError
Update status to needs-auth"]:::error + + CONNECT --> AUTH + AUTH -->|"no"| OK + AUTH -->|"yes"| PROVIDER --> TOKENS + TOKENS -->|"success"| OK + TOKENS -->|"failure"| NEEDS + + CALL --> CALL_AUTH + CALL_AUTH -->|"no"| OK + CALL_AUTH -->|"yes"| RETRY + RETRY -->|"success"| OK + RETRY -->|"failure"| MCE + + classDef start fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef check fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef auth fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef success fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef needsauth fill:#3d2b00,stroke:#fd7e14,color:#e0e0e0,stroke-width:2px + classDef error fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:2px +``` + +### Session Expiry Handling + +MCP servers return HTTP 404 + JSON-RPC error code `-32001` when a session expires. Claude Code detects this, clears the connection cache, and reconnects automatically (`McpSessionExpiredError`). + +--- + +## Key Files + +- `src/services/mcp/client.ts` (3,349 lines) -- Connection manager, tool wrapping, transport creation +- `src/services/mcp/config.ts` (1,579 lines) -- Config merging, policy filtering, deduplication +- `src/services/mcp/types.ts` -- Type definitions for server configs and connections +- `src/services/mcp/auth.ts` -- OAuth provider and step-up authentication +- `src/services/mcp/useManageMCPConnections.ts` -- React hook managing connection lifecycle +- `src/tools/MCPTool/MCPTool.ts` -- MCPTool wrapper implementing the Tool interface + +--- + +**Previous:** [<- Configuration](./10-configuration-and-system-prompt.md) | **Next:** [Data Flow Walkthrough ->](./12-data-flow-walkthrough.md) diff --git a/learn/12-data-flow-walkthrough.md b/learn/12-data-flow-walkthrough.md new file mode 100644 index 0000000..c688d47 --- /dev/null +++ b/learn/12-data-flow-walkthrough.md @@ -0,0 +1,341 @@ +# 12. Data Flow Walkthrough + +> Tracing a single user request from keystroke to rendered response -- end to end. + +--- + +## The Complete Flow + +This diagram traces exactly what happens when a user types a prompt and presses Enter. + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#4a9eff', 'primaryBorderColor': '#4a9eff'}}}%% +flowchart TD + TYPE["User types in terminal"]:::input + PASTE["Paste: text, image, file"]:::input + VOICE["Voice input via STT"]:::input + IDE_IN["IDE sends prompt"]:::input + SDK_IN["SDK submitMessage"]:::input + + PARSE["Parse input
slash commands, @mentions"]:::process + SLASH{"Slash
command?"}:::decision + LOCAL["Execute locally
clear, compact, model"]:::process + ATTACH["Build attachments
images, PDFs, CLAUDE.md"]:::process + UMSG["Create UserMessage"]:::process + + SYS["Build system prompt
tool descriptions +
user rules + CLAUDE.md"]:::context + GIT["Inject git context
branch, commits, status"]:::context + USR["Inject user context
CLAUDE.md, date"]:::context + CMP["Run compaction pipeline"]:::context + CACHE["Apply cache_control
prompt cache breakpoints"]:::context + + BUILD["Build API request
model, betas, effort,
task_budget, thinking"]:::api + STREAM["Stream SSE response
from Anthropic API"]:::api + RETRY["withRetry wrapper
429 backoff, 529 fallback"]:::api + + THINK["Thinking blocks"]:::response + TEXT["Text blocks"]:::response + TOOL_USE["tool_use blocks"]:::response + + VALIDATE["Validate input"]:::toolexec + PERMS["Check permissions"]:::toolexec + EXEC["Execute tool"]:::toolexec + RESULT["Map to tool_result"]:::toolexec + FEEDBACK["Push to messages
LOOP BACK"]:::feedback + + RENDER["Render in terminal"]:::output + TRANSCRIPT["Persist transcript"]:::output + SDK_OUT["Yield SDK events"]:::output + COST["Track cost + usage"]:::output + + TYPE --> PARSE + PASTE --> PARSE + VOICE --> PARSE + IDE_IN --> PARSE + SDK_IN --> PARSE + + PARSE --> SLASH + SLASH -->|"yes"| LOCAL --> RENDER + SLASH -->|"no"| ATTACH --> UMSG + + UMSG --> SYS --> GIT --> USR --> CMP --> CACHE + + CACHE --> BUILD --> STREAM + STREAM --> RETRY --> STREAM + + STREAM --> THINK --> RENDER + STREAM --> TEXT --> RENDER + STREAM --> TOOL_USE + + TOOL_USE --> VALIDATE --> PERMS --> EXEC --> RESULT --> FEEDBACK + FEEDBACK ==>|"loop back to
compaction"| CMP + + TEXT -->|"end_turn"| TRANSCRIPT --> SDK_OUT --> COST + + classDef input fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef process fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef decision fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef context fill:#2d1b4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef api fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef response fill:#3d2b00,stroke:#fd7e14,color:#e0e0e0,stroke-width:2px + classDef toolexec fill:#1a1a4e,stroke:#6f42c1,color:#e0e0e0,stroke-width:2px + classDef feedback fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:3px + classDef output fill:#333,stroke:#888,color:#e0e0e0,stroke-width:1px +``` + +--- + +## Phase by Phase + +### Phase 1: Input Capture + +The user's input can arrive from **5 different sources**, all converging into `processUserInput()`: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#17a2b8', 'primaryBorderColor': '#17a2b8'}}}%% +flowchart LR + subgraph Sources["Input Sources"] + KB["Keyboard input
Direct typing in terminal"] + PASTE["Clipboard paste
Text, images, files"] + VOICE["Voice
STT transcription"] + IDE["IDE Bridge
Prompt from editor"] + SDK["SDK
Programmatic submitMessage"] + end + + PUI["processUserInput
Parse slash commands
Resolve @mentions
Build attachments"]:::process + + subgraph Output["Parse Result"] + MSGS["messages array"] + SHOULD["shouldQuery boolean"] + end + + Sources --> PUI --> Output + + classDef process fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px +``` + +Slash commands (`/clear`, `/compact`, `/model`) are intercepted here and handled locally without an API call. Everything else proceeds to the model. + +### Phase 2: Context Assembly + +Before sending to the API, the system assembles context from multiple sources: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#6f42c1', 'primaryBorderColor': '#6f42c1'}}}%% +flowchart TD + subgraph SystemPrompt["System Prompt Assembly"] + DEFAULT["Default prompt
Claude Code identity"] + TOOL_DESC["Tool descriptions
42+ tool schemas"] + SECTIONS["System prompt sections
Cached or volatile"] + end + + subgraph UserContext["User Context"] + CLAUDE_MD["CLAUDE.md files
Project instructions"] + DATE["Current date"] + end + + subgraph SystemContext["System Context"] + GIT_STATUS["Git snapshot
Branch, status, log"] + end + + subgraph Compaction["Compaction Pipeline"] + SNIP["Snip compact"] + MICRO["Micro compact"] + AUTO["Auto compact"] + COLLAPSE["Context collapse"] + end + + SystemPrompt --> FINAL["Final API request"]:::api + UserContext --> FINAL + SystemContext --> FINAL + Compaction --> FINAL + + classDef api fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +### Phase 3: API Call and Streaming + +The request goes through `claude.ts` with retry logic: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#28a745', 'primaryBorderColor': '#28a745'}}}%% +flowchart LR + REQ["API Request
messages + system + tools"]:::req + + STREAM["SSE Stream"]:::stream + + subgraph Events["Stream Events in Order"] + E1["message_start
model, id, usage"] + E2["content_block_start
type + index"] + E3["content_block_delta
incremental text/tool JSON"] + E4["content_block_stop"] + E5["message_delta
stop_reason + final usage"] + E6["message_stop"] + end + + REQ --> STREAM --> Events + + classDef req fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef stream fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px +``` + +Each content block is one of three types: **thinking** (internal reasoning), **text** (user-facing response), or **tool_use** (triggers tool execution). + +### Phase 4: Tool Execution Loop + +When the model responds with `tool_use`, the agentic loop kicks in: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#e83e8c', 'primaryBorderColor': '#e83e8c'}}}%% +flowchart TD + TOOL_USE["Model returns tool_use blocks"]:::start + + PAR{"Multiple tools?
isConcurrencySafe?"}:::check + + PARALLEL["Execute in parallel
Read-only tools"]:::exec + SEQUENTIAL["Execute sequentially
Write tools"]:::exec + + VALIDATE["Zod schema validation"]:::step + HOOKS_PRE["PreToolUse hooks"]:::step + PERM_CHECK["Permission check chain
deny - allow - tool -
hooks - classifier - dialog"]:::step + EXECUTE["tool.call(input, context)"]:::step + HOOKS_POST["PostToolUse hooks"]:::step + MAP_RESULT["Map to tool_result message"]:::step + + INJECT["Inject CLAUDE.md attachments
for newly-discovered dirs"]:::step + + PUSH["Push tool_results to messages
Continue loop"]:::feedback + + TOOL_USE --> PAR + PAR -->|"yes"| PARALLEL + PAR -->|"no"| SEQUENTIAL + + PARALLEL --> VALIDATE + SEQUENTIAL --> VALIDATE + + VALIDATE --> HOOKS_PRE --> PERM_CHECK --> EXECUTE --> HOOKS_POST --> MAP_RESULT --> INJECT --> PUSH + + PUSH ==>|"Back to
compaction"| TOOL_USE + + classDef start fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px + classDef check fill:#2d2d0d,stroke:#ffc107,color:#e0e0e0,stroke-width:2px + classDef exec fill:#0d4f4f,stroke:#17a2b8,color:#e0e0e0,stroke-width:2px + classDef step fill:#1b3a1b,stroke:#28a745,color:#e0e0e0,stroke-width:2px + classDef feedback fill:#4a1a1a,stroke:#dc3545,color:#e0e0e0,stroke-width:3px +``` + +The loop continues until the model returns `stop_reason: end_turn`, hits `max_turns`, or is cancelled. + +### Phase 5: Response Rendering + +The final response is rendered and persisted: + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#fd7e14', 'primaryBorderColor': '#fd7e14'}}}%% +flowchart LR + RESPONSE["Model final response
stop_reason: end_turn"]:::input + + subgraph Rendering["Terminal Rendering"] + MD["Markdown rendering
via marked library"] + SYNTAX["Syntax highlighting
for code blocks"] + DIFF["Diff rendering
for file changes"] + end + + subgraph Persistence["Persistence"] + TRANSCRIPT["recordTranscript
Persist to disk"] + USAGE["Accumulate usage
input + output tokens"] + COST["Track cost
per-model pricing"] + end + + subgraph SDK_Events["SDK Events"] + MSG["SDKMessage stream
Normalized events"] + RESULT_E["Result event
total_cost, usage, duration"] + end + + RESPONSE --> Rendering + RESPONSE --> Persistence + RESPONSE --> SDK_Events + + classDef input fill:#1a2d4a,stroke:#4a9eff,color:#e0e0e0,stroke-width:2px +``` + +--- + +## A Concrete Example + +Let's trace what happens when a user types: `fix the bug in utils.ts` + +```mermaid +%%{init: {'theme': 'dark', 'themeVariables': { 'primaryColor': '#1a1a2e', 'primaryTextColor': '#e0e0e0', 'lineColor': '#4a9eff', 'actorTextColor': '#e0e0e0', 'actorBorder': '#4a9eff', 'signalColor': '#4a9eff', 'noteBkgColor': '#16213e', 'noteTextColor': '#e0e0e0', 'activationBkgColor': '#2d1b4e', 'activationBorderColor': '#e83e8c'}}}%% +sequenceDiagram + participant U as User + participant R as REPL.tsx + participant QE as QueryEngine + participant Q as query.ts + participant C as claude.ts + participant T as Tools + + U->>R: Types "fix the bug in utils.ts" + R->>QE: submitMessage(prompt) + + Note over QE: Not a slash command, proceed + + QE->>QE: Persist transcript + QE->>Q: query(messages, systemPrompt, tools) + + Note over Q: Turn 1 - Model reads code + + Q->>Q: Run compaction pipeline + Q->>C: queryModel(messages) + C-->>Q: tool_use: FileRead("utils.ts") + Q->>T: Execute FileRead + T-->>Q: File contents + + Note over Q: Turn 2 - Model analyzes + + Q->>C: queryModel(messages + file contents) + C-->>Q: tool_use: Grep("error pattern") + Q->>T: Execute Grep + T-->>Q: Search results + + Note over Q: Turn 3 - Model fixes + + Q->>C: queryModel(messages + grep results) + C-->>Q: tool_use: FileEdit("utils.ts", patch) + + Note over T: Permission check! + T->>U: Allow FileEdit? (Y/n/always) + U->>T: Y + + T-->>Q: Edit applied + + Note over Q: Turn 4 - Model confirms + + Q->>C: queryModel(messages + edit result) + C-->>Q: text: "Fixed the bug" + end_turn + + Q-->>QE: Terminal result + QE->>QE: Record transcript, track usage + QE-->>R: Render response + R->>U: Display formatted answer +``` + +This shows the typical pattern: **read first, analyze, then write** -- with permission checks only on the write operation. + +--- + +## Key Takeaways + +1. **Multiple entry points, single pipeline**: Whether input comes from keyboard, IDE, or SDK, it all converges into the same `processUserInput -> query -> claude.ts` pipeline. + +2. **Compaction runs every turn**: The 5-stage compaction pipeline runs *before every single API call*, not just when limits are hit. + +3. **Tools loop back**: Tool results are pushed to messages, and the entire pipeline (including compaction) runs again. This is the "agentic" part -- the model keeps going until it's done. + +4. **Permissions only interrupt writes**: Read-only tools (FileRead, Grep, Glob) in default mode flow through silently. Only write operations (FileEdit, Bash, FileWrite) trigger permission dialogs. + +5. **Everything is streamed**: From SSE events to async generators to React state updates, nothing blocks waiting for a full response. The user sees tokens as they arrive. + +--- + +**Previous:** [<- MCP Deep Dive](./11-mcp-deep-dive.md) diff --git a/learn/README.md b/learn/README.md index 4f081fe..f3276cc 100644 --- a/learn/README.md +++ b/learn/README.md @@ -39,6 +39,9 @@ Start from the top and work down, or jump to whatever interests you: | 7 | [Extension Model](./07-extension-model.md) | Skills, plugins, hooks, sub-agents, and swarms | | 8 | [API Client](./08-api-client.md) | `claude.ts` — streaming, retries, caching, and model fallback | | 9 | [UI Architecture](./09-ui-architecture.md) | React Ink, 113 components, and the 896KB REPL | +| 10 | [Configuration and System Prompt](./10-configuration-and-system-prompt.md) | Settings hierarchy, CLAUDE.md discovery, prompt assembly | +| 11 | [MCP Deep Dive](./11-mcp-deep-dive.md) | Transports, tool wrapping, auth, server lifecycle | +| 12 | [Data Flow Walkthrough](./12-data-flow-walkthrough.md) | End-to-end trace of a single user request | ---