AI Chat

AI Chat is a conversational AI companion that lives next to your terminals and sees what they see. Unlike spawning a full agent (Claude Code, Aider, …) in a PTY, the chat panel gives you a quick explain / summarise / suggest loop that shares the exact screen state of your active terminal.

Two progressive capability levels ride on the same panel:

The same panel switches modes — no separate UI.

Opening the panel

• Hotkey: Cmd+Alt+A (macOS) / Ctrl+Alt+A (others) — toggle.
• Toolbar: chat icon in the right section of the toolbar.
• Context menu: right-click a terminal → Send selection to AI Chat or Explain this error.

The panel docks on the right. Width is remembered per window (aiChatPanelWidth).

Providers

AI Chat speaks to four provider families plus a custom endpoint. Switch in Settings > AI Chat > Provider:

Model recommendations

API keys are stored in the OS keyring under service tuicommander-ai-chat — never written to disk in plaintext.

Local MLX models (Apple Silicon)

On Apple Silicon Macs, MLX models run natively via the MLX framework and are significantly faster than GGUF models for small inference tasks like enrichment and triage.

Option A: mlx_lm.server (recommended for enrichment)

Serves an MLX model with an OpenAI-compatible API. Configure as a Custom provider in TUIC.

# Install
pipx install mlx-lm

# Start server (port 8899, Qwen 2.5 Coder 3B 4-bit)
mlx_lm.server --model mlx-community/Qwen2.5-Coder-3B-Instruct-4bit --port 8899

In Settings > AI Chat > Providers, add a Custom provider with base URL http://127.0.0.1:8899/v1/ and model name mlx-community/Qwen2.5-Coder-3B-Instruct-4bit. Assign it to the Triage slot.

Benchmark (M4): ~120 tok/s generation vs ~98 tok/s for the same model via Ollama GGUF.

Option B: Ollama with MLX tags

Some models have native MLX tags on the Ollama registry (e.g. qwen3.5:4b-mlx-bf16). These run on Ollama’s built-in MLX runner automatically. Check available tags with ollama pull <model>:<size>-mlx-<dtype>.

Option C: MLX → GGUF conversion

For MLX models without Ollama MLX tags, convert to GGUF and import into Ollama:

# 1. Download MLX model
pipx run --spec huggingface_hub hf download \
  mlx-community/Qwen2.5-Coder-3B-Instruct-4bit \
  --local-dir /tmp/model-mlx

# 2. Dequantize MLX → HF safetensors
mlx_lm.convert --hf-path /tmp/model-mlx --mlx-path /tmp/model-hf -d

# 3. Convert HF → GGUF (requires llama.cpp + torch)
git clone --depth 1 https://github.com/ggerganov/llama.cpp /tmp/llama.cpp
cd /tmp/llama.cpp && cmake -B build -DGGML_METAL=ON && cmake --build build --target llama-quantize -j
python3 convert_hf_to_gguf.py /tmp/model-hf --outtype f16 --outfile /tmp/model-f16.gguf
./build/bin/llama-quantize /tmp/model-f16.gguf /tmp/model-q4.gguf Q4_K_M

# 4. Import into Ollama
cat > /tmp/Modelfile <<'EOF'
FROM /tmp/model-q4.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
EOF
ollama create my-model -f /tmp/Modelfile

Note: double quantization (MLX 4-bit → bf16 → GGUF Q4_K_M) introduces quality loss. Prefer Option A for MLX models or pull the native GGUF from Ollama when available (ollama pull qwen2.5-coder:3b).

Context injection

Every turn the backend assembles a compact context from the currently-attached terminal:

• Clean screen text — last context_lines rows from the VtLogBuffer (ANSI-stripped, TUI alternate-screen suppressed). Default: 150. Tune in settings.
• Session state — shell busy / idle, CWD, last exit code, detected agent type, terminal mode (Shell vs FullscreenTui).
• Recent parsed events — errors, questions, rate limits, status lines.
• Git context — branch, short diff stats, staged file list (same variables Smart Prompts use).
• Session knowledge (when the agent has been active) — compact markdown summary of recent command outcomes, error→fix pairs, TUI apps seen.

The panel follows the focused terminal automatically — the header shows the active terminal’s name as a badge. When no terminal is focused (e.g. a Git or settings tab is active), the panel enters frozen state: a banner reads “No terminal focused — chat is read-only”, the input placeholder changes to “Focus a terminal first…”, and the send button is disabled. Focus any terminal tab to resume.

Conversations

• Per-terminal state — each terminal tab maintains its own independent chat history, streaming state, and conversation ID (keyed by tuicSession). Switching tabs switches the conversation. Messages sent from a tab always target that tab’s PTY session.
• Hard cap: 100 messages per conversation in memory; older messages are evicted FIFO. Saved conversations keep the full history on disk.
• Streaming uses a Tauri Channel<ChatStreamEvent> — you see tokens as they arrive. Cancel mid-stream with the stop button or cancel_ai_chat.
• Conversation history panel — click the clock/history icon in the header to open a slide-in list of all saved conversations. Each row shows the title, terminal session name, message count, and date. Click a row to load that conversation into the current terminal’s chat.

Run-this, copy, and actions

Every fenced code block in the AI reply has a small toolbar:

Language hints in the fence control button visibility — a ```text block hides Run.

Agent mode (ReAct)

Flip the panel into agent mode via the header toggle or the command palette (Agent: start). Give it a goal (“set up pnpm and install deps”, “fix the failing test”) and press Enter. The loop:

1. Assemble context.
2. Ask the LLM with six tools available: read_screen, send_input, send_key, wait_for, get_state, get_context.
3. Dispatch tool calls — each appears as a collapsible card in the panel.
4. Record outcomes into the session knowledge store.
5. Stop on end_turn or when cancelled.

Safety gates

The SafetyChecker trait inspects every would-be send_input. Three verdicts:

• Allow — common commands, ls, git status, cargo build, editor launches …
• NeedsApproval — destructive patterns (rm -rf, git reset --hard, git push --force, DROP TABLE, dd of=, package uninstall …). The panel shows a Pending approval card with a one-line reason. Approve / reject with a click or approve_agent_action.
• Block — hard-coded refusals (e.g. rm -rf /, :(){ :|:& };:).

Pause / resume any time — the loop cleanly stops between iterations.

Session knowledge

As the agent runs, the SessionKnowledgeBar footer shows live telemetry:

• Commands run this session (count).
• Last 5 outcomes with kind badges (Success / Error / TuiLaunched / Timeout …).
• Last 5 errors with inferred error_type (e.g. rust-error-borrow, npm-missing-module).
• TUI mode indicator + list of TUI apps seen.

OSC 133 semantic prompts (OSC 133;A/B/C/D) feed accurate exit codes when the shell supports them (modern bash/zsh/fish with the integration enabled). Without OSC 133, the PTY silence timer records an Inferred outcome so the loop still learns.

Knowledge persists to <config_dir>/ai-sessions/<session_id>.json with a 2 s debounced background flush. Reopening a session rehydrates the store.

Knowledge history overlay

Click History next to the SessionKnowledgeBar to open a two-pane browser over every persisted session on disk — not just the currently active one. Useful for “find the command that fixed the build error last week”:

• Sessions list (left) — sorted by most recent activity, showing command count, error count, and last CWD.
• Detail pane (right) — one card per command with kind badge, timestamp, exit code, duration, CWD, output snippet, and a copy button.
• Filters — debounced full-text search (matches command, output, inferred error_type, and semantic_intent), errors only checkbox, date window (24h / 7d / 30d / all).

Esc closes. Backed by the list_knowledge_sessions + get_knowledge_session_detail Tauri commands.

External MCP surface (ai_terminal_* tools)

The same six ReAct tools are exposed to external MCP clients (Claude Code, Cursor, …) through the TUICommander MCP server:

Input tools are refused while the internal agent loop is active on that session, so an external agent can’t fight the internal one for the same PTY.

Settings reference (Settings > AI Chat)

Keyboard shortcuts

Files & storage

See also

• docs/backend/mcp-http.md — ai_terminal_* MCP tools + OAuth 2.1 upstream auth.
• docs/api/tauri-commands.md — Full Tauri command reference for chat + agent.
• docs/backend/pty.md — PTY lifecycle, OSC 133, TUI detection, silence-based idle.
• ideas/ai-assisted-terminal.md — Original 3-level plan (Level 1 = Chat, Level 2 = Agent, Level 3 = Knowledge).