Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Local Inference (llama.cpp / MLX)

caliban runs local models through its OpenAI-compatible provider pointed at a local inference server — not through a dedicated integration. Any server that speaks the OpenAI /v1/chat/completions API works: llama.cpp's llama serve, mlx_lm.server, LM Studio, llama-swap, vLLM, and others.

```admonish info title="The built-in ollama provider was removed" The bespoke ollama provider was removed (ADR 0056). Reach local models through the OpenAI provider + a base_url instead, as described below. This is the same seam that reaches OpenAI itself — no dedicated provider, so any OpenAI-compatible local engine is one config line away. --provider ollama is no longer valid; use --provider openai with a base_url.


## Why this instead of a dedicated provider

The decision, its benchmark and conformance evidence (gathered on Apple M5 Pro),
and the phased removal plan are recorded in **ADR 0056**. In short: the local
engine landscape is uniformly OpenAI-compatible now; **llama.cpp** is the best
all-round default (portable single binary, widest model coverage, strong
prompt-cache warm path), **MLX** (`mlx_lm.server`) is an optional per-host tune
on Apple Silicon, and all three engines drive caliban's agent loop correctly.

## Wiring caliban to a local server

Point the OpenAI provider at the server's `/v1` endpoint. When `OPENAI_BASE_URL`
is set, **no API key is required** — caliban sends an empty bearer and the local
server ignores it (a keyed proxy would reject the request itself). Set
`OPENAI_API_KEY` only if your endpoint actually enforces auth.

**Env override (quickest):**

```bash
export OPENAI_BASE_URL="http://192.168.1.240:9292/v1"
caliban --provider openai --model <model-name> "…"

Router route (per-model / per-host), in .caliban/settings.toml:

[router]
default_purpose = "main_loop"

[router.provider.openai]
base_url = "http://192.168.1.240:9292/v1"

[[router.route]]
purpose = "main_loop"
provider = "openai"
model = "<model-name>"

A standalone caliban.toml is no longer discovered automatically — it nests provider blocks one level higher ([provider.openai]) and must be passed with --config. See The Model Router.

The model name must be one the backend accepts

caliban sends the route's model field to the server verbatim. Engines differ in how they treat it:

  • llama.cpp ignores the field and serves whatever is loaded — any name works.
  • mlx-lm treats it as a model to load: an unknown name is fetched from Hugging Face and 404s. Use the exact HF repo id (e.g. mlx-community/Qwen3.6-27B-4bit).
  • llama-swap routes on the name, then passes it through to the backend — so a friendly alias is fine for a llama.cpp backend, but a llama-swap entry fronting mlx-lm must be keyed by the repo id.

Alias proxies don't trigger a model-mismatch warning

With an alias proxy (llama-swap, LiteLLM, …) the backend often reports its own model name in the response (e.g. llama.cpp's -hf id) rather than the alias you requested. caliban does not warn about this: when the requested id is present in the server's /v1/models listing, a differing stream name is treated as an alias/backend label, not a silent substitution (#715). The mismatch warning still fires for the case it exists for — a server that serves a different loaded model for an id it never listed (the LM Studio substitution case).

A local endpoint changes the agent-loop profile

Pointing the OpenAI provider at a loopback, private-range, or bareword-LAN host also classifies the run as local for the adaptive agent-loop profile — so the default profile becomes local-guarded, which turns verification guidance on at verify-when-cheap instead of leaving it off. That is deliberate: a weaker local model benefits from being nudged to check its own work. Set [agent_loop] profile or [agent_loop] verification_guidance explicitly to opt out. Note the detection only recognizes a local setup behind the OpenAI-compatible adapter via OPENAI_BASE_URL; a local model reached any other way should name a profile explicitly.

Engine setup

Standing up llama.cpp (llama serve), mlx_lm.server, and llama-swap (a proxy that hot-swaps engines behind one /v1 port) on a headless Apple-Silicon box — including the TLS-inspection cert fix and health-check settings — is an operator task. Two helper scripts in the repo support the migration:

  • scripts/bench-local-inference.sh — cold/warm prefill, TTFT, and decode throughput across engines.
  • scripts/conformance-local-inference.sh — a capability matrix (tool calling, stop reasons, reasoning field) asserting a backend behaves as caliban needs.

Reasoning models

Reasoning-family models (Qwen3.x, DeepSeek-R1, …) stream a thinking trace. llama.cpp emits it under reasoning_content; MLX servers use reasoning. caliban accepts both, so thinking is captured as a Thinking block regardless of which engine serves it.