How to make Claude Code work with open-weight models

An architecture tutorial. All code in this post is up on GitHub.

When we think of Claude Code, we think of Opus, Fable and even Sonnet, but in reality Claude Code is just a harness. Their agent loop, tool orchestration, permission system, and context management will drive essentially any model that speaks its protocol, and Anthropic actually documents the swap themselves: the CLI ships with an LLM gateway protocol, and pointing it at your own backend takes exactly one change in an environment variable, ANTHROPIC_BASE_URL.

That one variable opens up the harness to a whole world of open-weight models. In fact, whole inference platforms are being built on this insight at production scale. Agentic workloads on open-weight models at open-weight prices. This post is the local, single-file version of that idea. We'll build the proxy that translates between Claude Code's dialect and the OpenAI-compatible API that Nebius Token Factory, and virtually every other open-weight inference provider speaks. One Python file, about 800 lines including a small routing UI.

Nebius serves GLM-5.2 at $1.40 per million input tokens and $4.40 per million output tokens. Compare that to the models Claude Code normally runs on:

Roughly a third of Sonnet's output price, a sixth of Opus's. And the same swap works for anything in the Nebius catalog: Kimi K2.7-Code, DeepSeek V4 Pro, Qwen3.5, gpt-oss-120b.

But there is a small catch. Before you extrapolate a 6x saving, I'll have to remind you of prompt caching. Agent sessions resend the entire transcript on every turn, and Anthropic's cache serves those repeated tokens at roughly a tenth of the input price. Open-weight providers split on this. Some offer prompt caching with automatic discounts, DeepSeek and DeepInfra among them. Those savings survive the proxy because implicit caching needs no markers at all. Others bill every token at full price on every turn: Nebius is in that camp today, with an open feature request to change it. Either way the math usually still lands well below Anthropic's API pricing. But for long agent sessions, whether your provider caches matters as much as its "sticker" price, so check and do the math before you pick.

Let's get technical.

Set ANTHROPIC_BASE_URL and Claude Code sends everything to you instead of to Anthropic. What arrives is the Anthropic Messages API: POST /v1/messages with SSE streaming, plus an optional POST /v1/messages/count_tokens (it falls back to local estimation if you 404).

There are two relevant details:

• The SSE contract is specific. Anthropic streams typed events: message_start, then a content_block_start / content_block_delta / content_block_stop cycle per content block, then message_delta with the stop reason, then message_stop. OpenAI-style chat completions stream flat chunks. Translating between them is a big part of the work.
• Claude Code actually runs two models simultaneously. The main model drives the agent; a haiku-class model handles background chores. Your proxy sees both and should route them to a big and a small open model respectively.

Most of the request mapping is mechanical: system becomes a system message, max_tokens and temperature pass straight through, tool definitions move their JSON schema from input_schema to parameters. But there are two parts that deserve extra attention.

The first is structural. Anthropic packs tool results inside user messages as content blocks. Meanwhile, OpenAI wants each result as its own role: "tool" message. One Anthropic message can carry several tool results plus trailing text, so the translation fans one message out into many:

# One Anthropic user message can carry N tool_results plus trailing
# text -> N OpenAI "tool" messages, then one "user" message.
text_parts = []
for block in content or []:
    btype = block.get("type")
    if btype == "tool_result":
        result_text = flatten_text(block.get("content", ""))
        if block.get("is_error"):
            result_text = f"Error: {result_text}"
        messages.append({
            "role": "tool",
            "tool_call_id": block.get("tool_use_id", ""),
            "content": result_text,
        })
    elif btype == "text":
        text_parts.append(block.get("text", ""))
if text_parts:
    messages.append({"role": "user", "content": "\n\n".join(text_parts)})

The second is a policy decision: to drop unknown fields, never reject them. Claude Code grows new request parameters (output_config, thinking configs, beta features) faster than any translation layer keeps up. If your proxy 400s on a field it doesn't recognize, a routine CLI update takes your whole setup down. This isn't just theoretical, in fact, LiteLLM's Anthropic endpoint actually broke down for exactly this reason when Claude Code started sending output_config (litellm#22963). The proxy keeps an allowlist of fields it understands and logs-and-drops everything else.

The response direction is a small state machine. You read OpenAI chunks off the upstream stream and re-emit them as Anthropic events, tracking which content block is currently open. Text deltas open a text block, a tool call opens a tool_use block, only one block is open at a time, and each gets a clean start/stop pair.

OpenAI delivers streamed tool-call arguments as JSON fragments ({"pa, then th": "foo.py"}) spread across chunks. Anthropic has a dedicated delta type for exactly this, input_json_delta, and the client reassembles the fragments itself. So the correct move is to be lazy: just pass fragments through verbatim. Parsing or buffering them only adds bugs and unnecessary complexity.

for call in delta.get("tool_calls") or []:
    oai_index = call.get("index", 0)
    if oai_index not in tool_index_map:
        if (ev := close_block()):
            yield ev
        tool_index_map[oai_index] = next_index
        open_index, open_type = next_index, "tool"
        next_index += 1
        yield sse("content_block_start", {
            "type": "content_block_start", "index": open_index,
            "content_block": {"type": "tool_use",
                              "id": call.get("id") or "toolu_" + uuid.uuid4().hex[:12],
                              "name": (call.get("function") or {}).get("name", ""),
                              "input": {}},
        })
    fragment = (call.get("function") or {}).get("arguments")
    if fragment:
        yield sse("content_block_delta", {
            "type": "content_block_delta", "index": tool_index_map[oai_index],
            "delta": {"type": "input_json_delta", "partial_json": fragment},
        })

Close the open block when the stream ends, emit message_delta with the mapped stop reason (tool_calls → tool_use, length → max_tokens) and the token usage, then message_stop. That's the whole contract.

The proxy is just one Python file with inline dependency metadata, so uv is the only thing you install. It fetches an isolated Python and the dependencies on first run. From the project directory where you want to use Claude Code:

brew install uv
cp .env.example .env      # add your NEBIUS_API_KEY here directly or throught the UI after line 3
uv run proxy.py
claude --dangerously-skip-permissions

Two commands, no environment variables. uv run proxy.py starts the proxy in the background and writes ANTHROPIC_BASE_URL (plus a placeholder auth token and a generous timeout) into .claude/settings.local.json in your current directory, and Claude Code picks that up automatically. uv run proxy.py stop kills the proxy and removes the wiring again.

Then open http://localhost:8082. The UI is where everything else lives. Add a provider and paste its API key: Nebius is the suggested preset, but OpenRouter, Together, DeepInfra, Groq, or any custom OpenAI-compatible endpoint slot in the same way. Keys are stored server-side with file permissions locked down, and never echoed back to the browser. Below that sits the routing table, one row per Claude model class (Fable, Opus, Sonnet, Haiku, everything else), each a dropdown grouped by provider, so "Fable 5 to DeepSeek V4 Pro, Haiku to Qwen3-30B" is two clicks, applied on the next request with no restart. Each dropdown also has an "Anthropic (native)" option that forwards those requests untranslated to the real API with a key from console.anthropic.com, useful for keeping the background model on genuine Haiku while your main model runs open-weight.

And it just works. Ask it something and GLM-5.2 answers, cheerfully introducing itself as Claude, because the harness's system prompt tells it who to be. Ask it to read a file and it round-trips the Read tool in about six seconds. Ask it to write a FizzBuzz script and run it, and it streams the file through the Write tool, executes it with Bash, and summarizes the output, a full multi-turn agentic session in about fifteen seconds, indistinguishable from the shipped native Claude Code.

Day to day, the proxy disappears. It keeps running across sessions (start it once, run claude as many times as you like), and the wiring is per-directory, so only projects where you ran uv run proxy.py route through it. Everything else runs stock Claude Code on your normal account. Open-weight models for the big-token work, real Claude where it earns its high API prices.

One security aside we stumbled into while building this: if you point a logged-in Claude Code at a custom ANTHROPIC_BASE_URL with no other credential set, it sends your claude.ai subscription OAuth token to that URL. We watched it happen against a header-dumping fake gateway. Since a repo's .claude/settings.local.json can set ANTHROPIC_BASE_URL, a malicious repo could harvest your subscription token with two lines of JSON. That's exactly why this proxy wires a placeholder token alongside the base URL: the placeholder takes auth precedence, so your real credential is never sent anywhere. Be suspicious of gateway configs you didn't write.

• Anthropic-style prompt caching is gone. The proxy drops cache_control and always reports zero cached tokens, so Claude Code's cost display assumes full price. Whether your bill actually benefits from caching depends on the provider, as covered above.
• Reasoning tokens are invisible but billed. GLM-5.2 thinks before it answers, and that reasoning never reaches Claude Code: you see a pause, then output. In our very first test it spent an entire 200-token budget on hidden reasoning and got cut off before writing a single visible word. Budget max_tokens generously.
• Tool-argument JSON can be malformed. Open models occasionally emit broken JSON in tool calls. The proxy falls back to empty arguments and logs it but Claude Code retries. Rare with GLM-5.2 in our testing, but it happens.
• Images are dropped. Paste a screenshot and the model never sees it, you need multimodal models.
• Token counts are estimates. The proxy answers count_tokens with a chars-over-four guess, so context-window bookkeeping is approximate.

None of these are fatal for daily coding work. All of them are why this is a tutorial, not a product.

If you want a maintained translation layer rather than an understanding of one, LiteLLM's proxy exposes an Anthropic-compatible endpoint with multi-provider routing, retries, and spend tracking, and projects like claude-code-router and 1rgs/claude-code-proxy wrap the same idea specifically for Claude Code. They're the right choice for anything resembling production. The single file here is the right choice for knowing exactly what happens to every byte between your CLI and your model, and for having something to hack on when the next protocol field ships.

The full proxy, test suite, and setup instructions are here: github.com/THE-INFERENCE-HUB/claude-code-open-weights.

The Inference Hub is a global community for AI practitioners. If you're running agents on open-weight models, or deciding whether to, that's exactly the kind of conversation we're here for.