Run Qwen AgentWorld-35B-A3B Locally with GGUF: A 35B-A3B MoE for Agentic Coding

Qwen-AgentWorld-35B-A3B is a model in the Qwen family worth calling out on its own: it is a 35B-A3B MoE (mixture-of-experts) architecture, and its naming points at agent and tool-use scenarios. For developers who want a private agent that can run tools and write code locally, without sending code and data to the cloud, this kind of model is attractive. This guide explains what it is, why it suits local runs, and how to run it locally with GGUF.

What 35B-A3B means

The numbers in the name are fixed in meaning: 35B is the total parameter count, and A3B is the parameters actually activated per token. That is the signature of a MoE (mixture-of-experts) architecture, where the model is made of several expert sub-networks and each token only activates a small subset of them.

One caveat: the exact capability description on the model card (which tool-call formats it supports, context length, whether it natively supports a particular agent protocol) changes with releases, and GGUF builds from different quantizers may differ slightly. This article covers architecture basics and deployment only; trust the model card you download for specifics.

Why it suits agentic coding

The "AgentWorld" naming points at agents and tool use. In a local agent, this kind of model earns its place by:

• Code and data stay on-machine: when you wire a private code base or internal docs into an agent for retrieval or editing, the model infers locally and sensitive content never leaves your hardware.
• Free to experiment: local calls cost nothing, so an agent's multi-round tool calls, retries, and replays incur no API fees.
• Controlled latency: running on your own GPU, latency depends only on your machine, not on cloud queueing.

Choosing a quantization tier

Like other Qwen models, AgentWorld-35B-A3B ships multiple GGUF quantizations. Because MoE activation is small, its usable size at mid-to-low tiers is usually friendlier than a dense model of the same total size.

Start at Q4_K_M by default. Exact file sizes are on the model card README; MoE tiers differ in size just as much, so check VRAM or RAM before downloading.

Ollama route

# Download a quantizer's Q4_K_M file (file name per the model card)
huggingface-cli download <quantizer>/Qwen-AgentWorld-35B-A3B-GGUF \
  Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf --local-dir ./models

# Modelfile: FROM ./models/Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf

ollama create agentworld -f Modelfile
ollama run agentworld "Write a Python quicksort implementation"

llama.cpp route

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j

./build/bin/llama-cli \
  -m ./models/Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf \
  -ngl 40 \
  -p "Write a Python quicksort implementation"

-ngl 40 offloads 40 layers to the GPU. MoE layer and activation structure is special, so exact per-layer VRAM depends on the implementation and context length; start with a small context to confirm it runs, then scale up.

Plugging it into a local agent

When wiring the local model into an agent framework (LangChain, LlamaIndex, or a self-built loop), a common pattern is to treat it as an OpenAI-style local inference endpoint:

from llama_cpp import Llama

# Local GGUF as the chat model
llm = Llama(
    model_path="./models/Qwen-AgentWorld-35B-A3B-Q4_K_M.gguf",
    n_gpu_layers=40,
    n_ctx=16384,   # agents often need a long context for tool specs and history
)

# The exact tool-call format follows the prompt template on the model card
messages = [
    {"role": "system", "content": "You are a coding assistant that can use tools."},
    {"role": "user", "content": "Read all .py files under ./src and find unused imports"},
]
out = llm.create_chat_completion(messages=messages)
print(out["choices"][0]["message"]["content"])

Local vs cloud

A practical pattern: use local AgentWorld during development to validate the agent's tool-call loop and prompts, then switch to a cloud endpoint once it needs stable external service. Tool definitions and prompts are largely reusable.

MoE in practice: throughput and batching

Because only a fraction of the parameters activate per token, MoE models behave differently under load than dense ones:

• Single-stream latency is often better than the total parameter count suggests, since each token touches fewer weights.
• Batched inference behaves differently: with more requests in flight, more experts become active across the batch, so memory pressure and compute can grow faster than a naive per-token estimate implies.
• Expert routing is input-dependent, so the same prompt can activate different expert subsets after small edits. That is normal for MoE and not a sign of instability.
• Quantization affects experts unevenly. Lower-bit tiers can degrade less-used experts more visibly, which is why a tier that feels fine for general chat can still lose accuracy on specialized tool-call formats.

For a local agent, the practical consequence is that you should load-test with your real concurrency, not extrapolate from one request. A config that looks comfortable at a single stream can still exhaust VRAM once several tool-call rounds run in parallel.

Common troubleshooting

Load failure / OOM. Lower the quantization tier (Q4_K_M to Q3_K_M) or reduce -ngl and try pure CPU to confirm it loads.

Wrong tool-call format. The tool-call convention follows the model card's prompt template. Construct a few samples per the official template and replay to confirm the format before wiring the framework.

Slow at long context. Agent context length drives VRAM or RAM up. Set n_ctx to what you actually need; do not blindly max it out.

FAQ

Related reading

• Qwen GGUF Local Deployment Primer — GGUF, quantization tiers, Ollama/llama.cpp basics
• Run Qwen 3.8 27B Locally with GGUF — local practice on a larger-parameter model
• Which Qwen Model to Use — selecting among the three cloud Qwen models
• Model Catalog — verify currently available Qwen model IDs
• Pricing — live pricing and plans