Speculative Decoding in Practice: Speeding Up Local Inference

The most visible bottleneck of local inference is "generation is slow." Quantization fits a model into VRAM; speculative decoding tries to speed up generation without changing the model. DeepSeek open-sourced three companion series on HuggingFace — DSpark, DFlash, and Eagle3 — covering its own V4 models as well as third-party bases like Qwen3 and Gemma4. This guide covers the mechanism, the ready-made companions, how to configure them, and when the technique is worth using.

The mechanism: why it is faster without changing results

Autoregressive generation is serial: producing token N requires token N-1 to be finished, so the GPU computes one token at a time and much of its compute sits idle. Speculative decoding's idea:

1. A much smaller "draft model" cheaply guesses several following tokens in a row
2. Those candidates are handed to the large model for parallel verification (one forward pass checks multiple positions, raising compute utilization)
3. The prefix the large model agrees with is accepted; at the first disagreement, drafting restarts from there

The key is verification: every accepted token is one the large model confirmed itself, so the final output is identical to unaccelerated generation. It is not "letting the model be sloppier" but "less waiting for the same computation." That is why papers and tool docs repeatedly stress an unchanged output distribution — the fundamental difference between speculative decoding and "switching to a smaller model."

What DeepSeek open-sourced

Based on the deepseek-ai HuggingFace organization releases of 2026-06-27/28. Download counts below are reference values at collection time — follow each model card for live data.

The three series correspond to different draft-generation strategies (the block7 and ttt7 suffixes are architecture/training configuration markers — see each model card for exact meaning). The naming is practical: dspark_qwen3_8b_block7 is "the DSpark draft model for Qwen3-8B." When choosing a companion, match the base name and scale first — do not pair a 14B draft with a 4B target.

Configuration approach

Field names differ across inference frameworks, but the approach is the same three steps:

1. Load the target model and point at a draft model. llama.cpp-family tools use a draft-model path flag (such as -md / --model-draft — check your version's --help); vLLM uses a speculative config specifying the draft model and method.

# llama.cpp shape illustration: field names follow your version --help
./build/bin/llama-server \
  -m ./models/Qwen3.8-27B-Q4_K_M.gguf \
  -md ./models/draft-model.gguf \
  -ngl 40 -c 8192 \
  --host 127.0.0.1 --port 8080

2. Get it running, then tune. Draft length (how many tokens guessed at once) and parallel verification batch size both affect gains — run defaults first, then compare tokens/s with the same prompt set with acceleration on and off.

3. Benchmark with one fixed prompt set. Record baseline tokens/s and VRAM, then the accelerated values. Gains vary by model, hardware, and task distribution — this article states no universal multiplier; your own measurement is authoritative.

When it is worth using

A practical heuristic: the longer and more predictable the output, the more speculative decoding pays off. Short Q&A has only a few serial steps, so the speedup ceiling is low to begin with.

Practical tuning checklist

Before enabling speculative decoding in a real deployment, walk this checklist:

1. Confirm engine support: not every llama.cpp or vLLM release supports every draft-method variant (Eagle3, DFlash, DSpark). Check your engine version release notes and the companion model card first.
2. Verify the base match: the draft model card should name the exact target model family and scale. A mismatch is the most common cause of low acceptance.
3. Measure baseline: record tokens/s and peak VRAM with acceleration off, on your real prompts.
4. Enable with defaults: do not tune draft length on the first run. Get a clean apples-to-apples measurement.
5. Compare on the same prompt set: change nothing else — same temperature, same context length, same quantization.
6. Inspect acceptance indirectly: if tokens/s barely moves, the draft is being rejected early. Try a companion closer to your target base.
7. Watch VRAM: the draft model plus its own small KV cache must fit alongside the target.
8. Decide per workload class: keep acceleration on for batch and long-output paths, off for latency-insensitive short requests if overhead shows up.

This is deliberately empirical. Speculative decoding is a workload-dependent optimization, not a universal switch.

Relation to other acceleration methods

Local speedup methods act on different layers and can be combined:

Note the essential difference between "switching to a smaller model" and speculative decoding: the former trades quality for speed, the latter does not. If your task is quality-sensitive, speculative decoding is the cleaner choice.

Measuring whether it actually helped

Speculative decoding is easy to enable and easy to misjudge. Before declaring a win, measure the right quantity on your own workload rather than trusting a published multiplier:

• Tokens per second end to end, including draft generation, verification, and any rejected-token recomputation — not just the verifier's forward pass
• Acceptance rate, the share of drafted tokens the large model keeps; a low rate means the draft model is a poor match for your prompts and the extra work is wasted
• Memory headroom, since the draft model and its KV cache occupy extra space that may force a lower context length or a smaller quantization tier
• Output equivalence, verified by replaying a fixed prompt set with and without speculation; the method should not change the distribution

Run the comparison at three prompt shapes — short chat, long-context reading, and structured generation — because acceptance rate varies sharply between them. Keep a small table of results per model pair and per configuration; when you change the draft model or the draft length, rerun that table rather than assuming the earlier result still holds. If the measured gain disappears under your real traffic, turning the feature off is a legitimate outcome: the mechanism is an optimization, not a requirement.

FAQ

Related reading

• VRAM Calculation in Practice — budgeting VRAM
• The Complete Local Deployment Playbook — seven-step flow
• DeepSeek V4 Family Overview — V4 family and DSpark background
• Qwen Cost Optimization — saving on the cloud side
• Qwen GGUF Local Deployment Primer — quantization and loading basics