High Draft Acceptance Does Not Guarantee a Throughput Gain: A Lean CPU Speculative-Decoding Test

The draft model was working. It proposed tokens, the target accepted many of them, and the acceptance counter looked healthy.

The request still got slower.

That is the trap in speculative-decoding evaluation: acceptance is easy to celebrate because it proves that some target work was skipped. But the skipped work is only one side of the ledger. The drafter also consumes compute and memory bandwidth, and the target still has to verify its proposals. The operational question is not “Were tokens accepted?” It is “Did the complete request finish faster?”

Draft acceptance is a mechanism metric. Throughput and wall time are outcome metrics.

The Bounded Result

I compared speculative drafting disabled with draft depths n=1, n=2, and n=3 at two decode-heavy output lengths. Every cell contains three serial runs with the same fixed prompt, generated-token count, sampling controls, target/drafter pair, runtime, and CPU host class.

The runs used a fixed variant order. No dedicated warm-up, interleaving, or randomization was recorded, so cache, thermal, background-load, and order drift remain possible confounders. That makes this a bounded local enablement test rather than a production-grade benchmark.

Every tested draft depth lost on both outcome metrics. Mean runtime-reported generation throughput fell by 5.7% to 18.3%, while mean request wall time increased by 5.7% to 18.8%. Yet pooled draft acceptance ranged from 48.6% to 80.4%.

The bounded decision was straightforward: keep drafting disabled on this runtime. The result does not say speculative decoding is ineffective. It says this target/drafter/runtime combination did not earn enablement under this workload.

Define the Metrics Before Comparing Them

Draft acceptance here is the pooled number of accepted draft tokens divided by the pooled number of proposed draft tokens across all three runs in a cell. It is not the arithmetic mean of three per-run percentages.

Generation tokens/s is the runtime's backend-specific generation-rate field; it was not independently instrumented. Wall time is measured separately around the request, so the conclusion does not depend on treating the runtime metric as an end-to-end timer.

The generated-token count was fixed within each output-length comparison. The same requests underpin both generation rate and wall time, so those metrics are two differently scoped views of each run, not independent replications.

Keeping those surfaces separate prevents two common mistakes:

• using a high acceptance ratio as a proxy for speedup; and
• using an internal decode metric as though it captured every request cost.

The Acceptance Arithmetic Is Publicly Recomputable

The exact percentages matter less than the contradiction they expose: the highest observed pooled acceptance, 80.4%, still came with 7.0% lower reported generation throughput and 7.3% higher wall time.

The Means Are Not Hiding One Bad Run

Here are all three per-run timing values in every cell, shown to six decimal places. The means and deltas above were calculated from the full-precision source measurements rather than from these displayed values.

Every drafting run was slower than every corresponding drafting-off run in this sample on both reported generation rate and wall time. With only three runs per cell, this is still not a population estimate or a tail-latency study. It does show that the arithmetic means were not flipped by one isolated outlier.

Why Acceptance Can Lose

Speculative decoding trades target-model work for a different bundle of work:

• run the draft model;
• materialize proposed tokens;
• verify them with the target;
• discard or repair rejected proposals; and
• coordinate the extra state and memory traffic.

Accepted tokens are the benefit side. Drafting and verification are the cost side. A high acceptance rate can coexist with a net loss when the drafter is too expensive relative to the target work it saves, especially on a CPU path with a comparatively heavy draft artifact.

This test did not separately instrument compute, verification, cache, or memory-bandwidth attribution. The result is consistent with draft overhead exceeding saved target work; it does not identify one microarchitectural bottleneck as the proven cause.

A Small Enablement Scorecard

For a lean first pass, I would record at least these five fields per cell:

When Not to Use This Test as a Final Answer

• Interactive traffic: test time-to-first-token and tail latency, not only forced long outputs.
• Concurrent serving: test queueing, batch shape, throughput under load, and interference.
• Accelerators: remeasure on the actual GPU or other backend; CPU cost balance does not transfer automatically.
• Quality-sensitive workloads: add output-quality and task-success checks where the decoding configuration can change behavior.
• Capacity decisions: include memory use, energy, concurrency, and cost per completed request.

Also do not compare a new drafter against an unmatched baseline with different prompts, token caps, sampling, cache state, runtime build, or host pressure. The cleaner the claimed delta, the stricter the control surface should be.

Limitations and Falsification

This evidence covers one target/drafter pair on one CPU runtime and host class, with a fixed prompt, forced-length generation, two output lengths, and N=3. The fixed serial variant order and absence of a recorded warm-up, interleaving, or randomization leave order and system-drift confounders unresolved. It does not establish a general result for other models, quantizations, backends, accelerators, prompts, concurrency levels, or workloads.

This is a sanitized reproduction: exact runtime/build identity, model artifacts, host label and specification, private paths, and hashes are deliberately withheld. The published arithmetic is independently checkable, but reproducing the environment requires an internal same-class run rather than a reader-runnable public package.

The transcription claim fails if recomputing the published per-run values at their stated precision does not reproduce the displayed means, deltas, and pooled acceptance percentages. The bounded operational decision would be weakened by a same-protocol replication on the same target/drafter/runtime class in which one draft depth reproducibly improves both runtime-reported generation throughput and wall time by more than 5%.

A smaller or cheaper draft artifact, a backend optimization, or materially different hardware would be a new hypothesis, not a contradiction by itself.

Conclusion

The draft path was active. It accepted many proposals. It still made every tested request slower.

That is why speculative-decoding qualification needs both mechanism and outcome metrics. Publish the accepted/proposed counts. Publish the per-run timing values. Define pooled arithmetic. Keep an end-to-end timer beside the engine's throughput counter. Then make the decision from the workload outcome, not from the most flattering internal ratio.

High draft acceptance can explain a speculative path. It cannot, by itself, justify enabling one.