D-Cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding

Jul 17, 2026

D-Cut is an adaptive pruning method for speculative decoding in large language models that selects draft tokens jointly across a batch, focusing verification resources on tokens most likely to be accepted. By using cross-request pruning and a runtime cost model, D-Cut adapts to different deployment environments. Experiments show that under high concurrency, D-Cut increases average speedup from 1.26× to 1.65× and achieves up to 3.0× speedup on mixture-of-experts (MoE) models compared to autoregressive decoding.

Why it matters: D-Cut offers a significant advance in efficient large language model inference, particularly under high concurrency, by reducing wasted computation and improving throughput without compromising output quality.

Full story at: arXiv Computation and Language ↗