Speeding Up LLM Inference with N-gram Speculative Decoding in Nutanix Enterprise AI

What is Speculative Decoding?

Speculative decoding is an optimization technique designed to speed up the inference process of large language models (LLMs) without compromising output quality, and can be enabled in Nutanix Enterprise AI when configuring the inference endpoint by enabling vLLM’s n-gram prediction mechanism.

In standard decoding, an LLM generates tokens one by one sequentially, which can be slow. Speculative decoding addresses this by employing a smaller, faster "draft" algorithm alongside the primary, larger target model. Drafting predicts a sequence of several potential future tokens. Then, the target model verifies all these speculative tokens in a single parallel step.

Speculative decoding has three steps:

1. Speculation phase
2. Verification phase
3. Rejection sampling of tokens

The drafting can be done in multiple ways. For example, it can be a statistical calculation (like n-gram), a separate LLM, an MLP layer (like Medusa), or any small neural network that can generate multiple tokens faster than the verifier LLM, and as close to its distribution as possible.

When is Speculative Decoding Effective?

Speculative decoding is not a silver bullet. It is effective only under certain conditions.

Conceptual Constraints

The performance gain offered by speculative decoding is functionally tied to the predictive agreement between the draft and the verifier. A higher acceptance rate of draft tokens leads to lower latency.

Ensuring this alignment imposes strict prerequisites on the models employed:

• Shared Tokenization Space: The draft and verifier models must share an identical tokenizer and vocabulary. The speculative process relies on passing sequences of token IDs; therefore, a specific ID must represent the exact same sub-word unit in both models for validation to occur meaningfully.
• Distributional Similarity: To maximize the acceptance rate, the probability distribution of the draft over the vocabulary space should be as close as possible to that of the verifier model.

System Constraints

Speculative decoding is a compute-for-memory trade. It is a massive win when the GPU is idle, but a net loss when the system is busy.

LLM inference is usually memory-bound. Speculative decoding uses this spare compute to run a draft and verify multiple tokens at once, bypassing the memory bottleneck.

The spare compute disappears when the GPU is fully utilized. This typically happens under three conditions:

• High QPS (Queries Per Second): When thousands of requests hit the server, the GPU stays 100% busy just keeping up with the volume. There is no "free" time left for drafting.
• High Parallelization (Batching): Processing many users simultaneously increases arithmetic intensity. Large batches saturate the GPU's math cores, making it compute-bound.
• Total GPU Utilization: Once the model's math completely fills the GPU’s capacity, any extra work (like running a draft or verifying rejected tokens) is pure overhead that slows down the entire system.

When to Use Speculative Decoding?

Speculative decoding primarily improves inter-token latency, i.e., time between generation of consecutive tokens. So, it is useful when reducing this time is advantageous.

For example, it may be beneficial for latency sensitive applications like RAG, chatbots, and agents.

It is generally not beneficial for scenarios with large input prompts and small output. Since the output is small, the gain will not be meaningful. It will also not be useful when the endpoint is used for high-throughput tasks, like large batch, very high concurrency workloads.

Why use Speculative Decoding?

Speculative decoding has the following advantages -

• Mathematically Lossless: Unlike most speed-up techniques, this is "lossless." The verification step ensures the final output is identical to what the large target model would have produced alone. You get the speed of a small model with the full intelligence of the large one.
• Orthogonal to Other Optimizations: It works perfectly alongside architectural optimizations like Flash Attention or Flex Attention, or quantization techniques.
• Dynamic Flexibility: The drafting algorithm can be swapped based on the task. Using a specialist draft model (e.g., one trained on code) for specific prompts significantly boosts the token acceptance rate and overall speed.

How Does Speculative Decoding Improve Performance?

The most expensive part of vLLM inference is moving billions of parameters from vRAM to the processor for forward pass computations. When standard decoding is happening, only one token is generated at once. The next set of weights is loaded again when the next cycle of inference starts. At lower batch sizes, processor cores are relatively idle. This makes decoding memory bound.

Speculative decoding improves this by decoding multiple tokens at once when the large model (validator) validates the draft. This improves the number of computations per bytes moved, improving the arithmetic intensity of the GPU.

N-gram Speculative Decoding

While many speculative decoding implementations rely on a smaller "draft" neural network (like a TinyLlama-160M for a Llama-70B), N-Gram Speculative Decoding takes a different approach. Instead of using a second model that requires its own memory and compute, it uses a simple, statistical heuristic based on the text already generated in the current window.

Think of it as the "Auto-Complete" of the LLM world. It assumes that if a specific sequence of words has appeared before, it is likely to appear again in the same way.

How It Works: The "Look-Back" Mechanism

N-gram speculation doesn't predict the future using weights; it matches the past. The process generally follows these steps:

1. Identify the Suffix: The system looks at the last n−1 tokens generated (the "suffix").
2. Scan the Context: It searches the previous prompt and the already-generated response for previous occurrences of that exact suffix.
3. Speculate: If a match is found, it "speculates" that the tokens following the historical match will be the same tokens that should follow the current suffix.
4. Verify: These tokens are bundled and sent to the Target Model for parallel verification, just like any other speculative method.

Why Use N-Grams Over a Draft Model?

If we have small LLMs available, why bother with a simple n-gram lookup? There are three primary reasons:

• Zero Cold-Start / Memory Overhead: You don't need to load a second model into VRAM. This is crucial for edge devices or systems already pushed to their memory limits.
• Domain Agnostic: A draft model trained in English might struggle with Python code or medical jargon. An n-gram speculator, however, naturally adapts to whatever "language" is currently in the context window because it copies directly from it.
• Extreme Speed: Searching a small KV cache or a text buffer for a matching string is orders of magnitude faster than running even the smallest neural network.

Limitations: When Does It Fail?

N-gram speculation is highly effective for repetitive structures, like code (boilerplate headers), structured data (JSON keys), or formal documents (legal disclaimers).

However, it struggles in:

• Creative Writing: Where the model rarely repeats long sequences of tokens.
• Short Contexts: If there is no "history" to look back on, there are no n-grams to match, leaving the target model to fall back to standard, one-by-one decoding.

Key Insight: N-gram speculative decoding is the ultimate "low-hanging fruit" for LLM optimization. It provides a significant speedup for structured tasks with virtually zero architectural complexity.

N-gram Speculative Decoding in Nutanix Enterprise AI

N-gram Speculative tokens can be enabled in NAI using the Create Endpoint user journey. It can be enabled only on GPU-based endpoints.

How to Enable?

It is available inside Advanced Configuration with a vLLM engine. You can check Enable Speculative Token Generation. For now, only n-gram is supported and it uses two parameters - promptLookupMax and numSpeculativeTokens.

The following table gives more information about the parameters and to set them. The NAI default, after experimentation, is set to 5 and 4 for speculation length and maximum prompt lookup.

Press Next, verify the details and create the endpoint. You should be able to verify the details in the Advanced Configuration Details page after the endpoint is created.

Benchmarking n-gram Speculative Decoding in Nutanix Enterprise AI

To benchmark speculative decoding, we will compare different configuration parameters, models, and measure the inter-token latency (ITL).  ITL is the time between two consecutive tokens, and is the primary metric that speculative decoding improves. The lower is the ITL, the better is the performance.

To evaluate the impact of speculative decoding on inter-token latency, we benchmarked Mixtral-8x7B-Instruct-v0.1. Using the CNN Dailymail dataset for a summarization task, we established a baseline without speculative decoding (values = 0) and compared it against various speculative settings. 

Note: Performance numbers are as per Nutanix Enterprise AI release 2.6 (February 2026)

The benchmarking data reveals that speculative decoding provides an approximate 18% improvement in inter-token latency (ITL) compared to the baseline. At low Requests Per Second (RPS), utilizing a higher number of speculative tokens is particularly effective, as it optimizes the compute-memory trade-off. However, as RPS increases, the system's default compute requirements naturally rise, which leads to a diminishing return on gains from higher speculative token counts.

When examining the relationship between RPS and the maximum prompt lookup, ITL shows proportional sensitivity to the prompt length as load increases. This indicates that the prompt-lookup max is primarily a task-oriented parameter, highly dependent on the nature of the input data, rather than a lever for system-level performance tuning.

A box plot analysis of ITL distribution at 5.32 RPS with (red) and without (blue) speculative decoding in the above graph highlights that while speculative decoding reduces the average latency, it introduces higher overall variability. Interestingly, the distribution between the p50 and p90 metrics is actually tighter in the non-speculative baseline. This suggests that while speculative decoding offers significant macro-gains, the consistency of token delivery can fluctuate more than standard generation based on the input request.

The Time to First Token (TTFT) remains largely unaffected by speculative decoding, confirming that the prefill phase is not fundamentally altered. We observed slightly higher macro TTFT values with speculative decoding enabled, which is attributed to the initial overhead of the system building the prompt-lookup table before generation begins.

Comparison of Models

We also evaluated ITL of different models as validators under default speculative decoding configuration parameters, and the results are given in the table below.

The efficacy of speculative decoding scales with model size; larger models consistently outperformed smaller ones in terms of relative gains. This is primarily due to the stochastic nature of smaller validator models, which leads to lower draft acceptance rates (15–25%). For these models, the speculative overhead can nearly negate the benefits. In contrast, the larger models achieved draft acceptance rates between 22–40%, resulting in more consistent performance boosts. Across the board (excluding outliers), gains hovered around the expected value of 15%.

Conclusion

Speculative decoding improves LLM inference speed by a draft mechanism, such as n-gram matching or an MLP layer, to predict sequences of tokens that the main model then verifies in parallel, offering a "lossless" performance boost of approximately 15–18%. Nutanix Enterprise AI simplifies this process by integrating n-gram speculation directly into the endpoint creation workflow, allowing users to enable it within Advanced Configuration without needing separate, complex draft models. By bypassing memory bottlenecks, NAI provides a highly efficient way to reduce inter-token latency for structured tasks like code generation or RAG applications.

©2026 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product and service names mentioned are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned are for identification purposes only and may be the trademarks of their respective holder(s).This content reflects an experiment in a test environment. Results, benefits, savings, or other outcomes described depend on a variety of factors including use case, individual requirements, and operating environments, and this publication should not be construed as a promise or obligation to deliver specific outcomes.