How OCRia achieved a 10x inference speedup on B200 GPUs

The engineering team behind OCRia-1 has detailed how they achieved a nearly tenfold increase in token generation speed, moving from approximately 285 tokens per second to 2800 tokens per second on a single NVIDIA B200 GPU. Published on November 3, 2026, the technical breakdown reveals that this performance leap was not the result of a single breakthrough but rather a systematic reduction of memory bottlenecks and computational overhead across the entire inference stack.

What happened

Latency has long been a primary constraint for document parsing workflows, where dense pages require lengthy structured outputs. The OCRia team identified that their inference pipeline was memory-bandwidth-bound rather than compute-bound, meaning the GPU spent more time moving model weights and state data than performing mathematical operations. To address this, they focused on three core areas: reducing the volume of data moved during each decode step, generating more tokens per forward pass, and minimizing the CPU and GPU overhead associated with each step.

The resulting optimization strategy combined established techniques like quantization with novel implementations of speculative decoding and custom kernel development. By rigorously profiling every stage of the inference process, the team eliminated redundant memory transfers and streamlined metadata preparation. These changes allow the system to handle concurrent requests more efficiently, enabling faster iteration on benchmarks and quicker results for end users without sacrificing accuracy.

How it works

The foundation of the speedup lies in workload-aware quantization. Rather than applying uniform precision reduction, the team used mixed precision, keeping sensitive layers at higher precision while compressing others to FP8 or NVFP4. They employed activation-aware quantization, which calibrates scaling factors using in-distribution data to ensure the compression remains effectively lossless for target tasks. Notably, the vision encoder was left unquantized to preserve visual accuracy, demonstrating a targeted approach to balancing size and fidelity.

To overcome the autoregressive limitation of generating one token per step, the team implemented speculative decoding using a drafter model called DFlash2. This smaller model predicts multiple future tokens in parallel, which the larger target model then verifies in a single forward pass. DFlash2 uses local convolutions and a candidate selector to achieve acceptance rates of over 12 tokens, significantly increasing throughput. The process is mathematically guaranteed to be lossless, as any incorrect speculations are discarded without affecting the final output distribution.

Further gains came from deep engine optimizations. The team adapted ReplaySSM to optimize state management in Gated Delta Networks, reducing the cost of rolling back rejected speculative tokens. They also fused consecutive operations, such as SiLU activation and quantization, to keep intermediate results in fast GPU registers instead of writing them to high-bandwidth memory. On the CPU side, they refactored metadata construction to avoid redundant calculations across different cache groups, reducing the number of GPU launches required per decode step from hundreds to just one.

Key details

• Performance gain: Token generation speed increased from ~285 tokens/sec to ~2800 tokens/sec on a single B200 GPU.
• Quantization strategy: Mixed precision using BF16, FP8, and NVFP4, with activation-aware calibration to maintain accuracy.
• Speculative decoding: Uses DFlash2, a block-diffusion drafter model that achieves acceptance rates of over 12 tokens per pass.
• Kernel optimization: Custom CUDA kernels reduced ReplaySSM verification time from 72–80 µs to 23–24 µs per layer.
• CPU efficiency: Metadata building was optimized to require only one GPU launch per step, down from approximately 290.
• Availability: OCRia-1 is currently available on LlamaParse, and generalizable optimizations will be contributed back to vLLM.

Why it matters

For engineers building document AI products, this case study highlights that raw hardware power is often insufficient without careful software optimization. The shift from compute-bound to memory-bound operations in large models means that reducing data movement is frequently more impactful than adding more compute units. Understanding the roofline model and identifying whether a workload is limited by bandwidth or computation allows teams to prioritize the right optimizations, such as quantization or kernel fusion, rather than blindly scaling resources.

The use of speculative decoding demonstrates a practical path to breaking the sequential bottleneck of autoregressive generation. By verifying multiple tokens in parallel, developers can significantly improve interactivity and throughput for users dealing with long documents. However, this requires careful implementation to ensure that the drafter model aligns well with the target model, as poor acceptance rates can negate the benefits. The OCRia team’s success with DFlash2 shows that investing in specialized drafter architectures can yield substantial returns.

Finally, the emphasis on profiling and incremental improvement underscores the importance of measurement in systems engineering. Many of the gains came from small, compounding optimizations like fusing operations or refining metadata handling. These changes, while individually modest, collectively contributed to over 60% of the additional throughput. This approach encourages developers to look beyond high-level framework settings and dive into the implementation details of their inference engines to unlock hidden performance.

What you can do

• Profile your inference pipeline to determine if it is memory-bandwidth-bound or compute-bound before choosing optimization strategies.
• Implement mixed-precision quantization with activation-aware calibration to reduce memory usage without significant accuracy loss.
• Explore speculative decoding with a lightweight drafter model to generate multiple tokens per forward pass.
• Fuse consecutive kernel operations to minimize writes to high-bandwidth memory and keep data in registers.
• Optimize CPU-side metadata preparation by caching shared structures and reducing redundant GPU launches.
• Contribute generalizable optimizations back to open-source inference engines like vLLM to benefit the broader community.