Tuning Flame serving to match standalone vLLM throughput

Researchers compared the performance of Flame, a Python-based serving framework that embeds vLLM, against standalone vLLM instances. Published on October 24, 2026, the study details how specific tuning strategies allowed Flame to achieve near-parity with the reference engine on two NVIDIA B200 GPUs. The results highlight the importance of prefix caching and asynchronous request admission in minimizing overhead for large language model deployments.

What happened

The evaluation focused on two distinct workloads using the Llama 3.1 8B Instruct model. The first workload involved 64 requests with no document reuse, while the second involved 256 requests where four questions were asked about each of four documents, enabling prefix reuse. In both scenarios, the system processed short, finite bursts of traffic rather than sustained steady-state loads. The goal was to determine if Flame could maintain at least 99% of the throughput achieved by a carefully configured standalone vLLM setup.

Initial tests revealed a significant gap when using default configurations. However, after implementing persistent worker placement to improve cache reuse, the researchers closed much of the difference. Further optimizations included increasing batch sizes, switching to asynchronous engine admission, and applying lossless compression to inputs. In the final configuration, Flame’s median completion throughput was only 0.55% lower than standalone vLLM without document reuse and 0.86% lower with prefix reuse. While two out of five trial pairs missed the strict 99% target during reuse tests, the worst-case deviation was just 2.03%.

How it works

The core mechanism for improving performance in both systems was prefix affinity. When an assistant answers multiple questions about the same document, it repeatedly processes the same initial tokens. By keeping that prefix on a single GPU, the system avoids rebuilding attention keys and values from scratch. The researchers implemented a client-side map that hashes token blocks to assign unseen prefixes to the shard with the least estimated prefill work. This ensured that subsequent questions about a specific document always reached the same worker, allowing vLLM to reuse resident KV blocks effectively.

Beyond caching, the team addressed how requests were admitted to the inference engine. The original Flame adapter passed batches synchronously, causing later tasks to wait for current invocations to complete. The optimized version used an asynchronous API that admitted all requests in a batch concurrently. It also employed zlib level 1 compression to reduce the size of transmitted inputs, which is particularly effective for text-heavy prompts. Finally, freezing the Python startup heap reduced garbage collection pauses during inference, further stabilizing throughput.

Key details

• The experiments used two NVIDIA B200 GPUs running vLLM 0.28.0 and Torch 2.13.0+cu130.
• Prefix affinity increased median throughput by 77.5% for standalone vLLM and 73.4% for Flame in reuse scenarios.
• Final median throughput for Flame was 7.586 requests/s without reuse and 28.350 requests/s with reuse.
• Input compression reduced the encoded service input size for the reuse burst from over 6 MB to roughly 388 KB.
• Freezing the startup heap reduced maximum per-worker garbage collection pauses from 8.80 ms to 4.59 ms.
• The study did not demonstrate a performance advantage from Flame’s data-aware scheduler in these specific tests.

Why it matters

For engineers building production LLM services, this comparison clarifies where overhead actually comes from in wrapped serving frameworks. It shows that the penalty for using a higher-level orchestration layer like Flame is not inherent to the architecture but stems from suboptimal batching and synchronization defaults. By understanding that prefix locality and asynchronous admission are critical, developers can tune their own stacks to minimize latency spikes and maximize GPU utilization.

The findings also caution against assuming that framework features automatically translate to performance gains. The data-aware scheduler in Flame did not outperform the simpler explicit client mapping used in the tests. This suggests that for many use cases, straightforward routing strategies combined with careful resource management may be more effective than complex scheduling algorithms. Teams should prioritize measurable factors like cache hit rates and batch occupancy over theoretical scheduling advantages.

What you can do

• Implement prefix affinity in your client routing to ensure requests sharing context hit the same GPU replica.
• Switch to asynchronous APIs for engine admission to prevent batch processing from blocking new requests.
• Enable lossless compression for text-heavy inputs to reduce network and memory overhead during transfer.
• Tune batch formation windows and limits to balance between latency and throughput based on your traffic patterns.
• Monitor garbage collection pauses and consider freezing long-lived objects during worker initialization.
• Validate performance with finite burst tests that mimic your actual usage patterns, including prefix reuse scenarios.