Building with AI

Automated hill-climbing harness tunes vLLM and SGLang on single A100

A new open-source Kubernetes harness automates coordinate-ascent tuning for LLM inference engines, completing 432 trials to optimize vLLM and SGLang configurations.

A glass cube with glowing circuits next to notes and a stopwatch.
Illustration generated for this article

Johnson Shi released an automated Kubernetes harness on November 22, 2026, designed to tune large language model inference engines through coordinate-ascent hill climbing. The tool targets vLLM and SGLang, running a comprehensive 432-trial benchmark study on a single NVIDIA A100 80GB GPU to identify optimal serving configurations.

What happened

The repository, titled a100-inference-hillclimb, provides a structured method for optimizing inference performance without manual trial and error. It executes isolated trials within Kubernetes pods, ensuring that each configuration change is tested under consistent hardware conditions. The system pairs the inference server container with a benchmark sidecar, allowing for precise measurement of latency, throughput, and resource utilization.

Shi’s approach focuses on reproducibility and noise reduction. By pinning each trial to a dedicated GPU node, or "lane," the harness eliminates interference from other processes. The system automatically sizes shared memory, captures server logs, and samples GPU metrics such as power, clock speed, and utilization via nvidia-smi. All structured trial metrics are recorded in a SQLite database, creating a permanent ledger of every experiment run.

The project includes a full report of 337 benchmark trials specifically for the Nanbeige4.3-33B model. This dataset details which configuration changes improved performance and which caused regressions. Engineers can access these results directly through the provided RESULTS.md file or by querying the results/ledger.sqlite database, offering a transparent view into the optimization process.

How it works

The core mechanism relies on versioned workload plans and specialized sidecar drivers. The harness defines several types of tests, including single-user decode benchmarks and Poisson capacity searches. In capacity searches, the system uses a geometric rate ladder followed by bisection to find the maximum request arrival rate where at least 90 percent of requests meet strict service level objectives. These objectives require time to first token (TTFT) to remain under two seconds and time per output token (TPOT) to stay below 100 milliseconds.

To ensure reliability, the harness employs noise-calibrated coordinate ascent. When a new configuration challenger attempts to beat the current best setup, or incumbent, it must exceed the incumbent’s score by a statistically significant margin. This margin is defined as the maximum of three standard deviations or a fixed floor value, calculated from repeated runs of the incumbent on the same node. If the challenger succeeds, it must repeat the win in a fresh confirmation run before being accepted as the new incumbent.

Quality gates are integrated directly into the testing loop to prevent performance gains from compromising model accuracy. The system evaluates configurations using 200 GSM8K math problems, long-context passkey retrieval tasks buried at various context lengths, and teacher-forced negative log-likelihood checks against BF16 reference tokens. This ensures that any speed improvements do not come at the cost of reasoning capability or context retention.

Key details

  • The harness supports 22 vLLM and 21 SGLang settings across multiple layers, managed through a canonical config hashing system.
  • Multi-turn agentic sessions simulate 48 concurrent users with six turns each, scoring performance based on p90 TTFT across 288 total turns.
  • Steady-state capacity searches run for 420 to 540 seconds, ignoring data from the initial ramp-up period to ensure stable measurements.
  • Pod-per-trial isolation automatically adjusts /dev/shm size, starting with a 32 GiB base plus any configured CPU KV offload region.
  • The complete 432-trial dataset is stored in results/ledger.sqlite, allowing users to regenerate charts and perform custom analysis.
  • Deployment scripts include pre-commit hooks to prevent leaking environment-specific identifiers like kube contexts or container registries.

Why it matters

For engineering teams managing production LLM services, manual tuning is often a bottleneck. Engineers typically adjust parameters like batch size, quantization levels, and memory allocation based on intuition or limited A/B testing. This harness replaces guesswork with a systematic search algorithm that explores the configuration space efficiently. By automating the climb toward better performance, teams can achieve higher throughput and lower latency without dedicating weeks to manual experimentation.

The emphasis on statistical significance and noise calibration addresses a common pain point in systems performance engineering: variability. Hardware fluctuations and background processes can skew benchmark results, leading to false positives. By requiring challengers to beat the incumbent by three standard deviations and confirming results on the same node, the harness ensures that reported improvements are real and reproducible. This rigor is critical when deploying models at scale, where small inefficiencies compound into significant costs.

Furthermore, the inclusion of quality gates bridges the gap between systems engineering and machine learning. Often, optimizations that improve speed can degrade model output quality. By integrating accuracy checks like GSM8K and passkey retrieval directly into the performance tuning loop, the tool ensures that operational efficiency does not compromise the user experience. This holistic approach allows technical leads to balance cost, speed, and quality with greater confidence.

What you can do

  • Run the offline simulator tests included in the repository to understand the climbing logic without needing a GPU cluster.
  • Inspect the results/ledger.sqlite database to analyze the 432-trial dataset and identify high-performing configurations for your own models.
  • Deploy the harness on your Kubernetes GPU cluster by configuring the .env file with your specific node names and container registry details.
  • Use the provided CLI commands to run individual trials or batch campaigns, allowing you to test specific hypotheses about parameter interactions.
  • Review the docs/decisions.md log to understand the engineering rationale behind the harness design and adapt its principles to your infrastructure.
  • Generate visual reports from your own campaign data using the tools/post_charts.py script to track performance trends over time.
  • #kubernetes
  • #llm inference
  • #performance tuning
  • #vllm
  • #sglang
  • #benchmarking

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories