A practical guide to self-hosting LLM inference for platform teams

As organizations seek to reduce reliance on third-party APIs and secure sensitive data, platform teams are increasingly tasked with running large language models internally. A new technical guide from Modelplane outlines the essential practices for inference engineering, detailing how to manage GPUs, optimize memory, and measure performance in production environments as of October 2026.

What happened

The rise of high-quality open models has made self-hosting a viable alternative to hosted APIs for many enterprises. With over two million models available on Hugging Face and competitive performances from releases like DeepSeek R1, companies can now run their own infrastructure to control latency, uptime, and data residency. This shift transforms platform teams into inference teams, requiring them to manage the full lifecycle of model serving rather than just consuming external endpoints.

Self-hosting becomes economically attractive when API spending grows significantly or when specific reliability and data governance needs exceed what vendors offer. At high volumes, running models internally can reduce costs by approximately 80 percent compared to commercial APIs. The primary driver for this cost reduction is not better hardware, but more efficient serving strategies, as the price of fixed AI capability drops roughly tenfold annually due to software improvements.

The guide emphasizes that inference engineering is essentially site reliability engineering applied to a new workload. Skills in scheduling, capacity planning, and observability transfer directly, though the metrics change. For instance, autoscaling must respond to queue depth and key-value cache usage rather than CPU load, and rollouts involve complex matrices of model versions, hardware types, and engine builds. The role is still emerging, with no settled job title, but demand is growing as more companies adopt internal AI infrastructure.

How it works

An inference engine acts as a stateful server that loads model weights into GPU memory and manages per-session state, similar to a database with a specialized accelerator. The stack consists of four layers: kernels that perform arithmetic, the engine that schedules work, a control plane that manages deployments, and a gateway that routes traffic. Engines like vLLM or SGLang handle the heavy lifting, converting static weights into responsive APIs while managing the critical resource of GPU memory.

Memory management is the central challenge in inference. Model weights are fixed and must reside entirely in GPU memory, while the key-value (KV) cache grows dynamically with each conversation. The KV cache stores attention keys and values for every token in flight, allowing the model to avoid recomputing history. Its size depends on context length and concurrent users, often becoming the limiting factor for how many requests a single GPU can handle. For example, a 70-billion parameter model at half-precision requires significant memory just for weights, leaving limited space for the cache.

Request processing occurs in two distinct phases with different bottlenecks. The prefill phase reads the entire input in parallel to generate the first token, limited by compute power. The decode phase generates subsequent tokens one by one, limited by memory bandwidth as it reads weights and the growing KV cache. Optimizations must target the correct bottleneck; techniques that speed up prefill may not improve decode speed. Metrics like time to first token and tokens per second help engineers identify which phase needs tuning for their specific workload.

Key details

• Memory math matters: An 8-billion parameter model at 2 bytes per parameter needs about 16 GB of GPU memory for weights alone, leaving the rest for the KV cache and runtime overhead.
• Two-phase processing: Prefill is compute-bound and determines time to first token, while decode is memory-bound and determines streaming speed.
• Cost calculation: Cost per million tokens is derived from GPU hourly rate divided by throughput, allowing direct comparison with hosted API prices.
• Agent workloads differ: Coding agents send long, repetitive prompts, with median sizes reaching 59,000 tokens, making prefix caching highly effective.
• Quantization benefits: Storing weights in fewer bits frees up memory for the KV cache, allowing more concurrent users without changing the model architecture.
• Scaling signals: Autoscaling should trigger based on queue depth and KV cache utilization, not just CPU or GPU utilization percentages.

Why it matters

For software builders, understanding inference engineering enables greater control over product performance and cost structure. Relying on external APIs introduces latency variability and data privacy risks that may be unacceptable for enterprise applications. By mastering the inference stack, teams can tailor their infrastructure to match specific workload patterns, such as optimizing for low-latency interactive chat or high-throughput batch processing.

The shift also impacts hiring and team structure. As inference roles become more common, platform engineers need to update their skill sets to include GPU memory management and model-specific observability. Recognizing that most optimization gains come from software efficiency rather than hardware upgrades allows leaders to invest in the right tools and training. This knowledge empowers teams to build resilient AI features that scale predictably as user demand grows.

What you can do

• Audit current API spend: Calculate your cost per million tokens and compare it with estimated self-hosted costs using current GPU pricing.
• Start with small models: Deploy a smaller open model to learn the operational basics of weight loading, KV cache management, and monitoring.
• Monitor key metrics: Track time to first token, inter-token latency, and goodput to understand user experience and system efficiency.
• Enable prefix caching: If your application involves repeated prompts or agent workflows, ensure your engine uses prefix caching to reduce redundant computation.
• Plan for memory overhead: Leave sufficient GPU memory for runtime overhead and fragmentation, not just the raw model weights and estimated cache.
• Define SLOs clearly: Establish service level objectives for latency and availability based on whether your workload is interactive or batch-oriented.