Why LLMs remain non-deterministic even at temperature zero

In September 2025, researchers at Thinking Machines demonstrated that large language models (LLMs) fail to produce identical outputs even when configured for maximum determinism. By sending the same prompt one thousand times with temperature set to zero, they received eighty different completions. This finding underscores a critical architectural challenge for developers building production AI systems: generative models are inherently probabilistic, not deterministic.

What happened

The experiment focused on a simple prompt asking for information about Richard Feynman. The team used an open model with temperature zero, a setting theoretically designed to always select the most likely next token. Despite this constraint, the outputs diverged after the first 102 tokens. While 992 responses stated Feynman was born in Queens, New York, eight responses cited New York City. This variation occurred despite identical inputs and seeds, proving that standard configuration parameters do not guarantee repeatability.

This behavior is not limited to academic benchmarks. In production environments, developers often encounter situations where a demo works perfectly, but the same input yields different results during client presentations or live usage. The issue becomes critical when AI agents make decisions involving money, access control, or compliance. Unlike traditional software, where a bug produces the same error every time, an LLM might give a correct answer on one run and an incorrect one on the next, making debugging and auditing significantly more difficult.

The root cause lies in how inference servers operate. Providers group requests into batches to optimize GPU utilization. Research indicates that common GPU kernels are not batch-invariant, meaning the mathematical results can shift slightly depending on the number of concurrent requests. When two potential tokens have nearly identical probabilities, these tiny numerical differences cause the model to pick different paths, leading to divergent text generation. Additionally, provider-side updates to serving stacks or model versions can change behavior without notice, further breaking reproducibility.

How it works

Deterministic systems, such as rule engines or trained classifiers with frozen weights, function as pure functions. They map specific inputs to specific outputs without sampling from a distribution. If you run them twice with the same data, you get the exact same result, byte for byte. This allows for unit testing, version diffing, and clear audit trails. In contrast, LLMs do not return answers directly. They calculate a probability distribution for every possible next token and then sample from that distribution.

Even when temperature is set to zero, the sampling process is vulnerable to external factors. The seed parameter offered by many providers is described as best-effort. It cannot account for changes in the underlying hardware state, batch sizes, or silent model updates. Furthermore, agents often rely on external tools like search engines or databases. Since these sources change over time, the context provided to the model drifts, ensuring that no two runs are truly identical. As noted in a 2024 study, accuracy on benchmark tasks can vary by up to fifteen percent between runs under deterministic settings, with gaps reaching seventy percent in extreme cases.

Key details

• Thinking Machines observed 80 different completions from 1,000 identical prompts at temperature zero.
• Divergence typically occurs when candidate tokens have nearly equal probabilities, influenced by server-side batching.
• Batch-invariant kernels can enforce determinism but may increase inference latency from 26 to 55 seconds.
• Provider model aliases may point to updated versions, causing silent behavior changes in production.
• Traditional ML classifiers offer fully repeatable outputs between releases, unlike generative models.
• Non-determinism concentrates on borderline cases where the model is genuinely uncertain.

Why it matters

For software engineers and technical founders, treating LLMs as deterministic components leads to fragile architectures. When an AI system controls financial transactions, user permissions, or regulatory compliance, variability is a liability. A customer complaint regarding a denied refund cannot be resolved by rerunning the prompt, as the model may provide a different justification or outcome. This lack of reproducibility complicates regression testing and makes it difficult to explain decisions to auditors or legal teams.

Understanding the distinction between deterministic and probabilistic AI allows teams to allocate tasks appropriately. Generative models excel at interpreting unstructured data and drafting text, but they should not be solely responsible for final decisions. By recognizing that nondeterminism is a feature of the technology rather than a bug, developers can design systems that isolate high-stakes logic within deterministic layers. This approach ensures that the product remains testable, explainable, and reliable, even when the underlying language model behaves unpredictably.

What you can do

• Implement hard rules in code for limits, permissions, and arithmetic rather than relying on prompt instructions.
• Use calibrated classifiers for high-volume decisions where labeled historical data is available.
• Constrain LLM outputs to strict JSON schemas with enumerated values to reduce variation.
• Define explicit rating scales in prompts with concrete examples to anchor model judgments.
• Run critical queries multiple times and use voting or averaging to estimate confidence levels.
• Log all inputs, model versions, and outputs to enable auditing without regenerating responses.