Mistrial measures LLM judge reliability with statistical rigor

Software teams increasingly rely on large language models to evaluate other AI outputs, but these judges often produce inconsistent results across identical inputs. Mistrial, a new open-source utility released on GitHub, addresses this instability by running evaluators multiple times and applying statistical methods to measure their reliability before they impact production decisions.

What happened

Developers shipping AI features frequently use LLMs as judges to score model outputs, gate continuous integration pipelines, and select winning prompts. However, because these judges sample from a probability distribution, they can return different verdicts for the same input when asked twice. This variability, often called flakiness, remains hidden when evaluation platforms report only a single score per case without measuring the underlying noise.

The creator of Mistrial highlights that raw percent agreement significantly overstates reliability compared to chance-corrected metrics like Cohen’s kappa. Research cited in the project documentation indicates that raw agreement can be 30 to 40 points higher than kappa across various judges. Despite the availability of standard statistical tools like test-retest reliability and Krippendorff’s alpha, these methods rarely appear in engineering workflows, leaving teams unaware of how much their evaluation scores fluctuate.

Existing tools such as LangSmith, Braintrust, and Langfuse typically run a judge once per case. While some allow experiment repetition, they do not natively report self-agreement or identify specific cases where verdicts flip between runs. Other academic or niche projects like judgecheck and Assay address parts of this problem but lack integration with existing evaluator functions or clear guidance on handling flaky results in CI pipelines.

How it works

Mistrial operates by pointing at an existing evaluator, which can be a specific model name, a custom Python function, or an integration with platforms like LangSmith. It executes the judge multiple times across one or more models and under different rubric wordings. The tool then calculates the spread of results, reporting metrics such as self-agreement, flipped cases, and inter-judge agreement using kappa and alpha statistics.

The tool provides a "reliability card" that summarizes these findings. It identifies exactly which cases caused disagreement among judges or across repeated runs. For instance, in a demo involving thirty-two tricky cases, the tool revealed that weaker models flipped verdicts on up to sixteen percent of inputs, while stronger models remained stable. It also detects parser errors, ensuring that formatting issues are not mistaken for model failures.

Beyond measurement, Mistrial offers probes to simulate production risks. It can estimate the probability that a CI gate will fail falsely due to judge variance, check how sensitive verdicts are to rubric rewording, and compare results between model versions. The system caches all responses locally, allowing users to re-analyze data without incurring additional API costs.

Key details

• Mistrial calculates self-agreement, Cohen’s kappa, and Krippendorff’s alpha to quantify judge consistency beyond simple accuracy.
• The tool supports multiple providers including OpenAI, Anthropic, Google Gemini, Groq, and local models via Ollama or custom endpoints.
• It identifies "flipped cases" where the same judge gives different verdicts on identical inputs across multiple runs.
• A gate-risk probe uses Monte Carlo simulation to predict how often a CI pipeline might fail falsely due to judge noise.
• The package includes demo judges that are intentionally flaky, allowing developers to test the tool without spending API credits.
• Results are exported as JSON, HTML reports, and markdown summaries suitable for pull request reviews.

Why it matters

For engineers building AI products, trusting an evaluation score is as critical as the score itself. If a judge is flaky, a CI gate based on its output may block valid deployments or approve poor ones purely due to sampling variance. Mistrial shifts the focus from asking "how good is my model" to "can I trust the number telling me," providing a safety layer for automated decision-making.

The tool also highlights that stronger models are not immune to inconsistency, though their failures occur in more nuanced areas. In tests with advanced models, flips occurred in cases involving reasonable interpretations of rounding or implicit units. Identifying these edge cases helps teams refine their rubrics and understand where human review is still necessary, rather than assuming automation is perfect.

By integrating statistical rigor into the evaluation loop, teams can avoid the pitfalls of relying on single-shot judgments. This approach reduces the risk of deploying features based on noisy data and provides a clear path to stabilizing evaluation pipelines through majority voting or adaptive re-asking strategies.

What you can do

• Install Mistrial via pipx and run the demo command to see how flaky judges behave on synthetic data without using API keys.
• Point the tool at your existing evaluator function or model configuration to generate a reliability card for your current setup.
• Use the gate-risk probe to determine if your current CI pass mark is safe given the observed variance in your judge.
• Analyze the "flipped cases" report to identify ambiguous inputs that require clearer rubrics or human review.
• Compare different judge models using inter-judge agreement metrics to select the most stable option for your workload.
• Cache your evaluation runs locally to iterate on rubric wording and analysis without incurring repeated inference costs.