Using decision models for scalable agent evaluation in AWS Bedrock

Amazon Web Services has expanded its Amazon Bedrock AgentCore Evaluations capability to support custom evaluators powered by decision models. Released in November 2026, this update allows developers to integrate specialized models such as TypeSafe’s Jev and the open-source Strands Decider 2B directly into their agent assessment pipelines. This approach offers a structured alternative to general-purpose large language model judges for specific, repeatable validation tasks.

What happened

AgentCore Evaluations is designed to score AI agents throughout their lifecycle, from development to production traffic. While the service already includes 15 built-in evaluators that use large language models (LLMs) to assess responses against rubrics, it now supports custom evaluators defined as either LLM-as-a-judge or code-based logic. The new focus is on code-based evaluators that can call external systems, including decision models, via AWS Lambda functions.

Decision models differ from standard LLMs by returning calibrated probabilities over a fixed set of answers rather than generating free-form text. This makes them ideal for narrow questions where an informed person could make a quick judgment, such as verifying if a response stays on topic or if a tool call is safe. By integrating these models, developers can create a screening layer that flags issues efficiently before escalating complex cases to more expensive, explanatory LLM evaluators.

The announcement highlights two specific backends for this integration: Jev, a hosted service from TypeSafe, and Strands Decider 2B, an open-source model from the Strands Agents team. Both support three question types: binary yes/no (noul), multiple-choice (choice), and ordered levels (score). This structure allows for precise, programmatic thresholds that are difficult to achieve with the variable output of generative models.

How it works

The integration relies on AWS Lambda functions acting as the bridge between AgentCore Evaluations and the decision model. When an evaluation is triggered, the Lambda function receives OpenTelemetry spans containing the agent’s interaction data. A normalization module formats this data into a request suitable for the decision model, sending only the relevant turn history for trace-level checks or the full session for broader assessments.

Once the decision model processes the input, it returns a probability distribution. The Lambda adapter converts this distribution into a format AgentCore expects. For binary questions, the probability of "true" becomes the value. For scored questions, the system calculates a probability-weighted expected level scaled between zero and one. This result is then registered as a code-based evaluator within the AgentCore control plane, allowing it to run alongside built-in metrics like correctness and helpfulness.

Developers can deploy this setup using provided Python scripts that handle role creation, Lambda packaging, and evaluator registration. The system supports both on-demand batch evaluations against stored sessions and continuous online evaluation of live traffic. Flags in the deployment script allow users to opt-in to Jev or Strands Decider backends, with options to auto-provision necessary EC2 instances for self-hosted models.

Key details

• Decision models return calibrated probabilities over fixed options, unlike LLMs that generate free-form text and rationales.
• Two supported backends are Jev (hosted by TypeSafe) and Strands Decider 2B (open-source, self-hosted).
• Three question types are supported: noul (yes/no), choice (up to 255 options), and score (2-10 ordered levels).
• Code-based evaluators run in AWS Lambda, receiving OpenTelemetry spans and returning structured results to AgentCore.
• The "cascade pattern" recommends using decision models for cheap, fast screening and escalating only low-confidence turns to LLM evaluators.
• Validation requires comparing decision model outputs against human-labeled samples to check agreement rates and calibration.

Why it matters

For engineers building production AI agents, cost and consistency are major concerns. General-purpose LLM evaluators provide rich explanations but are expensive and can suffer from confidence drift. Decision models offer a way to enforce specific rubrics consistently without the variability of generative text. By using them as a first pass, teams can reduce the volume of interactions sent to larger models, lowering operational costs while maintaining high coverage.

This approach also improves the reliability of automated testing. Because decision models output numerical probabilities, developers can set precise thresholds for passing or failing specific criteria. This is crucial for compliance-heavy domains like HR or finance, where policies must be applied uniformly. The ability to threshold and aggregate these scores programmatically allows for more robust monitoring and alerting systems.

Furthermore, the separation of "reading" versus "deriving" tasks clarifies the evaluation strategy. Narrow checks for groundedness or goal completion can be handled by decision models, while complex reasoning tasks remain with LLM judges. This hybrid architecture leverages the strengths of both technologies, ensuring that deep investigative power is reserved for cases where simple binary checks are insufficient.

What you can do

• Identify narrow, repeatable evaluation questions in your agent workflow, such as policy adherence or fact grounding.
• Build a labeled sample of your agent’s traffic with human reference labels to validate any new decision model evaluator.
• Deploy the provided Python scripts to set up Lambda-based evaluators for Jev or Strands Decider in your AWS account.
• Implement the cascade pattern by setting confidence thresholds that trigger deeper LLM evaluation only when necessary.
• Monitor calibration by grouping turns by reported probability and comparing them against human agreement rates.
• Re-run validation checks whenever you update the decision model version or reword the evaluation questions.