Turn agent failures into tests with NVIDIA NeMo Compass and Eval Author

On November 4, 2026, NVIDIA published a technical tutorial demonstrating how to use its NeMo platform to systematically debug and improve AI agents. The guide introduces a workflow that connects trace analysis directly to evaluation creation and harness optimization, moving beyond generic benchmarks to address specific production failures.

What happened

The article details a method for closing the loop between observing agent errors and fixing them. It highlights two key components: NeMo Compass, which analyzes stored agent traces to find recurring problems, and NeMo Eval Author, which helps engineers write custom tests to verify fixes. The process is designed to turn raw production evidence into a prioritized backlog of issues.

To illustrate the workflow, NVIDIA used the NemoClaw blueprint, which features a Hermes enterprise assistant connected to various company tools via the Model Context Protocol. In one example, the agent was asked to draft a message but incorrectly sent it without user confirmation. By analyzing 42 recorded traces against a behavior definition file called ETHOS.md, NeMo Compass identified this unauthorized action as a recurring anomaly.

The tutorial also describes a more complex engineering scenario involving Dassault Systèmes’ LEO agent. In this case, the agent repeatedly failed to format arguments correctly for a simulation tool, resulting in empty error messages. Using the same optimization loop, engineers reduced malformed tool calls from 142 to zero and improved overall evaluation scores by 28 percent while reducing tool usage by 35 percent.

How it works

NeMo Compass acts as an automated analyst for agent traces. Instead of requiring developers to manually inspect every log, it processes JSON trace files and compares agent actions against defined behavioral rules. It looks for signs of unusual behavior, inefficient tool use, or divergence from intended workflows. The output is a structured list of findings, each linked to specific trace IDs, allowing engineers to focus on the most impactful issues first.

Once an issue is identified, NeMo Eval Author steps in to create reproducible tests. This tool provides skills for coding agents, such as Codex or Claude Code, to generate evaluations in Harbor or NVIDIA NeMo Gym formats. It uses the original traces, the behavioral guidelines, and the agent’s codebase as context to build verifiers. For example, if an agent sends a message without consent, the verifier checks for that specific tool call and marks the run as a failure if it occurs.

After the evaluation suite is ready, engineers modify the agent harness, such as updating the SOUL.md system prompt or adding middleware validation. The updated agent is then tested against both development tasks and a held-out set of scenarios. This separation ensures that fixes generalize well and do not simply memorize the test cases, preventing overfitting while confirming that regressions are caught early.

Key details

• NeMo Compass analyzes stored traces and evaluation evidence to prioritize improvements and link findings to specific trace IDs.
• NeMo Eval Author generates custom evaluations in Harbor or NeMo Gym formats using coding agent skills and domain requirements.
• The optimization cycle involves identifying an issue, creating a test, updating the harness, and verifying results on held-out tasks.
• In a Dassault Systèmes example, the workflow reduced malformed tool calls from 142 to zero and cut tool calls per run by 35 percent.
• Prerequisites include a Linux machine with Docker, OpenShell 0.1.2, Harbor 0.22.0, and an NVIDIA API key.
• Held-out evaluation tasks must remain hidden from the coding agent during the fix proposal phase to ensure valid generalization testing.

Why it matters

For teams building domain-specific agents, generic benchmarks often fail to capture real-world reliability issues. An agent might score well on standard tests but still choose the wrong tool or miss critical context in production. This workflow shifts the focus from abstract performance metrics to concrete behavioral correctness, ensuring that agents meet specific business requirements rather than just linguistic fluency.

The ability to automate the creation of evaluations from production traces significantly reduces the manual effort required to maintain agent quality. As agents interact with more complex systems, the number of potential failure modes grows. By linking every fix to a reproducible test, engineers can iterate faster with confidence, knowing that changes are validated against real usage patterns and not just synthetic examples.

What you can do

• Install the NemoClaw blueprint to run a complete optimization cycle on the provided example agent.
• Set up a Linux environment with Ubuntu 24.04, 8 vCPUs, and 32 GB of RAM to run the required Docker containers.
• Record your own agent traces in JSONL format and define clear behavioral expectations in an ETHOS.md file.
• Use NeMo Compass to analyze your traces and identify the most frequent or severe deviations from expected behavior.
• Leverage NeMo Eval Author to generate custom verifiers that check for the specific failures identified in your analysis.
• Always reserve a set of held-out tasks to validate that your harness changes improve performance without causing regressions.