Simulating customer service agents at scale before production deployment

Researchers from Nubank and academic collaborators have published a detailed account of how they test customer experience AI agents before releasing them to the public. The paper, released on November 6, 2026, describes a simulation system called Snowglobe that allows engineers to evaluate agent performance against synthetic customers rather than real users. This approach addresses the high risks associated with deploying large language models in regulated financial environments.

What happened

Customer experience agents in banking must navigate complex operational policies, detect user intent accurately, and use internal software tools reliably. Traditional testing methods often fall short because manual end-to-end tests cannot cover every possible conversation path, while live experiments expose real customers to potential failures. These failures can damage trust and violate regulatory standards. To solve this, the team developed a hypothesis-driven simulation workflow designed to screen candidate agents before they reach production.

The researchers applied this method to two specific agents at Nubank: the Card Delivery agent and its successor, Card Management. Card Management is the highest-volume chat-support agent in Brazil for the bank. The team used the Snowglobe simulator to run thousands of conversations where synthetic customers interacted with the AI. These synthetic users reacted to agent responses, and the system simulated tool outputs so the agents could complete multi-step workflows without touching live backend systems.

The results showed a strong correlation between simulated scores and actual production performance. Across four deployed versions of the agent, the binary evaluator scores from the simulation matched the outcomes seen in live environments. In one live A/B test, improvements guided by simulation increased the transactional net promoter score by 36.69 points. In another test involving over 16,000 simulated conversations to screen open-weight model configurations, the selected model increased the self-service rate by 8.82 percentage points to a record high, with no negative impact on customer satisfaction.

How it works

The core of this approach is the Snowglobe simulator, which creates a safe sandbox for AI agents. Instead of connecting to real databases or payment gateways, the agent interacts with simulated tools that mimic the behavior of production backends. This allows the agent to execute complex, multi-step tasks, such as checking a card status or updating delivery details, without any risk to real customer data or financial records.

Synthetic customers drive the interaction. These are not simple script readers but dynamic entities that react to the agent’s responses. If an agent provides a confusing answer, the synthetic customer might ask for clarification or express frustration, just as a human would. This feedback loop enables the evaluation of the agent’s ability to handle ambiguity and recover from errors. By running these interactions at scale, engineers can test a wide variety of prompts, reasoning settings, and model configurations quickly.

Key details

• The simulation workflow was tested on Nubank’s Card Delivery and Card Management agents.
• Snowglobe simulator enables multi-step agentic workflows without invoking production backends.
• Simulated binary evaluator scores showed high correlation with production version-level scores across four deployments.
• Simulation-guided iteration led to a 36.69-point increase in transactional net promoter score in a live A/B test.
• Over 16,000 simulated conversations were used to screen open-weight configurations.
• The selected model from simulation screening increased self-service rate by 8.82 percentage points in production.

Why it matters

For engineering teams building AI products, especially in regulated industries, the cost of failure is high. Live A/B testing is often the only way to know if a new model works better, but it carries reputational and compliance risks. This paper demonstrates that high-fidelity simulation can replace much of that risk. It allows teams to iterate rapidly on prompts and model choices without exposing users to broken or hallucinated responses.

The ability to screen open-weight models is particularly significant. Many organizations hesitate to try new, smaller, or open-source models because they fear performance drops. By using simulation to validate these models against thousands of scenarios, teams can confidently adopt more efficient or cost-effective architectures. The correlation between simulation and production metrics suggests that this method is not just a theoretical exercise but a reliable predictor of real-world success.

What you can do

• Implement a simulation layer that mocks your backend tools to allow safe agent testing.
• Develop synthetic user personas that represent diverse customer intents and frustration levels.
• Establish baseline metrics in simulation and verify their correlation with historical production data.
• Use simulation to screen multiple model configurations and prompt variants before live deployment.
• Monitor key indicators like self-service rate and customer satisfaction scores to validate simulation accuracy.
• Iterate on agent logic in the sandbox environment to resolve edge cases before they reach users.