Rescale cuts test triage time by 20% using AI agents and mob programming

Rescale engineers have reduced the time spent blocking production deployments by 20% after deploying an automated AI triage agent. Built in just five days during a workshop with AWS architects, the system classifies nightly end-to-end test failures and routes them to the correct teams. The project demonstrates how structured human-AI collaboration can accelerate complex infrastructure tooling without sacrificing architectural rigor.

What happened

Rescale, a digital engineering platform for industries like aerospace and life sciences, faced a growing operational bottleneck. Their nightly test suites, which gate weekly production releases, frequently produced hundreds of failures. On-call engineers spent hours manually pulling logs, analyzing root causes, and routing issues to the right teams. This repetitive work consumed significant engineering capacity and delayed deployments when unresolved failures required cross-team coordination.

To solve this, Rescale partnered with AWS to build an intelligent agent using the AI-Driven Development Lifecycle (AI-DLC) framework. The team moved from problem definition to a staging deployment in five days, with production rollout following within weeks. The agent now handles the initial classification of test failures, allowing engineers to focus on validation and complex debugging rather than manual triage.

The success of this pilot has led Rescale to expand the use of AI-DLC across its engineering organization. Teams are now applying the same structured approach to other internal tools, including HPC simulation catalog management and container vulnerability tracking. The methodology has shifted from a one-off experiment to a standard practice for building AI-native development tools.

How it works

The triage agent operates as a pod within Rescale’s existing Argo Workflows CI/CD pipeline on Amazon Elastic Kubernetes Service (Amazon EKS). When a nightly Playwright test suite fails, the pipeline triggers the agent. It invokes Claude Opus 4.8 on Amazon Bedrock to classify each failure into categories such as infrastructure issues, code bugs, flaky tests, or environment problems. The agent retrieves historical failure patterns from a knowledge base and queries pod logs via the Model Context Protocol (MCP) to gather runtime evidence.

Instead of taking autonomous action, the agent generates a structured remediation report in JSON and HTML. This report includes specific file references, function names, and recommended fixes for engineer review. By integrating directly into the existing reporting pipeline, the agent provides actionable insights without disrupting current workflows. The use of MCP allows the agent to switch data sources, such as moving from third-party logging to Amazon CloudWatch, without changing the core interaction pattern.

Key details

• Development timeline: The team went from vision to staging deployment in five days during an in-person workshop.
• Performance improvement: Classification agreement with human triage rose from 44% at launch to 68% currently.
• Operational impact: Time blocking production deployments due to test analysis decreased by 20%.
• Tech stack: The agent uses the Strands SDK, runs on Amazon EKS, and leverages Amazon Bedrock for model inference.
• Methodology: The project used AI-DLC, which emphasizes "mob elaboration" for requirements and "mob construction" for implementation.
• Architecture pivot: The team moved the agent from Amazon Bedrock AgentCore to Amazon EKS to align with their existing CI/CD infrastructure.

Why it matters

For engineering leads and platform teams, this case study highlights the value of structuring AI development around human decision-making. Many organizations struggle with AI projects that produce code but lack architectural coherence. By using mob sessions where AI generates plans and humans validate them in real time, Rescale avoided the common pitfall of asynchronous handoffs. This approach ensured that critical boundaries, such as what to include in Phase 1 versus Phase 2, were defined explicitly before coding began.

The integration of agents into CI/CD pipelines also offers a practical blueprint for reducing toil. As test suites grow in complexity, manual triage becomes unsustainable. Automating the first pass of failure analysis allows senior engineers to focus on high-value tasks like architecture and product judgment. The use of standardized protocols like MCP further ensures that these agents remain flexible and maintainable as underlying infrastructure changes.

What you can do

• Identify high-friction workflows: Look for repetitive tasks in your CI/CD or on-call rotations that involve gathering data from multiple sources.
• Adopt structured AI methodologies: Consider frameworks like AI-DLC that separate requirement elaboration from code generation to maintain architectural control.
• Start with small scopes: Define clear Phase 1 boundaries and defer complex features like automated pull request generation until the core logic is stable.
• Use evaluation harnesses: Build PyTest-based frameworks with live and stubbed modes to measure agent accuracy against a golden dataset.
• Leverage existing infrastructure: Host agents within your current Kubernetes clusters to reduce operational overhead and simplify integration.
• Iterate on prompts: Continuously refine prompt engineering and training data to improve classification agreement over time.