AI agents automate robot simulation workflows from prompt to policy

NVIDIA has outlined a new workflow that uses artificial intelligence agents to automate the creation of simulated environments for robot training. Published on October 23, 2026, the technical deep-dive details how tools like Hermes, Nemotron, and Astra collaborate to turn a simple text prompt into a fully functional obstacle course for a robot named Microduck. This process bridges the gap between high-level planning and low-level physics simulation, allowing developers to train locomotion policies entirely in software before deploying them to hardware.

What happened

The core of this development is the use of NVIDIA RTX Spark as a local development hub. In this setup, AI agents act as intermediaries that translate human intent into technical execution steps. The workflow begins when a developer provides a prompt describing the desired task, such as building an obstacle course for Microduck. The agents Hermes and NVIDIA Nemotron 3.5 Bolt analyze this request and generate a structured project plan. This plan includes researching the robot’s capabilities, defining the course layout, and listing the necessary digital assets.

Once the plan is established, the agent Astra takes over to create the 3D simulation assets. Astra enhances reference images, models geometry using FreeCAD, and applies materials in Blender. These assets are then exported as Universal Scene Description (USD) files. The environment is assembled in NVIDIA Isaac Sim, which uses Omniverse libraries for rendering and Newton for physics modeling. Finally, the robot’s behavior is trained using NVIDIA Isaac Lab, which runs thousands of parallel simulations to teach the robot how to walk and navigate obstacles without falling.

How it works

The technology relies on a chain of specialized AI models and simulation engines working in sequence. First, large language models interpret the natural language prompt to break down the problem into manageable tasks. They identify what objects are needed and how they should interact. For asset creation, the system uses image editing and computer-aided design tools programmatically. Astra ensures that the visual fidelity of the generated objects matches the initial concept while maintaining physical accuracy for the simulation engine.

Training occurs through reinforcement learning in a massively parallel environment. NVIDIA Isaac Lab spawns 4,096 instances of the Microduck robot simultaneously on a single GPU. Each instance interacts with the simulated physics engine, Newton, which calculates forces, balance, and collisions in real time. The robot receives observations from its head-camera depth sensors and internal state data. Based on rewards for staying upright and moving forward, the policy updates its neural network weights. This loop repeats millions of times, allowing the robot to learn complex motor skills much faster than in real-world trials.

Key details

• Local Hub: NVIDIA RTX Spark serves as the central workstation for planning and local validation.
• AI Agents: Hermes and Nemotron 3.5 Bolt handle planning, while Astra manages asset creation and environment assembly.
• Simulation Tools: NVIDIA Isaac Sim handles rendering and sensor simulation, while Newton manages physics.
• Training Scale: Policies are trained using 4,096 parallel environments on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU.
• Asset Library: Developers can supplement generated assets with over 2,000 ungated models from the SimReady Central Library.
• Sim-to-Real Prep: The workflow includes Better Actuator Models and domain randomization to prepare policies for physical deployment.

Why it matters

For software engineers and robotics developers, this workflow significantly reduces the friction involved in setting up simulation environments. Traditionally, creating a realistic test bed requires manual 3D modeling, careful physics tuning, and extensive scripting. By automating these steps with AI agents, teams can iterate on robot behaviors much faster. The ability to go from a text description to a trained policy in a unified pipeline means that prototyping new robot tasks becomes accessible to those who may not have deep expertise in computer graphics or physics engine configuration.

Furthermore, the separation of training and validation stages ensures robustness. Training at scale in the cloud or on powerful workstations allows for exhaustive exploration of edge cases, such as uneven terrain or unexpected obstacles. Validating the final policy locally on RTX Spark provides a controlled environment to verify performance before risking expensive hardware. This approach helps bridge the sim-to-real gap, ensuring that behaviors learned in simulation transfer effectively to physical robots like Microduck.

What you can do

• Download the Windows x86 binaries for NVIDIA Isaac Sim to run on RTX Spark via emulation.
• Access the SimReady Central Library to find pre-made USD assets for your own simulation environments.
• Review the Isaac Lab documentation to understand how to define rewards and observations for reinforcement learning.
• Follow the Microduck sim-to-real tutorial on Hugging Face to adapt this workflow for physical robot deployment.
• Experiment with the Microduck example environment in Isaac Lab to practice training locomotion policies.
• Use Better Actuator Models in Isaac Lab to simulate servo response delays and mechanical backlash for more realistic training.