NVIDIA Cosmos simplifies physical AI simulation and deployment

NVIDIA has introduced Cosmos, a comprehensive platform designed to accelerate the development of physical AI through world foundation models and specialized development tools. Released in early November 2026, this suite addresses critical bottlenecks in sensor adaptation and reliable model deployment for industries ranging from autonomous driving to surgical robotics.

What happened

The launch centers on two primary components: the open-source Cosmos Framework and the optimized Cosmos NIM microservices. The Framework provides modular tools for the entire lifecycle of world model development, including training, reinforcement learning, optimization, and inference. It supports a growing ecosystem of models, including Cosmos 3, NVIDIA Alpamayo, GR00T, Qwen, and pi0.5, allowing developers to train on diverse data types such as text, video, audio, and action sequences.

Simultaneously, NVIDIA released specific simulation recipes for Cosmos 3 Nano. These recipes enable the generation of synchronized camera video and LiDAR observations for autonomous vehicle, robotics, and surgical scenarios. By providing released checkpoints that run immediately, the company aims to lower the barrier for engineers who need to evaluate models against custom sensors and action spaces without building complex physics engines from scratch.

On the deployment side, Cosmos NIM microservices offer GPU-accelerated serving via application APIs. These containers are engineered to reduce memory usage and increase inference speed through techniques like quantization and speculative decoding. This dual approach allows teams to iterate quickly on model architecture using the Framework while relying on NIM for stable, high-performance production serving.

How it works

Cosmos Framework operates by integrating agent skills directly into the development workflow. Developers can use these skills to automate tasks such as post-training, dataset validation, and evaluation. The framework supports various reinforcement learning paths, including offline preference optimization (DPO) and online RL with rule-based rewards (GRPO). It also facilitates model distillation and quantization, enabling engineers to shrink model sizes or reduce generation steps while maintaining quality. Inference can be executed using standard libraries like PyTorch and Hugging Face Diffusers, or optimized backends like vLLM and TRT-LLM.

For simulation, the platform uses World Scenario Maps (WSM) to control scene elements like lanes, objects, and traffic signs. In autonomous vehicle workflows, models like Cosmos3-Nano-Transfer-Auto generate multi-view clips by denoising entire sequences at once, suitable for offline synthetic data generation. Conversely, Cosmos3-Nano-Sim-Auto uses causal, few-step distillation to generate observations autoregressively. This allows for closed-loop evaluation where a policy’s actions update the scenario in real time, creating a dynamic feedback loop between the AI agent and the simulated environment.

Cosmos NIM handles the heavy lifting of production inference. It employs KV-cache-aware routing and token pruning to maximize compute utilization. On NVIDIA H200 GPUs, speculative decoding delivers mean speedups of approximately 2–3x. The system automatically selects configurations based on the target GPU, balancing latency for interactive tasks against throughput for batch processing. Memory requirements are strictly defined, with FP8 quantization and offloading options available to fit larger models onto available hardware.

Key details

• Cosmos Framework supports training and inference for Cosmos 3, Alpamayo, GR00T, Qwen, and pi0.5 models.
• Cosmos3-Nano-Transfer-Auto generates up to 400 frames at 480p or 720p resolution at 30 fps for offline synthetic data.
• Speculative decoding in Cosmos NIM achieves roughly 2–3x speedups on H200 GPUs compared to standard inference.
• Generating with Cosmos3-Nano requires 58 GiB VRAM in BF16, reducible to 44 GiB in FP8, or 31 GiB with offloading.
• Surgical simulation models like Cosmos3-H-Dreams enable real-time, closed-loop testing of policies using live robot kinematics.
• NIM containers provide quarterly updates for features, optimizations, and security, with automatic configuration selection for supported GPUs.

Why it matters

For software engineers building physical AI, the gap between training a model and deploying it in a realistic environment has traditionally been wide and fraught with compatibility issues. Cosmos Framework bridges this by offering standardized recipes for sensor adaptation. Instead of writing custom code to align LiDAR data with camera feeds, developers can use pre-built agents to handle synchronization and validation. This reduces the engineering overhead required to create high-fidelity synthetic datasets, which are essential for training robust perception systems.

The introduction of optimized NIM microservices also shifts the focus from infrastructure management to application logic. By containerizing the inference engine with built-in optimizations like quantization and caching, NVIDIA removes the need for teams to manually tune performance parameters for different GPU architectures. This is particularly valuable for startups and enterprise teams alike, as it ensures that models run efficiently on available hardware without extensive trial-and-error profiling.

Furthermore, the ability to simulate complex physical interactions, such as tissue deformation in surgery or multi-agent traffic scenarios, without explicit physics modeling represents a significant leap. Generative simulation learns visual dynamics directly from data, allowing for more realistic and varied test environments. This capability enables safer and more thorough evaluation of AI policies before they interact with physical hardware, reducing risk and accelerating iteration cycles.

What you can do

• Download the Cosmos Framework from the repository to access agent skills for training, post-training, and troubleshooting.
• Run the Cosmos3-Nano-Transfer-Auto recipe using the provided sample checkpoints to generate synchronized video and LiDAR data.
• Evaluate your current GPU memory against the published VRAM requirements table to select the appropriate quantization strategy.
• Launch a Cosmos NIM container using Docker and your NGC API key to test accelerated inference on your local machine.
• Use the Sim-Auto Interactive Drive guide to set up closed-loop simulations for autonomous vehicle policy evaluation.
• Explore the reinforcement learning pipelines, such as DPO and GRPO, to fine-tune models on your specific proprietary datasets.