NVIDIA releases open Cosmos 3 models for interactive world simulation

NVIDIA has expanded its Cosmos world model development platform with four new open-weight checkpoints under the Cosmos 3 model family. Released in early November 2026, these models are designed to generate interactive world simulations and multi-sensor, multi-view driving scenes for researchers and engineers.

The release targets developers working on autonomous driving and robotics who need high-fidelity, controllable synthetic environments. By making the weights open under the OpenMDW1.1 license, NVIDIA aims to provide a shared foundation for adapting models to specific sensor rigs and domains.

What happened

The update introduces four specific post-trained examples of the Cosmos 3 Nano architecture, each containing 16 billion trainable parameters. These checkpoints are not just static video generators but are built for interactive simulation where observations and actions influence one another in real time.

The first two models focus on autonomous driving. Cosmos3-Nano-Transfer-Auto generates multi-view driving data with varied weather, lighting, and time of day conditions. It supports synchronized camera and LiDAR observations, allowing developers to study how driving policies respond to changes in road layouts and traffic signals. The second driving model, Cosmos3-Nano-Sim-Auto, is a causally distilled variant optimized for fast streaming. It generates successive chunks of synchronized video and LiDAR data using generated history and updated controls, enabling long, interactive rollouts.

The other two models address general robotics and simulator integration. Cosmos3-Nano-Sim-Transfer converts spatial controls from a physics simulator, such as depth maps or segmentation masks, along with text prompts, into photorealistic video. This allows the visual appearance to be generated by AI while the physical state remains grounded in a traditional simulator. Finally, Cosmos3-Nano-Sim-Bimanual predicts video observations directly from robot actions, supporting interactive two-arm simulation for specific robotic embodiments.

How it works

These models rely on a Mixture-of-Transformers (MoT) backbone that connects separate text-processing and visual-generation networks through attention mechanisms. The system operates on compressed video representations called latents, where each latent time step corresponds to four RGB frames. To generate video, the model uses temporal-causal attention, which allows it to read available history and current controls while blocking access to future target observations. A key/value cache retains attention information to speed up subsequent predictions.

The training process follows a four-stage recipe designed to balance quality with inference speed. Stage 0 involves bidirectional adaptation to learn domain-specific control-to-video relationships from recorded clips. Stage 1 uses teacher forcing to predict chunks from earlier observations. Stage 2 applies consistency distillation, where a frozen teacher model trains a student model to generate each chunk in just four denoising steps. The final stage, self-forcing, trains the model on its own generated history. This step is critical because it exposes the model to its own errors, preventing minor mispredictions from compounding into major artifacts during long interactive sequences.

Key details

• All four models are based on the Cosmos 3 Nano architecture with 16 billion trainable parameters.
• The models are released under the OpenMDW1.1 license, permitting both commercial and non-commercial use.
• Cosmos3-Nano-Transfer-Auto supports joint RGB and LiDAR generation across up to 11 camera views, including fisheyes.
• Cosmos3-Nano-Sim-Auto uses causal distillation to generate video chunks in few-step denoising for fast streaming.
• Cosmos3-Nano-Sim-Bimanual supports interactive two-arm simulation for YAM and AgiBot embodiments using a 64-value action schema.
• Inference requires Linux, an NVIDIA GPU, Docker, and the NVIDIA Container Toolkit, with testing performed on GB200 hardware.

Why it matters

For engineers building autonomous vehicles or robots, collecting real-world training data is expensive, dangerous, and limited in scope. These open-weight models allow teams to generate vast amounts of synthetic data with precise control over environmental variables. By synchronizing visual data with LiDAR and physics controls, developers can test edge cases, such as rare weather conditions or complex traffic scenarios, without leaving the lab. This capability is essential for validating safety-critical systems before deployment.

The open nature of the release also lowers the barrier to entry for world model research. Instead of building generative models from scratch, teams can adapt these base weights to their specific sensor configurations and operational domains. The provided post-training recipes offer a clear path from offline data generation to real-time interactive simulation. This standardization helps the community compare methods and contribute improvements, accelerating the development of more responsive and accurate simulated worlds.

What you can do

• Download the four new checkpoints from Hugging Face to experiment with synthetic data generation for your specific domain.
• Use Cosmos3-Nano-Transfer-Auto to adapt the model to your own camera layout and LiDAR configuration using your proprietary data.
• Implement the causal training and distillation pipeline to convert adapted models into fast-streaming variants for interactive simulation.
• Integrate Cosmos3-Nano-Sim-Transfer with your existing physics simulator to render photorealistic visuals from depth and segmentation controls.
• Test Cosmos3-Nano-Sim-Bimanual for robotic manipulation tasks by providing normalized action sequences according to the published schema.
• Evaluate model quality on held-out episodes to verify execution and convergence before committing to long training runs.