Evidence at a glance
Four Checkpoints, Four Roles
NVIDIA released four open-weight checkpoints on Hugging Face, each post-trained from Cosmos 3 Nano with 16B trainable parameters and licensed under OpenMDW1.1, which permits commercial and non-commercial use. Each targets a different simulation task: Transfer-Auto generates offline multi-sensor driving data; Sim-Auto streams observations from driving controls; Sim-Transfer turns simulator controls into frames; and Sim-Bimanual predicts subsequent video from robot actions or camera motion.
Transfer-Auto and Sim-Auto form a progression: the former generates data and can be adapted to a new sensor configuration; the latter builds on Transfer-Auto, using causal training and few-step distillation for streaming generation. Transfer-Auto supports up to 11 synchronized camera views, including fisheyes, and can jointly generate LiDAR aligned with the cameras. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance. Sim-Transfer and Sim-Bimanual target simulator-frame generation and robot manipulation prediction, respectively. The four checkpoints share the Cosmos 3 Nano base but are post-trained for different inputs and observation tasks.
From Full Clips to Streaming Generation
NVIDIA describes a four-stage post-training recipe. First, bidirectional adaptation uses complete recorded clips to teach the relationship between controls and video. Second, teacher forcing predicts later chunks from earlier observations and available controls, while masking the clean answer and future targets. Third, consistency distillation trains a student to generate each chunk in four denoising steps. Fourth, self-forcing trains on the student’s own generated history. This last stage addresses error accumulation: a small prediction error can become context for later generations. A frozen teacher and an auxiliary score model provide the distribution-matching signal.
Sim-Auto is the streaming version of Transfer-Auto, produced through causal training and distillation. It generates a small chunk of synchronized camera video and LiDAR at a time, then uses the generated history and updated driving controls for the next prediction. Unlike offline generation, closed-loop interaction can use only available history and current controls to generate the next observation. A simulation runtime and WSM renderer are still needed to update controls from driving inputs; Sim-Auto does not replace them. NVIDIA’s adaptation path is to first tune Transfer-Auto on a team’s own data, then distill it into Sim-Auto.
It Generates Observations, Not Physical State
Sim-Transfer uses spatial controls from a simulator and a text prompt to generate RGB frames. The simulator computes object motion, contacts and physical state; the model generates visual observations that a policy can use. Closed-loop Sim-Auto also needs an external simulation runtime and WSM renderer to update controls. Sim-Bimanual, by contrast, predicts the next observation from robot or camera actions. Their inputs and tasks differ, and none should be mistaken for a standalone physics simulator.
That distinction defines Cosmos 3 Nano’s place in a simulation pipeline: it generates visual results but does not replace the simulator that supplies physical state. NVIDIA says that measuring whether the models faithfully support policy learning and evaluation, and understanding their limits, remain work ahead. Whether generated frames match simulator state and how they behave over continuous generation therefore need to be assessed for each task; the release does not offer a general conclusion on those questions.
Published Throughput and Control Requirements
The Cosmos Lab page reports operating figures for three models: Sim-Bimanual reaches 48.1 FPS with 166 ms latency on one GB300; Sim-Transfer reaches 15.85 FPS with 240 ms latency on one GB200; and Sim-Auto runs at 16.7 FPS for 7 480p RGB views on 16 GB200s. These are the configurations reported on the page, not minimum hardware requirements for every deployment.
Sim-Bimanual also has a defined control-data contract. The mixed-action example contains 59 control values, padded to 64 during training; its schema specifies coordinate frames, rotations, gripper calibration, validity masks and control domains. The model supports YAM and AgiBot bimanual embodiments, and inputs should follow the corresponding conventions. Timing must also align: T transitions require T+1 observations. Released inference actions are already normalized, while training data must first be converted into the model’s action representation.
Open Weights Offer a Starting Point; Evaluation Is Still Needed
OpenMDW1.1 permits commercial and non-commercial use, subject to its terms. NVIDIA also released post-training recipes for adapting models with a team’s own data, starting from Cosmos 3 Nano or compatible base weights. The recipes reflect each checkpoint’s role: Transfer-Auto targets driving data and sensor configurations; Sim-Auto adds causal training and distillation to an adapted Transfer-Auto; and Sim-Transfer and Sim-Bimanual are post-trained for their respective environments or robot embodiments.
The checkpoints and recipes are starting points for adaptation; teams still need to prepare data, scene controls and a simulation environment. NVIDIA recommends checking the environment, validating a sample, and testing training and restart before a longer run. Held-out episodes can be used to assess generation quality. A successful short run verifies that the process executes, not that the model has converged. Teams still need to test whether generated observations support their target policy’s learning and evaluation, and identify the settings in which the model falls short.