Building with AI

Post-training Nemotron 3.5 Super VL for scientific accuracy with RL

NVIDIA released a recipe to customize Nemotron 3.5 Super VL using reinforcement learning and distillation, boosting drug-metabolism prediction accuracy by 29%.

Illustration of scientific AI modeling with molecular structures and code
Illustration generated for this article

NVIDIA has published a technical guide detailing how to customize its open multimodal model, Nemotron 3.5 Super VL, for specialized scientific workflows. Released on November 16, 2026, the guide demonstrates how researchers can use reinforcement learning and knowledge distillation to significantly improve model accuracy in domains like computational chemistry.

The approach targets agentic AI systems that plan and execute complex scientific tasks, addressing the critical need for higher precision and lower latency in research environments where errors can compound quickly across autonomous iterations.

What happened

The NVIDIA Technical Blog outlined a practical recipe for post-training Nemotron 3.5 Super VL, an open and customizable multimodal model designed for scientific benchmarks. The guide focuses on adapting the model for specific research domains by building custom reinforcement learning environments from experimental data. This process leverages NVIDIA NeMo Gym for environment infrastructure and NVIDIA NeMo RL for the post-training framework.

To validate the method, the team used drug-metabolism prediction as a case study. They utilized records from MetxBioDB, a database of experimentally confirmed biotransformation reactions. The goal was to predict the resulting metabolite given a parent compound and its transforming enzyme. The results showed that supervised fine-tuning improved accuracy by 12%, while subsequent reinforcement learning post-training lifted it by a further 17% at a 16,384-token output budget.

The guide also introduces the Nemotron RL Litmus-Bench V2.0, an open-source dataset and benchmark designed to train and evaluate large language models and agentic AI on molecular properties. This benchmark supports the development of models that can handle longer, more complex, and visual scientific workflows with greater reliability.

How it works

The core mechanism involves creating a reinforcement learning with verifiable rewards (RLVR) environment. Researchers start with a CSV file of experimental records, such as Simplified Molecular Input Line Entry System (SMILES) strings, which describe molecular structures. These input-output pairs are reformatted into prompt-response formats where the experimental ground truth serves as the verifier. This setup eliminates the need for human preference labeling, as the reward signal is derived directly from the data.

In the drug-metabolism example, the model receives the parent molecule and enzyme, then predicts the metabolite. A verifier compares the predicted molecular fingerprint with the reference using Tanimoto similarity, a metric measuring the overlap between two sets. The score ranges from 0 to 1, with higher values indicating greater structural similarity. This dense reward signal guides the model during training via group relative policy optimization (GRPO) with low rank adapters (LoRA).

An optional distillation step can precede reinforcement learning to accelerate convergence. A stronger teacher model, such as Claude Sonnet 5.5, generates high-quality reasoning traces for the same tasks. These traces are filtered based on their accuracy, keeping only those where the teacher’s logic leads to a correct prediction. This "warm-start" provides the student model with a better starting point, making the subsequent RL training more efficient and likely to find a strong policy.

Key details

  • Nemotron 3.5 Super VL is an open, customizable multimodal model optimized for scientific coding and benchmarks.
  • Supervised fine-tuning improved accuracy by 12%, with reinforcement learning adding a further 17% gain in drug-metabolism prediction.
  • The Nemotron RL Litmus-Bench V2.0 dataset evaluates LLMs and agentic AI on molecular properties.
  • Distillation uses Claude Sonnet 5.5 to generate reasoning traces, filtering 1,123 high-quality examples from 1,934 prompts.
  • The RL environment uses Tanimoto similarity to score predictions against ground truth without human labeling.
  • Training utilizes GRPO with LoRA and is supported by NeMo Gym for environment creation and NeMo RL for post-training.

Why it matters

For engineers building scientific AI agents, this guide offers a concrete path to reduce error propagation in autonomous workflows. As agents take on longer and more complex tasks, small inaccuracies in early steps can lead to significant failures later. By post-training models on domain-specific data with verifiable rewards, developers can ensure higher fidelity in critical predictions, such as molecular structures or reaction outcomes.

The ability to customize open models like Nemotron 3.5 Super VL also addresses cost and latency concerns. Frontier-level accuracy is often associated with massive, closed models that are expensive to run. This recipe allows teams to achieve comparable performance on specialized tasks with a customizable model, potentially reducing inference costs and improving time-to-solution for scientific coding tasks.

Furthermore, the inclusion of distillation from stronger teacher models provides a strategy for overcoming cold-start problems in reinforcement learning. When base models lack sufficient capability or RL convergence is slow, warm-starting with high-quality reasoning traces can raise the performance floor. This makes the training process more robust and accessible for domains where data is scarce or expensive to generate.

What you can do

  • Explore the NeMo Gym repository to build a custom RL environment from your own experimental data.
  • Use the NeMo RL repository to apply reinforcement learning post-training to your specific scientific models.
  • Implement the provided recipe to create a verification layer using tools like cheminformatics libraries or molecular simulators.
  • Consider using distillation from a stronger teacher model to generate high-quality reasoning traces before starting RL training.
  • Evaluate your customized model using the Nemotron RL Litmus-Bench V2.0 to benchmark performance on molecular properties.
  • Adapt the CSV-to-JSONL conversion scripts to format your domain-specific data for use in the NeMo Gym environment.
  • #nvidia
  • #reinforcement learning
  • #scientific ai
  • #nemotron
  • #post-training

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories