CWE-bench v2 raises the bar for AI security evaluation

Collinear AI has launched CWE-bench v2, a public benchmark designed to rigorously test the cybersecurity capabilities of frontier AI models. Released in November 2026, this update provides developers with a standardized way to measure how effectively agents can identify and patch security weaknesses in real-world open-source repositories.

What happened

The new version opens up a curated set of 100 tasks that anyone can use to evaluate their models. Unlike previous iterations or private benchmarks, this release emphasizes transparency and reproducibility. The setup remains consistent with earlier versions: an AI agent receives access to a checkout of a genuine open-source repository along with a concise instruction. Its objective is to locate a specific security weakness and apply a fix without disrupting existing functionality.

Scoring is fully programmatic, ensuring that results are deterministic and can be reproduced by any developer without incurring additional costs. To maintain high standards, Collinear AI employed large language model judge panels to refine the programmatic checks, focusing on hardness and fairness. The organization also maintains a held-out set of tasks to track progress on challenges that models have not yet encountered. Access to this private set requires contacting the team directly, preventing models from memorizing solutions.

This release covers 91 different Common Weakness Enumerations (CWEs), representing a broad spectrum of vulnerability classes. The goal is exhaustiveness, reflecting the reality that defenders must anticipate every possible attack vector. Each task underwent manual review to ensure realistic instructions, verifiable behavior, and robust verification logic that does not rely on specific code structures. The team explicitly checks that starter code fails, reference fixes pass, and original functionality remains intact.

How it works

Each task in CWE-bench v2 is a self-contained Harbor task, which consists of a Dockerized environment, a natural language instruction, and a deterministic verifier. This structure allows the benchmark to run entirely from the Hugging Face repository without requiring users to clone files manually. The dataset includes a registry.json file that Harbor uses to fetch and execute tasks efficiently.

The verification process is strict. It ensures that the AI agent’s solution actually resolves the security issue while preserving the software’s intended behavior. Crucially, the verifiers are designed to fail if an agent simply reproduces a known upstream patch or memorized solution, forcing the model to demonstrate genuine understanding rather than recall. This approach tests whether the model can recognize unfamiliar patterns and apply logical reasoning to novel security surfaces.

Developers can run the benchmark using simple command-line tools. By accepting the access conditions on the dataset page and configuring their Hugging Face token, users can execute evaluations with a single command. The system supports running multiple attempts per task, providing a mean reward score that offers a more stable measure of performance than single-shot results.

Key details

• The public set contains 100 tasks covering 91 distinct CWEs, ensuring broad coverage of vulnerability types.
• Scoring is deterministic and programmatic, allowing for zero-cost reproduction of results by any user.
• Gemini 4 Argon leads the leaderboard with a 67.8% pass@1 score, followed closely by Claude Opus 5.5 at 67.3%.
• DeepSeek V4.1 Flash offers a cost-effective option at $0.04 per rollout, achieving a 57.0% pass@1 score.
• The benchmark uses Harbor tasks with Dockerized environments to isolate execution and ensure consistency.
• A held-out set remains private to prevent overfitting, with access available via direct contact with Collinear AI.

Why it matters

For software engineers and security teams, this benchmark provides a concrete metric for evaluating AI assistants before deploying them in production environments. Security is a domain where partial correctness is insufficient; a defender must be right every time. CWE-bench v2 highlights that even top-performing models fail nearly one-fifth of tasks on every attempt, indicating that the "long tail" of security vulnerabilities remains a significant challenge.

The transparency of the benchmark allows organizations to assess trade-offs between cost and performance. With data showing a clear Pareto frontier among models like DeepSeek, Claude, and Gemini, teams can make informed decisions about which tools fit their budget and security requirements. This level of detail helps prevent overreliance on marketing claims and grounds expectations in empirical evidence.

Furthermore, the open-source nature of the grader and harness encourages community contribution and continuous improvement. By tagging tasks as reinforcement learning environments, the benchmark also supports the training of defensive agents, not just their evaluation. This dual utility fosters a ecosystem where better evaluation leads to better training data, ultimately raising the baseline for AI-assisted security practices.

What you can do

• Accept the access conditions on the Hugging Face dataset page to gain permission to use the benchmark.
• Install Harbor and configure your Hugging Face token to run evaluations locally without cloning repositories.
• Test your preferred AI model against the public set to establish a baseline for its security reasoning capabilities.
• Compare cost-efficiency by running small-scale tests with models like DeepSeek V4.1 Flash versus premium options.
• Review the failed cases in your evaluations to identify specific CWE categories where your model struggles.
• Contribute feedback or new tasks to the public set to help improve the quality and coverage of future versions.