AI excels at coding but struggles with judgment-heavy research tasks
METR’s updated evaluation shows AI models improve rapidly on verifiable tasks like coding, yet fall short on open-ended research and judgment-based challenges.
The AI evaluation nonprofit METR released an updated report on November 20, 2026, revealing that while frontier models continue to improve exponentially on verifiable tasks, they still lag significantly in areas requiring human-like judgment. This update addresses widespread misinterpretations of their previous time horizon plots by incorporating more complex, open-ended tasks that better reflect the demands of automated research and development.
What happened
METR’s latest analysis updates its famous time horizon plot, which tracks AI progress by measuring the length of tasks models can complete compared to humans. The original plot showed that every four months, the capability of frontier models doubled in terms of task duration. However, that earlier version focused heavily on verifiable domains like coding and math, where success is binary and easy to check automatically. The new report expands this scope to include tasks that are harder to define and evaluate, such as designing new AI evaluations or scoring philosophical arguments.
The findings highlight a growing divergence in capabilities. While advanced models now outperform almost every human in software engineering, they failed to produce any useful AI evaluation designs when asked to create them for METR’s own research. Similarly, when tasked with scoring philosophical arguments, the models did not match human performance. These results suggest that the reinforcement learning techniques driving success in coding do not yet translate effectively to open-ended research tasks where success criteria are fuzzy or subjective.
This distinction is critical given recent concerns about recursive self-improvement in AI. A separate, heavily redacted METR report had previously suggested that AI might be accelerating progress at Anthropic by a factor of two, sparking fears of autonomous AI design. By testing models on judgment-heavy tasks, METR aims to provide a clearer picture of whether AI systems can truly automate the research process or if they remain limited to executing well-defined instructions.
How it works
METR evaluates AI capabilities by assigning tasks to both models and human testers. In the original framework, difficulty was determined by the actual time humans took to complete specific tasks. Models were then rated based on the longest tasks they could successfully finish. For verifiable tasks like coding, this method is robust because automated tests can confirm whether the code works. Reinforcement learning leverages this clarity by strengthening neural pathways that lead to correct outputs and weakening those that cause errors.
For the updated report, METR introduced tasks that are difficult to verify automatically. Instead of paying humans to complete these potentially hundreds-of-hours-long tasks, researchers estimated completion times due to cost and expertise constraints. Some tasks, like designing evaluations, were scored by human employees at METR. Others, such as the philosophical argument scoring, relied on agreement with a single rater. This shift from empirical timing to expert estimation and subjective scoring introduces new variables but allows for assessment of higher-level cognitive skills.
Key details
- Frontier AI models double their task-completion capability every four months for verifiable tasks.
- No AI-designed evaluation was considered useful by METR employees, despite models outperforming humans in coding.
- Models fell short of human performance in scoring philosophical arguments, a proxy for judgment.
- Newer models like Opus 5.5 achieved "borderline above the bar" performance on evaluation design, up from "clearly below the bar" a year ago.
- Task difficulty for open-ended jobs is now mostly estimated rather than measured by actual human completion time.
- Experts agree that current evaluation methods for judgment-heavy tasks are still immature and hard to quantify.
Why it matters
For software developers and technical leads, this report clarifies the current boundaries of AI assistance. It confirms that while AI can handle complex coding and mathematical problems with high reliability, it cannot yet be trusted to make strategic decisions or define new research directions. This distinction is vital for teams considering automating parts of their R&D pipeline. Relying on AI for tasks that require "research taste" or subjective judgment may lead to poor outcomes, as the models lack the ability to discern quality in undefined spaces.
Furthermore, the difficulty in evaluating these judgment-heavy tasks means that progress metrics may become less reliable. As AI moves into domains where success is not binary, the clean exponential trends seen in coding benchmarks may not apply. Engineers and founders need to be cautious about extrapolating current growth rates to predict when AI will fully automate research. The gap between verifiable execution and open-ended creation remains a significant bottleneck.
What you can do
- Use AI for verifiable tasks like code generation, debugging, and unit test creation where success is easily measured.
- Avoid delegating strategic planning, hypothesis generation, or qualitative assessment to current AI models.
- Implement human-in-the-loop workflows for any task involving subjective judgment or open-ended problem solving.
- Monitor emerging evaluation frameworks that focus on judgment and creativity, rather than relying solely on coding benchmarks.
- Be skeptical of claims that AI can fully automate R&D until robust metrics for judgment-heavy tasks are established.
- Invest in training teams to effectively critique and refine AI outputs in ambiguous domains.



