Surge AI releases DAYJOB: Tax benchmark for professional agent evaluation

Surge AI has released DAYJOB: Tax, a new benchmark designed to evaluate how well artificial intelligence agents handle professional tax work. Published in late October 2026, this addition to the DAYJOB family tests agents on real-world scenarios involving tax preparation, compliance, and advisory services where instructions are often incomplete or ambiguous.

The benchmark reveals that even leading frontier models struggle with the nuance required in professional domains. Claude Sonnet 5.5 achieved the highest score with a 10.8% pass rate, indicating that reliable autonomous tax advice remains a significant challenge for current technology. This evaluation moves beyond simple question-answering to assess whether agents can identify hidden risks and exercise professional judgment.

What happened

DAYJOB: Tax expands on previous benchmarks focused on healthcare and finance by targeting the specific complexities of tax practice. The suite includes 50 distinct environments covering eight areas of tax work, such as individual and family tax, small-business tax, partnerships, and international tax. In each environment, an AI agent receives a client request along with a set of supporting records. The agent must then produce deliverables like tax returns, technical reviews, or client advice letters.

Unlike standard benchmarks that provide a clearly defined problem, DAYJOB: Tax presents messy, realistic situations. Agents must determine what information is missing, resolve conflicting evidence in documents, and decide which issues warrant further investigation. For example, one task involves a handwritten receipt for merchandise that the agent must reconcile with spending records to determine inventory costs. Another scenario requires analyzing field dispatch logs to uncover multistate tax obligations that the client was unaware of.

The evaluation criteria are strict. An attempt is considered a pass only if it meets at least 95% of the task’s rubric, which checks for accurate calculations, proper use of supporting evidence, identification of consequential issues, and completion of the required deliverable. This high bar ensures that passing scores reflect genuine professional competence rather than partial correctness.

How it works

The benchmark relies on a large corpus of realistic data to simulate professional workflows. Each of the 50 environments contains a workspace of files, with a median of 30 documents per assignment and some containing up to 123 files. These workspaces include 1,487 PDFs, 117 Excel workbooks, and various other formats like emails, images, and JSON records. The materials mimic what a tax professional would see, including filed returns, IRS notices, general ledgers, and even photographs of physical receipts.

Tax experts with backgrounds from firms like PwC, Deloitte, and Baker Tilly created and reviewed these tasks. They developed the scenarios based on real-world professional workflows, ensuring that the expected outcomes and grading criteria reflect actual industry standards. Each task underwent a three-layer expert review process to verify correctness, completeness, and solvability. This human-in-the-loop design ensures that the benchmark measures capabilities that matter in practice, such as recognizing when a client’s assumption about their marital status might be legally incorrect.

Key details

• Top performance: Claude Sonnet 5.5 leads the leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%.
• Scope: The benchmark covers 50 environments across eight tax practice areas, including estates, gifts, trusts, and corporate transactions.
• Data volume: Workspaces contain 1,789 total files, with a median of 30 files per assignment, including PDFs, Excel sheets, and images.
• Strict grading: A pass requires satisfying 95% of the rubric criteria, assessing calculations, evidence usage, and identification of consequential issues.
• Expert validation: Tasks were created and reviewed by tax professionals with experience at major firms like PwC, Deloitte, and Exxon.
• Realistic complexity: Scenarios include unstructured data like handwritten receipts and conflicting evidence such as multi-state work logs.

Why it matters

For engineers building AI agents, DAYJOB: Tax highlights the gap between answering a specific question and managing a professional responsibility. Many existing benchmarks test whether a model can calculate a tax return or answer a legal query. However, in professional services, the primary value often lies in identifying what the client forgot to ask. As seen in the case of Anna’s condo sale, an agent might correctly answer the immediate question while missing a critical deadline that could cost the client hundreds of thousands of dollars. This distinction is vital for developers aiming to deploy agents in high-stakes environments.

The low pass rates also suggest that current models are not yet ready for fully autonomous deployment in tax advisory roles. While they can perform individual steps like retrieval or calculation, they frequently fail to integrate these skills into a coherent professional judgment. This implies that human oversight remains essential. Developers should focus on building systems that assist professionals by highlighting potential issues and organizing evidence, rather than replacing the decision-making process entirely.

What you can do

• Review the DAYJOB: Tax leaderboard to understand the current limitations of frontier models in professional domains.
• Design agent workflows that explicitly prompt for missing information or conflicting evidence before generating final advice.
• Implement strict verification steps for any financial or legal calculations produced by AI agents.
• Use hybrid approaches where AI handles document retrieval and summarization, but humans make final judgments on consequential issues.
• Test your own agents against similar messy, unstructured datasets to identify failure modes in instruction following.
• Stay updated on benchmark methodologies to ensure your evaluation metrics align with real-world professional standards.