Dayjob: Tax – evaluating AI agents on 50 professional tax assignments

Anna was contemplating the optimal time to sell her condo. She had vacated the property in August 2024 and needed to determine whether closing in October 2026 or January 2027 would be more advantageous for taxes. An AI agent analyzed both scenarios and accurately determined that either date would meet the residence requirement for a tax exclusion worth up to $250,000 in gains.

Under standard rules, homeowners must reside in a property for at least two of the five years prior to selling it to qualify for the exclusion. Since Anna moved out in August 2024, she would no longer satisfy this requirement by late August 2027. If her sale was postponed beyond August 2027, she risked losing eligibility. Across five trials, Opus successfully identified the deadline in every attempt, while Astra failed to do so.

The agent correctly evaluated the proposed dates, but its advice failed to inform Anna of a critical deadline that could impact a significant financial decision. A tax advisor should exercise professional judgment even when the client is unaware of the issue. DAYJOB: Tax, the latest addition to Surge AI's DAYJOB benchmark family, assesses AI agents on professional tax assignments that demand specialized knowledge, complex materials, and decisions not fully outlined in the initial instructions.

DAYJOB: Tax extends the previous benchmark releases on Healthcare and Finance into tax preparation, compliance, and advisory services. The benchmark consists of 50 environments across eight tax practice areas. Agents are presented with client requests and supporting records, then generate tax returns, workpapers, technical reviews, or client advice.

These assignments assess professional judgment alongside tax knowledge, calculations, retrieval, and adherence to instructions. Claude Sonnet 5.5 achieved a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2% on the DAYJOB: Tax leaderboard. Pass@1 with a 95% rubric threshold was used to evaluate agent performance.

An attempt is considered successful when it meets at least 95% of the task's criteria, including calculations, supporting evidence, consequential issues, and the final deliverable. DAYJOB: Tax covers eight areas, each requiring distinct rules, records, and expertise. Assignments range from filing complete packages with supporting workpapers to investigating transactions, reviewing previous filings, or advising clients on future decisions.

The benchmark includes 1,789 workspace files, with an average of 30 per assignment and a maximum of 123 files. These files encompass PDFs, Excel workbooks, Word documents, emails, images, CSVs, JSON records, and text files. The workspaces contain various tax-related documents, such as filed and draft returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts, correspondence, and unstructured evidence like photographs of handwritten receipts.

Assignments also involve complex situations, such as an industrial silo-services company operating in multiple states, requiring agents to analyze location data for multistate tax obligations. Other assignments include cross-border households with diverse employment and property records, which must be reconciled with cryptocurrency records and historical purchase details.

DAYJOB: Tax tasks are based on real-world professional workflows, developed by tax experts who create realistic assignments, define supporting records, expected outcomes, and grading criteria. Each task undergoes a three-layer expert review for accuracy, completeness, and solvability. Tax experts have diverse backgrounds in corporate taxation, partnerships, individual returns, trusts, and multistate compliance.

Opus 5.5 demonstrated that AI agents can make informed decisions based on available information, even when the client is uncertain. In contrast, Astra recognized the need to verify marital status in four out of five attempts but struggled to provide a definitive answer.