Fan-Out: Concurrent Tool Calls Under Shared-State Hazards

The Kaggle Benchmarking Challenge introduces the Fan-Out metric, evaluating the safety and near-optimal scheduling of multiple tool calls executed concurrently. Shared state, such as files, accounts, locks, and lifecycles, introduces hazards that require careful ordering and aliasing considerations. Most benchmarks focus solely on single-call correctness, while Fan-Out assesses safety (no hazard violations like overwrites or double-spends) and optimal scheduling efficiency (SAO), measured by the number of rounds required.

The metric includes true concurrent scenarios (C0), sequential baselines, and a control condition. The oracle checks conflict DAGs and performs exhaustive interleaving to determine safety. No LLM judge is involved. The hardest rung involves eight concurrent calls (N=8), with models scoring 85.71% safe, 85.71% SAO, and a mean speed of 1.0× the oracle-optimal schedule.

Other models show varying performance in safety and SAO percentages, with some achieving 100% safety but lower SAO. The fan-out-swarm task highlights issues with cross-agent conflicts and the difficulty of achieving optimal multi-agent schedules. This novel approach to concurrency testing differs from previous work focused on data-flow/resource capacity, leaving shared-state hazards unaddressed.

The current implementation uses small item counts (n=7) and excludes certain runs due to format_ok=0 errors. Future work aims to expand the ladder to N=12, test additional models, and analyze safety versus speed trade-offs.