AI Agent Evaluation: How the LFORLA JS 3D Builder Leaderboard Actually Scores Models

The LFORLA JS 3D Builder Leaderboard is a benchmark for evaluating AI agents that generate 3D buildings from written specifications. It works by having the model read a building specification, plan the structure, select the appropriate Three.js parts, and output code that assembles the building. The key difference between evaluation methods is that this one uses server-side static analysis of the code, without running it or relying on another model's judgment.

This means the score reflects the code as written, rewarding correctness over presentation. The leaderboard uses a fixed rubric that scores the model's code, reasoning, 3D construction, and spatial relationships. A score below 70 indicates gaps in these areas, while a score above 78 means the model's code is more closely aligned with the specification.

The categories matter because a model might excel at code generation but struggle with spatial reasoning. Practitioners should consider the categories when choosing a model for 3D or spatial tasks, as a lower overall score may not indicate a fundamentally weaker model. The benchmark's limitations include not executing the code, its fixed rubric which may miss certain aspects of agent capability, and a small number of entries.

Despite these limitations, the leaderboard provides a transparent mechanism for evaluating AI agents' ability to plan, source parts, and assemble a build plan, with scores reflecting the code's adherence to the specification.