44% on ARC-AGI-1 in 67 cents

A small transformer model, named ARC-AGI-1, was trained in just 1.5 hours from scratch, achieving results comparable to many large language models (LLMs). This marks an upgrade to the previously trained Faster, Better, Cheaper model. The 3rd installment in a series of works on ARC-AGI, this model was developed with the intention to address the problem of sample efficiency in AI.

The research aims to find the limits of sample efficiency when restricted to transformers and deep learning methods, ultimately reducing costs to make iteration faster and cheaper.

The key improvements in the new model include training solely on output tokens instead of input tokens, resulting in a supervised approach rather than the unsupervised style of previous models. This change led to a slight increase in performance, with the model scoring 44% compared to the previous 40%. Additionally, the training data was expanded by incorporating non-overlapping tasks from ARC-2, carefully avoiding any data leakage.

The model's performance is attributed to good representations, such as 3D RoPE and per-task embedding. The code is open source, encouraging modifications to improve the score or reduce cost. The authors found that reaching 65% performance does not require many modifications, indicating the potential for further advancements within the transformer framework.

Despite reaching 44% performance with a transformer model, the authors express surprise, as they did not anticipate such results. They also note that costs can potentially be reduced by 10x with handmade GPU code, architectural changes, or removing data augmentations. The authors believe that 65% performance can be reached within the transformer framework, but they are puzzled by the lack of progress made by others in this area.