Why a Small Transformer Can't Copy a Word It Hasn't Seen

A small transformer is unable to copy words it has not seen before. The transformer in question, weighing in at 11.9 million parameters, 6 layers, and a dimension of 384, was trained on 4,176 generated programs. When given a support ticket system with various features, it successfully generated a 1,158-token Python file that compiled and implemented the required functionalities without including comments, edit, delete, or category features. This transformer exhibited a 5 out of 6 success rate for held-out feature combinations.

The author built this transformer to validate a specific claim, which was later found to be incorrect. The purpose of this article is to retract the previous argument and provide an explanation for why the model fails. The author conducted three experiments to understand the reasons behind the transformer's shortcomings, one of which corrected a previously identified diagnosis.

The claim that the shipped tool does not utilize the model is being retracted. Instead, the tool employs compositional synthesis: a parser converts the user's request into an entity schema and feature set, while hand-written emitters construct the corresponding program. This synthesis process is exact within its grammar and takes approximately 2 milliseconds on a CPU.

The cost of this approach lies in human labor, as each program kind requires 160 to 287 lines of hand-written emitter code, and each language necessitates an additional 104 to 231 lines. Importantly, there is no transfer of knowledge between different program kinds or languages.

Initially, the author argued that a learned generator would amortize the labor of creating hand-written emitters. However, it was later discovered that training the model to 3.5 times lower loss resulted in no change in the failure distribution. The author then identified two one-line changes to the data pipeline as the solution: splitting docstrings and introducing an 'eos' token.

These changes increased the vocabulary size from 1,817 to 1,817 and improved the output quality, achieving a 6/6 compile and 5/6 serves HTTP success rate for strict feature combinations.

Despite the improvements, the model still failed when presented with a library domain (entity book, table bookings) that was held out of the training corpus. The generated code consistently copied nine consecutive words from the input prompt, swapped 'book lending' for 'address book', and replaced 'books' with 'bookings', resulting in incorrect tables for the generated code.

This failure demonstrated that the transformer could copy nine-word spans but struggled to copy the crucial single word needed to generate the correct output.

The author's initial diagnosis, which suggested that capacity or training time were the primary issues, was proven incorrect. To rectify the problem, the author conducted a 30-line probe against the checkpoint, which led to the realization that the model requires pointer attention or byte-level entity tokens to successfully copy words it has not seen before. The learned generator's failure to perform as expected highlights the limitations of current transformer architectures in handling unseen input data.