Just Train More: Measuring the Exchange Rate
The previous experiment found that a count table over the current document overtakes a small transformer at a few hundred tokens of document, and that the transformer's marginal contribution collapses past that point. There is one obvious objection and it deserves a measurement rather than an argument: Just train more. 8M tokens is nothing. Of course a 2.45M-parameter transformer loses; train it…
The investigation found that increasing the amount of training data substantially improves a transformer's performance, but the rate of improvement tapers off as the data pool grows larger. An experiment measured the crossover point where a 16,000-parameter cache outperforms a 2.45M-parameter transformer in predicting code. The crossover moved from 74 tokens of code for 500K training tokens to 492 tokens for 8M training tokens.
Extrapolating this trend suggests that reaching the performance of a zero-parameter cache would require 97 million tokens of code training data. This translates to roughly 12 times more code training data than what exists in popular Python libraries and deep learning frameworks combined. The exchange rate between document length and training data is relatively constant, with 16 times more training data moving the crossover 6.6 times for the transformer.
The experiment highlights the difference between training data and a zero-parameter cache, with the latter being more effective at a fixed short context window.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
