The Last Non-Neural Candidate, and It Did Not Clear the Bar
The question this whole series exists to answer is whether language-model-like behaviour can come from something other than a large number of trained parameters. An early set of experiments turned that into a bar that a non-neural method has to clear, and the bar is a slope rather than a score: keep converting extra data into accuracy after exact-context statistics have saturated. Measured as…
This investigative piece examines whether a non-neural language model can achieve the same performance as neural models. Researchers established a bar for non-neural methods based on a slope, not a score, measuring top-1 gain per doubling of the training corpus. The results show that the single-pass Pitman-Yor model, the strongest non-neural candidate identified, does not clear this bar.
The model's performance plateaus at 0.008 per doubling, similar to weaker models. The full hierarchical Pitman-Yor model with Gibbs sweeps and inferred hyperparameters, the strongest remaining non-neural candidate, also fails to clear the bar. However, the research reveals that the single-pass approximation is leaving real accuracy behind, and the full model outperforms it at various sizes.
The study concludes that non-neural methods clearing this bar will require a different kind of state, not a better-fitted n-gram hierarchy.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
