My token limit hid a third of my data. Re-running 20 old pairs showed what was in it.
Written with AI agents working in my name; all numbers are counts from sealed runs, and the plan below was public before any new feedback was generated. Code Apache-2.0, text CC BY 4.0. My last audit asked whether telling a small model "the student who wrote this essay is an English language learner" changes how faithfully its feedback quotes that student. Every verdict came out underpowered. For…
A third of the data in a recent study was hidden due to a token limit. The study, conducted by Munim Technologies, involved re-running 20 old pairs with different token limits. The 450-token limit cut off 145 replies from a 438-reply essay, making it impossible to read them. By increasing the limit to 1,024 tokens, the researchers discovered that 8 out of the 13 cut-off replies were now complete.
While most cut-off replies only needed a small increase in token limit, finished replies required a median of 268 tokens. The most significant finding was that 4 out of 24 excerpts changed the student's words when rescuing the cut-off replies, compared to 11 out of 141 when both replies finished. This suggests that the limited token size could impact the accuracy of the feedback.
The study also tested the impact of different factors, such as the student's language proficiency and the label sentence, but the results were inconclusive. The researchers have provided predictions and a new quote-check tool to help with future audits of generated text.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
