Six AI models graded student essays. Do they quote the student's words exactly? This time all three predictions held

Six AI models were used to grade student essays, confirming three predictions during this study. Across all six models and three prompts, only 1.18% of the 27,462 checked quotes changed the student's words (95% CI 0.83–1.76%). Short quotes (4–6 words) were more likely to change than long quotes (11+ words): 2.29% versus 0.68%. This trend aligns with previous findings and remains true despite long quotes having more words that could change.

Larger models changed words more often than smaller ones (3.85% versus 0.81%), but this is not due to size alone. LearnLM, an education-tuned model, quoted the most (21,317 of the 27,462 quotes) and changed words the least (0.22%), pulling the average down for larger models. Mistral-24B, a larger model, had the highest rate at 13.2%.

Per-model rates ranged from 0.22% to 13.2%, a nearly two orders of magnitude difference. Few-shot prompting had the highest change rate (4.32%) and quoted the least. Human teachers rarely used quotation marks for student words, appearing only 9 times in 198 teacher files. The study also found that a third of changed quotes were "loose," and a similarity band could not determine genuine misquotes from suggested phrases put in quotation marks.

The open-source tool quote-check, used in this study, confirms its ability to flag per-quote changes regardless of the model, displaying similarity and length together. The full plan, code, tests, and counts are available at https://github.com/munimtechnologies/feedback-quote-audit.