Typos don't break LLM prompts. One missing quote mark does.

In a test to determine the impact of typos on language models, the following findings emerged:

1. Spelling and grammar mistakes did not affect the accuracy of the models' responses. They consistently provided correct answers even when 35% of the words were misspelled or when the prompt contained non-native grammar or voice-to-text errors.

2. The punctuation marks had a significant impact on the models' performance. One missing or misplaced punctuation mark resulted in a drop of 8 to 23 points in accuracy across various models.

3. The specific punctuation mark that caused the most damage varied among models. For example, the missing closing quote led to incorrect answers in 33% of the cases for Opus 5 and Opus 5.5, while the colon dropped before a list resulted in a 100% accuracy for those models.

4. The missing quote mark had a particularly severe impact, causing models to count extra instances of a word in a quoted paragraph. This led to incorrect answers even when the models recognized the structural error and tried to correct it.

5. The dropped colon also caused confusion for models, leading to incorrect pluralization of words and misinterpretation of the intended meaning in a sentence.

6. Dictation, especially by voice-to-text models, performed poorly when faced with spelling mistakes, numerical errors, and missing punctuation, such as colons.

In summary, the study revealed that while spelling and grammar mistakes had minimal impact on language models' performance, punctuation marks, particularly missing or misplaced ones, had a significant effect on their accuracy and ability to understand the intended meaning.