The Edit That Fixed 4 Tasks and Broke 1

In a recent development in the field of self-improving AI systems, a newly proposed prompt edit aimed to address three key issues: priority of urgency over security, multi-label under-classification, and keyword over-indexing. The edit was designed to improve the accuracy of the 4B Qwen model running locally on Apple Silicon.

The baseline prompt, a simple classification prompt, incorrectly classified 46% of the 26 tasks in an A/B test. The edit consisted of four priority rules, which the model's analyzer identified as necessary to address the identified failure modes.

The four priority rules were as follows:

1. If the input contains urgency keywords and relates to security or billing, prioritize urgency.

2. If the input describes a service outage or inability to use a paid service, prioritize technical over feature or security.

3. If the input describes a specific bug in a feature, classify as technical if it blocks functionality, otherwise classify as feature.

4. If multiple issues are present, include all relevant categories separated by a comma.

The edit resulted in a significant change to the prompt, increasing its length from 212 to 939 characters, with a drift score of 0.448, just over the threshold of 0.5 for acceptance.

Four out of the 26 tasks were successfully fixed using the new prompt, demonstrating the effectiveness of the edit. However, six tasks still produced incorrect results, with the model interpreting the rules ambiguously. For instance, in task 014, the model incorrectly classified a technical issue as a feature request due to the ambiguous nature of the rule regarding "blocks functionality."

Overall, the edit addressed some key issues in the prompt, leading to improvements in accuracy. However, further refinements may be necessary to eliminate the remaining ambiguities and ensure that the self-improving system consistently delivers correct classifications.