I measured whether my Claude skills help. One of them made the code worse

The SKILL.md file is a program written in English, which a model runs and can have bugs. The most common way to check one is to read it back and decide if it sounds right, but the author of this story used a different method to measure the effect of their Claude skills on code generation. They built skill-be, an experiment that changes one thing and compares the results.

Each skill was tested with 46 scenarios and 139 criteria, with the model writing code using the skill and another model grading the results. The results showed that one skill made the code worse, while 111 criteria improved with the skill. The skill improved the average criterion score from 22% to 96%.

The specific skill that made the code worse was the validate skill, which required validating email and password synchronously, and checking that the email is not already taken with an async isEmailTaken(email): Promise. With this skill, the model wrote plain TypeScript without using mini-fp, resulting in code that accumulated every error correctly. However, the model also branched on isInvalid() and unwrapped the result, which is what mini-fp exists to replace.

The skill-be tool's triage command suggested a skill gap as the cause, as the skill lacked an example of saving on success and responding with the errors otherwise. The solution was adding a short recipe and a gotchas section to the skill's description, which helped the model generate the correct code on the next run.