What I Learned Evaluating AI Image, Video and Audio Models

During my week, I work as a freelance AI evaluator, assessing output from models that create images, videos, audio, and user interfaces, and grading them using a structured rubric. This meticulous task has altered my perspective on software development. The first key takeaway is that adhering to instructions is more crucial than creating visually appealing results.

An image may appear beautiful, but if the initial prompt specified three people, but the model depicted four, it has failed the task. A simpler result that accomplishes exactly what was requested is considered superior. This principle is applicable to nearly every software I now create: fulfill the user's request first, then enhance the appearance.

The second insight is that false positives are far more detrimental than missed issues. Incorrectly flagging a problem that does not exist misguides the individuals training the model. I would rather be precise and overlook a minor problem than be inaccurate and make a loud claim. In my own projects, this translates to rigorous testing before labeling anything as either broken or fixed.

The third lesson is that concise and precise feedback is superior to lengthy opinions. A brief verdict that identifies the specific issue is more beneficial than a paragraph of subjective analysis. This practice extends to code reviews and client interactions. Engaging in evaluation work also underscores the significant role human judgment plays in refining these systems.

Models enhance due to human oversight, meticulous and consistent. For anyone venturing into AI or software, I suggest attempting evaluation work. You rapidly understand what constitutes good output and become aware of the discrepancy between a demonstration and a dependable product. Originally published at tonycantave.com by Tony Sterlin Cantave, a software developer from Connecticut and AI model evaluator.