Is prompt engineering real AI development, or just good documentation?
I’ve heard both answers from people I respect. One senior engineer told me prompt engineering is “writing a very long comment and hoping.” A teammate who ships LLM features for a living told me it’s the highest-leverage skill in AI development right now. After building a fair number of LLM-powered features, I think they’re both […] The post Is prompt engineering real AI development, or just good…
Two opinions on prompt engineering persist: some consider it akin to good documentation, while others view it as genuine AI development. A senior engineer likened it to "writing a very long comment and hoping," while a teammate handling LLM features for a living referred to it as the most impactful skill in AI development today. After working on numerous LLM-powered features, this author believes both perspectives hold some validity, and the key question is when each one applies.
Advocates of "prompt engineering as documentation" argue that an effective prompt resembles well-written documentation. It outlines the objective, defines terms, provides examples, addresses edge cases, and specifies the desired output. If you have experience crafting clear READMEs or onboarding documents, you are already familiar with most of the skills required.
However, unlike code, prompts lack a compiler and cannot be unit tested. A single prompt may yield differing results across days, and "it works on my five examples" offers little insight. Consequently, if prompt engineering consists of merely rephrasing until a demo looks satisfactory, then it is more akin to editing rather than engineering.
On the other hand, proponents of "prompt engineering as real AI development" draw a distinction when evaluation comes into play. Genuine prompt engineering involves a test set of real inputs, encompassing challenging scenarios, alongside a scoring method - be it human review, predefined rules, or a model-as-judge. Versioning prompts enables tracking of changes and their impact, while regression checks are conducted whenever the prompt or underlying model undergoes modifications.
Decisions regarding where a prompt should cease and where code, retrieval, or fine-tuning should commence are also integral. This process resembles an engineering loop but utilizes natural language as the interface, turning the prompt into a component with inputs, outputs, failure modes, and maintenance costs.
The author ultimately concludes that the differentiating factor between the two notions is measurement. A prompt without evaluations is akin to a guess with good grammar, while one with evaluations, versioning, and clear ownership constitutes a real part of the system. The author advises teams embarking on prompt engineering to initially write the prompt as a specification, build a range of real test cases before tuning, and meticulously track each prompt change, akin to code modifications.
The author also emphasizes the importance of acknowledging when to stop prompting and relying on retrieval, tools, or conventional code when necessary. The prompts' reliance on the specific model they were created for further emphasizes the need to re-run tests whenever the model is updated.
Ultimately, the author posits that if prompt engineering is indeed considered "real" development, then a considerable number of AI projects may have been deployed without actual development work. Robust demos accompanied by inadequate testing could have been shipped. Conversely, if prompt engineering merely amounts to good documentation, teams might be relying on an undisclosed process that ensures reliable results.
The author invites readers to share their perspectives on where to draw the line between prompt engineering and good documentation, asking specifically about experiences with shipping prompt-heavy features to production and the factors that contributed to their reliability.
Written by urgent.news from e27's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.