Urgent.News

What's breaking now, across thousands of outlets.

AI

Is prompt engineering real AI development, or just good documentation?

I’ve heard both answers from people I respect. One senior engineer told me prompt engineering is “writing a very long comment and hoping.” A teammate who ships LLM features for a living told me it’s the highest-leverage skill in AI development right now. After building a fair number of LLM-powered features, I think they’re both […] The post Is prompt engineering real AI development, or just good…

Is prompt engineering real AI development, or just good documentation?

Two opinions on prompt engineering persist: some consider it akin to good documentation, while others view it as genuine AI development. A senior engineer likened it to "writing a very long comment and hoping," while a teammate handling LLM features for a living referred to it as the most impactful skill in AI development today. After working on numerous LLM-powered features, this author believes both perspectives hold some validity, and the key question is when each one applies.

Advocates of "prompt engineering as documentation" argue that an effective prompt resembles well-written documentation. It outlines the objective, defines terms, provides examples, addresses edge cases, and specifies the desired output. If you have experience crafting clear READMEs or onboarding documents, you are already familiar with most of the skills required.

However, unlike code, prompts lack a compiler and cannot be unit tested. A single prompt may yield differing results across days, and "it works on my five examples" offers little insight. Consequently, if prompt engineering consists of merely rephrasing until a demo looks satisfactory, then it is more akin to editing rather than engineering.

On the other hand, proponents of "prompt engineering as real AI development" draw a distinction when evaluation comes into play. Genuine prompt engineering involves a test set of real inputs, encompassing challenging scenarios, alongside a scoring method - be it human review, predefined rules, or a model-as-judge. Versioning prompts enables tracking of changes and their impact, while regression checks are conducted whenever the prompt or underlying model undergoes modifications.

Decisions regarding where a prompt should cease and where code, retrieval, or fine-tuning should commence are also integral. This process resembles an engineering loop but utilizes natural language as the interface, turning the prompt into a component with inputs, outputs, failure modes, and maintenance costs.

The author ultimately concludes that the differentiating factor between the two notions is measurement. A prompt without evaluations is akin to a guess with good grammar, while one with evaluations, versioning, and clear ownership constitutes a real part of the system. The author advises teams embarking on prompt engineering to initially write the prompt as a specification, build a range of real test cases before tuning, and meticulously track each prompt change, akin to code modifications.

The author also emphasizes the importance of acknowledging when to stop prompting and relying on retrieval, tools, or conventional code when necessary. The prompts' reliance on the specific model they were created for further emphasizes the need to re-run tests whenever the model is updated.

Ultimately, the author posits that if prompt engineering is indeed considered "real" development, then a considerable number of AI projects may have been deployed without actual development work. Robust demos accompanied by inadequate testing could have been shipped. Conversely, if prompt engineering merely amounts to good documentation, teams might be relying on an undisclosed process that ensures reliable results.

The author invites readers to share their perspectives on where to draw the line between prompt engineering and good documentation, asking specifically about experiences with shipping prompt-heavy features to production and the factors that contributed to their reliability.

Written by urgent.news from e27's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at e27.co →

More in AI

AI Manages Sleep and Bowel Movements

Artificial intelligence is increasingly being used to monitor sleep, track vital signs and manage daily care in nursing homes across Hokkaido as Japan's aging population and worsening labor shortages…

  • AI monitors sleep and vital signs in Japanese nursing homes
  • Elderly care delegation studies Nordic AI welfare systems
  • Swedish residents show positive outlook on aging

Culture ministry to host forum on AI use in culture, sports, tourism

The Ministry of Culture, Sports and Tourism will host a forum Tuesday on how artificial intelligence (AI) is being used in culture, sports and tourism and on how these sectors should shift to AI. The ministry said Monday the fourth Culture, Sports and Tourism Digital Innovation Forum, themed “New Culture Through AI,” will be held at L…

More from Monday 16 November →