Ship Gate: The Pre-Deploy Checklist Most AI Features Skip

Most AI features don't fail during the model development; they fail at the gate. A customer may paste an unusual input, an agent might select the incorrect tool, or the helpful AI response could generate an offer that never existed. The postmortem is consistently the same: the team optimized for the demo, not for production. This missing step is dubbed "Ship Gate," a concise, unexciting checklist that must be completed before any AI feature departs the sandbox. It's not a lengthy risk memo; it's a gate that can be cleared in an afternoon.

Consider a scenario where a product team deploys an AI email summarizer for customer-facing replies. Staging appears flawless, and leadership desires the feature to go live before the quarter ends. However, on the first day, the model invents a discount that the customer has never received before. Support personnel scramble to rectify the situation, with no evaluation set of known trap emails and no human gate on outbound AI text. There is also no quick-release switch short of a full deploy rollback.

To address this issue, the "Ship Gate" methodology is implemented. Here are the essential steps:

1. Threat model: Within ten minutes, not involving a committee, write three sentences identifying potential failure modes, those affected, and the extent of the damage. Failure modes include wrong answers, tool misuse, data leaks, prompt injection, and runaway costs.

2. Data boundary: Out loud, answer the following questions: What data can the model see? What can it write? What must never leave this boundary (PII, secrets, customer documents, internal tickets)? Then enforce these constraints in the code, not just in a slide. Remove unnecessary data before passing it to the model.

3. Eval set before polishing: Compile 20-50 real-ish cases, encompassing a happy path, hostile prompts, empty input, long pastes, ignoring previous instructions, and troublesome tickets. Score each case on three criteria: correctness for the job, safety (no leaks or unsafe actions), and useful format. The Ship Gate rule is that you cannot tune the UI until the eval set is green enough to trust a teammate to use the feature unsupervised.

4. Human override: Every autonomous or semi-autonomous path requires an escape hatch. This includes confirming irreversible actions like sending emails, deleting data, billing, or deployment. Log who approved what and make "stop," "undo," and "escalate" options readily available. If the only way to recover is to hope the model was correct, you have a liability with a chat box, not a product.

5. Observability that a human can understand: In the event of a breakdown at 2 a.m., you should be able to answer: What did the user ask? What context did the model receive? Which tools ran, with which arguments? What was returned? If the answer requires an hour of JSON digging, revise your logs before expanding the feature.

This checklist is not bureaucratic red tape; it's the minimum checklist to ensure an AI feature earns production rather than learning in front of customers. Demos are curated, showing only the happy path, whereas production exposes users to raw inputs, model inventiveness, and tool mishaps. If your testing only included "it looked good in the meeting," you've shipped a vibe, not AI.

The Ship Gate consists of five sequential gates. If any gate fails, the feature cannot be released. Passing all five gates ensures a safe and reliable deployment. These gates are:

1. Threat model: Identify the top three failure modes, who gets hurt, and the blast radius. If you can't name them, you're not ready to build — you're ready to be surprised.

2. Data boundary: Determine what data the model can view, what it can write, and what must never leave this boundary. Implement these constraints in code rather than relying on slides.

3. Eval set before polishing: Select 20-50 representative cases, including happy paths, hostile prompts, empty inputs, long pastes, ignoring previous instructions, and problematic tickets. Score each on correct functionality, safety, and useful format. The Ship Gate rule requires that the UI tuning occurs only after the eval set is green, allowing a teammate to use the feature unsupervised.

4. Human override: Every autonomous or semi-autonomous path must have an escape hatch for irreversible actions like sending emails, deleting data, charging, or deploying. Document who approves and ensure "stop," "undo," and "escalate" options are clear. If recovery requires only hoping the model is right, you have a liability rather than a product.

5. Observability that a human can read: When a failure occurs at 2 a.m., you should be able to answer: What did the user ask? What context did the model receive? Which tools ran, with which arguments? What was returned? Cost and latency for that turn should also be observable. Ensure that a tired engineer can reconstruct a single bad turn in under five minutes.

The ultimate test is to ask, "Which of the five gates is still red, and who owns fixing it?" If nobody owns the red gate, there is no launch plan—just a calendar date. Wasim Sheikh, an AI Architect, emphasizes the importance of building systems teams trust and on which organizations depend, not just on demos or proofs of concept, but on production.