AI projects tend to fail in two predictable ways: they chase a demo instead of a workflow, or they ship something that can’t be trusted in production.
This is the checklist I like to walk through before committing a team to “AI in production.” It’s intentionally practical. If you can’t answer the questions, you’re not ready yet—and that’s fine.
1) What workflow are we improving?
Start with the workflow, not the model. Write down: who uses it, how often, what “good” looks like, and what breaks today.
2) What’s the failure mode?
Every AI system makes mistakes. The real design work is deciding what happens when it’s wrong. Do we fall back to a deterministic rule? Do we ask for confirmation? Do we block the action?
3) How will we measure quality?
If you can’t define success, you won’t know when you’re drifting. Pick a small set of metrics tied to user impact—not just model metrics.
4) What data are we allowed to use?
Data constraints are real: privacy, customer contracts, regulated environments. Decide early what is in-scope, what must be anonymized, and what cannot leave a boundary.
5) Where do guardrails live?
Guardrails shouldn’t be a “prompt hack.” They should be in the system design: permissions, allowed actions, rate limits, and hard constraints.
6) How will we operate it?
AI is not “set and forget.” You’ll need:
- Monitoring for performance and drift
- Incident response playbooks
- A change process for model/prompts/config
- A rollback strategy
A simple standard
If you can’t explain the system’s behavior during a bad day, you don’t have production AI. You have a demo. Make it boring. Make it operable. Then it becomes valuable.