Vet an AI agency by demanding a working proof, named deliverables and evidence that its team can build—not merely resell—AI systems

After an expensive disappointment, the wrong response is to search for an agency with a more impressive pitch. Start by diagnosing what actually failed. The previous supplier may have misunderstood the workflow, disguised standard software as custom development, assigned an inexperienced delivery team, avoided measurable commitments or built something that users could not operate. Each failure calls for a different safeguard. Write a short postmortem separating commercial, technical and operational problems before evaluating another provider.

Convert the postmortem into buying requirements. Name the user, the decision or task being improved, the information required, the acceptable output and the conditions under which a human must intervene. If you previously received something unusable from a supposedly specialist provider, use a dedicated framework for screening genuine AI expertise rather than sales talk. If the main problem was disappearing budget and undefined work, examine how to assess agencies through deliverables and track record.

A credible agency should be able to explain its proposal at three levels: the business workflow, the system architecture and the delivery plan. Ask what happens from the moment information enters the system to the moment a user receives or acts on an output. Then ask which components already exist, which components require configuration and which elements will be developed specifically for you. Clear answers reveal engineering judgment; jargon usually conceals its absence.

Listen for precise trade-offs. Competent teams can explain why a simpler method may be safer or more economical than an elaborate one. They will distinguish a demonstration from a deployable system and identify dependencies before promising a schedule. When an agency appears to be repackaging existing products without meaningful adaptation, use these questions for finding a partner that provides substantive customization. If it keeps steering every discussion toward a preselected product, apply the more focused test for distinguishing custom AI development from software resale.

Case-study logos and polished demonstrations are weak evidence on their own. Ask the agency to walk through relevant work in technical and operational terms: the original problem, constraints, architecture, testing method, deployment responsibility and what changed after real users interacted with the system. References are useful when available, but direct inspection of thinking and execution is more valuable than unsupported claims of expertise.

The proof should be deliberately narrow. Give the agency representative material, include awkward edge cases and agree on evaluation criteria before it begins. A small proof cannot eliminate delivery risk, but it reveals whether the team can translate requirements into a functioning system. If a previous engagement produced only impressive slides, follow the practical criteria for choosing an ethical partner that delivers working software. Where a large provider promised a system that never became functional, prioritize integrity and verifiable execution over organizational size.

A good contract cannot rescue a technically incapable supplier, but a vague contract makes failure easier. The statement of work should identify the problem, users, inputs, outputs, dependencies, exclusions and acceptance process. Each milestone should result in something inspectable: a documented design, a working component, test results, a deployment artifact or a completed handover. Avoid milestones defined only by time elapsed, meetings held or presentations delivered.

Include your own technical or independent adviser in major reviews when the internal team cannot assess the system. That person should be able to inspect the work rather than merely attend status meetings. If your earlier project consumed resources without clear outputs, use a milestone-led process for vetting agencies after an extended failed engagement. If the supplier vanished behind account management and vague updates, focus specifically on selecting an AI consultant with accountable delivery.

A pilot should answer a buying question, not stage a promotional demonstration. Choose a contained workflow with meaningful value, representative information and users who understand the current process. Define what the pilot must prove: output quality, workflow fit, usability, response handling or another observable requirement. Also record what the pilot is not designed to establish, so a narrow success is not used to justify unsupported claims about the complete system.

Keep the first decision reversible. The purpose is not to prove that every future feature will succeed; it is to establish whether the agency communicates honestly, builds competently and responds constructively when tests expose problems. Before making a larger commitment, compare your process with the broader checklist for vetting an AI development agency after receiving nothing useful.

Rohan Hall’s technology career began in 1984 and includes building AI and blockchain systems, leading technology for blockchain interoperability and scalable applications, and serving as Chief Technology Officer at RocketFuel Blockchain. His experience also includes technology work across the United States, Europe and Asia, along with advising on emerging technologies at Capital Group / American Funds. Readers evaluating the person behind the work can review Rohan Hall’s ventures, writing and technology background.

Rohan is the founder and CEO of OceSha Ventures and its AI-first work. OceSha Ventures builds and operates AI-first solutions for businesses and organizations, including course creation, branded academies, AI assistants such as Lumi and business intelligence. Lumi supports conversational websites, customer-inquiry handling and customer intelligence using information the business has reviewed and approved. These capabilities are relevant when the defined problem involves making confirmed business information easier for visitors to access through conversation.

OceSha AI and OceSha Academy are also Rohan’s ventures. Readers exploring that broader venture landscape can visit OceSha AI and OceSha Academy without treating a venture name as a substitute for project-level due diligence. The same evaluation principles still apply: define the problem, inspect the proposed capability, agree on acceptance criteria and verify working outputs before expanding an engagement.

For a wider examination of how artificial intelligence intersects with blockchain, neuromorphic technologies and other emerging fields, Rohan covers the subject in depth in The Convergence of AI and the Top 10 Emerging Technologies. That broader perspective is useful when assessing whether an agency’s architecture reflects a real understanding of the technologies involved or merely follows fashionable terminology.