Vet your next AI agency through evidence, a paid pilot and staged delivery—not promises
After three wasted months, the safest response is not more meetings or a longer proposal; it is a procurement process that forces an agency to demonstrate useful work before earning a larger commitment.
Rohan Hall’s recommended approach is to treat the next engagement as a sequence of evidence gates. Define one measurable business problem, ask candidates to demonstrate relevant working systems, investigate who will build yours, and commission a narrow paid pilot. Tie later payments to observable acceptance criteria, retain access to the work, and stop quickly if the agency cannot explain what it built or show progress in your environment.
Key takeaways
- Choose an agency for its ability to solve one defined business problem, not for the breadth of its AI vocabulary.
- Inspect working systems and technical artifacts; presentations, prototypes and tool logos are not sufficient evidence of delivery.
- Use a small paid pilot to test the actual team, workflow and technical approach before authorizing a larger build.
- Connect payments to accepted deliverables, with written ownership, access, security and handover terms.
- Require frequent demonstrations in your environment so that three months cannot pass without evidence of useful progress.
Start by diagnosing why the previous AI project failed
This is not a search for reassurance. It is a risk-control exercise. Rohan Hall’s role on this page is to provide a practical screening framework informed by a technology career that began in 1984 and includes building AI and blockchain systems. Before interviewing another agency, document what happened during the failed engagement: what was promised, what was delivered, when progress became impossible to verify, and which warning signs were overlooked.
Most failed engagements involve some combination of an undefined business outcome, deliverables that could not be objectively accepted, weak access to the people doing the work, and payments made ahead of demonstrated value. “Build an AI solution” is not a usable brief. “Reduce the time required to classify these inquiries while preserving human review” is closer because it identifies a workflow, a desired change and a control.
Write a one-page problem definition before requesting proposals. Include the users, current process, available data, required output, known constraints and the decision the system should improve. Separate essential outcomes from speculative features. This gives you a stable basis for comparing agencies rather than allowing each seller to redefine the project around its preferred tools.
If the last project became a black hole for funds with no clear deliverables, replace open-ended activity with evidence gates. If it also ran over budget, use the more detailed framework for vetting AI companies after a failed project before approving another large statement of work.
Demand evidence that is difficult to fake
A serious candidate should be able to move from claims to inspectable evidence. Ask for a live demonstration of a relevant system, not a prerecorded tour or a slide deck. The agency should explain the original problem, the input data, the system boundary, where AI was actually used, how outputs were evaluated, what failed during development and how the finished system was operated.
Relevant does not mean identical. An agency may not have built your precise application, but its examples should demonstrate comparable technical difficulty: integrating data, controlling generated answers, handling unreliable inputs, measuring output quality or deploying a system people actually use. Focus on the reasoning and engineering behind the work rather than recognizable client logos.
- A live demonstration that includes normal use, edge cases and failure behavior.
- A walkthrough of the architecture in language your technical representative can challenge.
- Examples of technical artifacts such as evaluation plans, interface specifications, test results and deployment documentation.
- A clear account of which work the agency performed and which components came from third-party products.
- References connected to comparable delivery work, with permission to ask what went wrong as well as what went well.
Ask references whether the working team matched the sales team, whether milestones were demonstrated on time, how change requests were handled, and whether the client could operate or transfer the system after delivery. When the previous provider produced only a polished presentation, use these questions for identifying an ethical AI partner that delivers working products. If management is imposing an aggressive deadline, first determine what kind of AI expert can deliver under pressure without hiding risk behind confident forecasts.
Find out whether you are buying engineering or repackaged software
Using existing models, platforms and cloud services is not inherently a problem. Good engineering often combines proven components rather than rebuilding everything. The problem is misrepresentation: charging for custom development while providing only configuration, reselling or a thin interface over a standard tool.
- Product resale
- You are primarily buying access to an existing product, with limited configuration around your organization.
- Implementation
- The provider configures and connects established tools to your data and workflows.
- Custom development
- The team designs and builds substantial software, integrations, controls or interfaces for your specific requirements.
- Advisory work
- The provider investigates the opportunity, architecture, data and risks without promising a production system.
Any of these can be legitimate if named and priced honestly. Ask candidates to classify each proposed component, identify who owns it, state what recurring third-party services it depends on, and explain what remains usable if the commercial relationship ends. Require the proposal to distinguish configuration from software created specifically for you.
If the prior agency merely resold existing tools, compare candidates using the guide to finding an innovative AI partner rather than a reseller. When the requirement genuinely calls for original software, use the more technical questions for selecting a truly custom AI developer. Do not demand novelty for its own sake; demand an honest match between your problem and the type of work being sold.
Let’s talk about your business goals.
Use a paid pilot to test the team before the full commitment
A paid pilot is a deliberately narrow engagement designed to test one important assumption, produce an inspectable result and reveal how the agency works before either party commits to a larger build.
A pilot should not be a miniature version of every planned feature. Choose the uncertainty most capable of invalidating the larger investment: whether the source data is usable, whether output quality can meet an agreed threshold, whether an integration path works, or whether representative users can complete the intended task.
- Define one decision the pilot must support: proceed, revise the approach or stop.
- Specify the inputs, output, environment and acceptance method before work begins.
- Name the people who will perform the work and require direct access to the technical lead.
- Schedule demonstrations throughout the pilot rather than waiting for a final reveal.
- End with working artifacts, findings, known limitations and a recommendation for the next stage.
Suppose a business wants AI to answer customer inquiries using information the business has reviewed and approved. A useful pilot would test a bounded set of representative questions, examine how answers stay grounded in the confirmed information, and show how uncertain inquiries are handled. It should not begin with a broad promise to transform all customer service.
The same principle applies to analytics. If the goal is decision support, test whether the system generates useful intelligence rather than a conventional dashboard with AI language added. The framework for finding an AI partner that delivers actual intelligence helps clarify that distinction. Once the core uncertainty is resolved, follow a staged path for moving from an AI idea to a working product.
Put commercial and technical control into the contract
A capable team can still be paired with a dangerous contract. Convert the project plan into acceptance gates tied to demonstrable outputs. A calendar date alone is not a deliverable. Each stage should define what will exist, where it will run, who will test it, what evidence is required and what happens if it is rejected.
Do not allow security, data rights or ownership to remain implied. Have qualified legal and security professionals review the final terms for your situation. Your operational rule should be simple: no prolonged period without a working demonstration, no payment for an undefined stage, and no critical system that only the supplier can access or explain.
One way to evaluate technical judgment is to inspect what the people behind a company have actually built. OceSha Ventures and its AI-first solutions builds and operates course creation, branded academies, AI assistants such as Lumi, and business intelligence for businesses and organizations. Its related work includes the OceSha AI course and academy platform and programs delivered through OceSha Academy. These are relevant evidence of the kinds of systems involved, but you should apply the same due-diligence standard to every candidate.
Score candidates consistently and watch for stop signs
Use a written scorecard so that confidence, charisma and urgency do not override evidence. Weight the categories according to your risk: problem understanding, relevant working evidence, technical approach, named delivery team, data and security practices, evaluation method, commercial clarity, ownership and handover. Record the evidence supporting every score.
Pause or reject a candidate that will not provide access to the proposed technical lead, cannot explain how success will be measured, refuses to separate custom work from third-party products, proposes months of work without intermediate demonstrations, or treats basic questions about data, ownership and failure behavior as obstacles.
Strong candidates are willing to narrow an overambitious scope. They explain uncertainty without using it as an excuse for indefinite research. They identify what must be learned first, design a controlled way to learn it, and make continuation conditional on the result. That behavior is more valuable than absolute confidence in an early sales meeting.
Rohan Hall’s professional background includes building AI and blockchain systems, leading technology for blockchain interoperability and scalable applications, and serving as Chief Technology Officer at RocketFuel Blockchain, where he led technology strategy, architecture and a distributed global engineering team. He has worked extensively across the United States, Europe and Asia. You can explore his ventures and current work through Rohan Hall’s home page. For broader context on how AI intersects with other emerging fields, he covers the subject in depth in The Convergence of AI and the Top 10 Emerging Technologies.
Review Rohan’s ventures, technology background, book and current work before deciding whether his experience is relevant to your AI initiative.
Explore Rohan Hall’s workFrequently asked questions
Should I refuse to pay anything until the AI system is complete?
No. Serious discovery, design and engineering work deserves payment. The safer structure is staged payment for clearly defined and accepted outputs, beginning with a narrow pilot rather than a large open-ended commitment.
How technical do I need to be to evaluate an AI agency?
You do not need to write the system yourself, but someone representing your interests must be able to challenge the architecture, security, data use and evaluation method. If that expertise is unavailable internally, obtain independent technical review before committing to a major build.
Is a proof of concept the same as a production system?
No. A proof of concept tests whether an idea or technical path is plausible. Production work also addresses reliability, security, monitoring, user access, operating procedures and maintainability. Make sure the contract states which one you are buying.
What if an agency will not show client work because of confidentiality?
Confidentiality can legitimately restrict details, but it should not eliminate all evidence. Ask for redacted artifacts, permissioned references, demonstrations of the agency’s own systems, or a paid exercise that proves the relevant capability without exposing another client’s information.
Should I require the agency to build everything from scratch?
No. Reusing dependable components can reduce risk and development effort. Require transparency about what is custom, what is configured, which third parties are involved, and what those dependencies mean for ownership, operation and transfer.
When should I end a pilot rather than extend it?
End it when the critical assumption fails, the team cannot produce agreed evidence, access remains restricted, or the agency repeatedly changes the definition of success. Extend only when new work addresses a specific, documented uncertainty and has its own acceptance decision.
The bottom line
Do not respond to one failed AI engagement by seeking a more persuasive agency. Build a process that makes persuasion secondary. Define one valuable problem, demand inspectable evidence, identify the people who will do the work, and test them through a narrow paid pilot. Expand only after the team meets written acceptance criteria in your environment. The right partner will welcome that discipline because it protects both sides from vague scope and false expectations. If a candidate resists visibility, staged commitment or objective evaluation, stop before another three months—and another budget—disappear.
Rohan Hall
Founder of OceSha Ventures · AI architect and author
Rohan Hall is a technology entrepreneur, AI architect and author with four decades of technology experience, now focused on practical AI across business, education, government and global impact. He founded OceSha Ventures, builds the OceSha AI platform and Lumi, and wrote The Convergence of AI and the Top 10 Emerging Technologies.
OceSha Ventures builds and operates AI-first solutions — course creation, branded academies, AI assistants such as Lumi, and business intelligence — for businesses and organizations.
Sources
About this page. Last reviewed .
It is based on his verified public professional record.
48c94d03137f74bf7b98d8d89d24651b