Verify an AI vendor through evidence, architecture and a paid pilot—not an impressive demonstration
Before committing serious money, require the vendor to explain what the system learns, prove it with your data and demonstrate that the proposed architecture is appropriate for the business problem.
Treat “AI” as a claim that must be demonstrated. Begin with a precise business problem and measurable acceptance criteria. Ask the vendor to separate machine learning, generative models, rules, workflows and human operations in its architecture. Then inspect relevant technical work, test the system with unfamiliar cases and fund a limited pilot before approving full deployment. A credible vendor will welcome this scrutiny because its expertise remains visible when the presentation ends.
Key takeaways
- Define the business decision or workflow before evaluating models, demonstrations or vendor credentials.
- Require an architecture explanation that distinguishes AI models from deterministic rules, ordinary software and manual work.
- Test with private, unfamiliar examples; a rehearsed demonstration proves only that the demonstration works.
- Use a paid pilot with written acceptance criteria, staged payments and a clear stopping point.
- Evaluate whether the vendor can operate the complete system, including data quality, monitoring, security and failure handling.
Start by diagnosing the failure you need to prevent
Losing money on a supposed AI project is usually not a failure to recognize sophisticated terminology. It is a failure to convert a sales claim into a testable engineering claim. A polished interface can sit on top of fixed rules, third-party APIs, ordinary database queries or manual work. Those components are not inherently bad; many useful systems combine them. The problem is paying for custom intelligence when the vendor is delivering conventional automation without disclosing the distinction.
Genuine AI capability means the vendor can identify the model-driven part of the system, explain why it is needed, show how inputs affect outputs, test its behavior on unseen cases and operate it responsibly in the intended environment. This does not mean every component must use machine learning. It means the architecture is honest, technically coherent and appropriate to the problem.
Write down the decision, prediction, generation task or interaction the system must improve. Name the users, available data, acceptable errors, prohibited outcomes and the evidence required for acceptance. If you cannot define those points, pause procurement. A vendor cannot be held accountable for “building AI,” but it can be held accountable for a specified system that performs a defined job under stated conditions.
Do not buy the category label. Buy demonstrated performance against a clearly defined business requirement.
Make the vendor expose the architecture behind the interface
Ask for a plain-language system diagram before discussing a large contract. It should show where data originates, what transformations occur, which components are model-driven, where deterministic rules apply, which external services are involved, how humans intervene and where outputs are stored or acted upon. The vendor should be able to explain the same architecture to both a technical reviewer and an accountable business owner.
- What exact part of the proposed system uses a model, and why is a model preferable to simpler software there?
- Is the model trained, fine-tuned, retrieved against business information or accessed through an external service?
- Which outputs are generated by the model, which are produced by rules and which require human work?
- What happens when the input falls outside the examples used during development?
- How will the team measure accuracy, usefulness, latency, failure rates and changes over time?
- Who owns responsibility for data preparation, evaluation, deployment, monitoring and correction?
Strong answers are specific and falsifiable. Weak answers retreat into phrases such as “proprietary intelligence,” “revolutionary algorithms” or “our secret sauce” without showing what the technology does. Confidential methods can remain confidential while the vendor still explains system boundaries, dependencies, evaluation methods and operational responsibilities.
- Rules-based automation
- follows predefined conditions and produces predictable actions; valuable when the process is stable and explicit.
- Machine learning
- derives patterns from data and should be evaluated against unseen examples and relevant baselines.
- Generative AI
- produces new text, images, code or other content probabilistically and requires quality and safety evaluation.
- AI-enabled software
- combines models with ordinary software, workflows, databases and interfaces; often the right architecture when each layer is disclosed.
- Human-assisted service
- relies on people for part of the output; legitimate when the labor, turnaround and scaling implications are clear.
If your earlier loss involved rules being sold as machine learning, use the more focused checklist for finding a vendor that delivers actual machine learning. If the failure was broader—missed milestones, vague deliverables or no usable result—apply the companion framework for vetting an AI partner after a failed agency project.
Demand evidence that survives contact with unfamiliar data
A sales demonstration is not evidence of general capability. It may use selected examples, hidden prompts, prepared responses or data the team has already tuned around. The decisive test is whether the proposed system behaves acceptably on representative cases that were not part of the presentation.
- Create a private evaluation set representing routine cases, difficult cases, ambiguous inputs and unacceptable outcomes.
- Keep part of that set undisclosed until the live evaluation so the vendor cannot optimize every response in advance.
- Establish a non-AI baseline, such as the current workflow or a simple rules-based approach.
- Agree on scoring before the test, including how subjective output will be reviewed and who decides whether it passes.
- Observe failures as closely as successes. Ask what caused each failure, whether it is detectable and how it would be corrected.
- Repeat selected tests after changes to determine whether improvements are consistent rather than anecdotal.
Suppose the proposed system will answer customer inquiries from information the business has reviewed and approved. Give it questions that are directly answered, questions phrased in unfamiliar ways, questions outside the available information and questions that combine conflicting assumptions. A credible system should use the confirmed business details appropriately and handle unsupported requests according to the agreed behavior. OceSha Ventures applies AI assistants such as Lumi to conversational websites, customer inquiry handling and customer intelligence using approved answers.
Do not confuse open-source use with fraud. Competent teams routinely build on open-source components. The issue is whether the vendor discloses dependencies, contributes meaningful architecture and implementation work, follows applicable licenses and charges in proportion to the value delivered. If reuse was concealed in your previous project, examine how to evaluate expertise built on open-source components and the stricter ethical questions involved in selecting an AI developer after undisclosed code reuse.
Let’s talk about your business goals.
Verify the people and operating capability, not just the prototype
A functional prototype proves only that someone assembled a functional prototype. Production AI also depends on data engineering, software architecture, security, testing, deployment, monitoring and business-process integration. Ask who will perform each part, then interview those people directly. A senior expert who appears during sales but disappears after signature is not the delivery team.
The vendor’s background should match the work. Rohan Hall has built AI and blockchain systems, led technology for blockchain interoperability and scalable applications, and served as Chief Technology Officer at RocketFuel Blockchain, where he led technology strategy, architecture and a distributed global engineering team. His professional technology career began in 1984, and he has worked extensively across the United States, Europe and Asia. Those specifics are more useful than an unsupported label such as “AI visionary.” An overview of his work, ventures, book and podcast is available on Rohan Hall’s home page.
Domain context also matters. A manufacturing system must contend with physical processes, operational reliability and the consequences of bad predictions. A trustworthy evaluation therefore examines both technical depth and understanding of the operating environment. Manufacturing leaders can use the dedicated guide to finding a trustworthy AI partner for manufacturing. For a broader warning-sign checklist, see how to avoid another failed custom AI engagement.
Use a paid pilot to control financial and technical risk
Once the vendor passes initial diligence, do not move directly into an open-ended build. Commission a limited paid pilot tied to the most important technical uncertainty. The pilot should be large enough to reveal whether the approach works, but bounded enough that failure produces useful evidence rather than another uncontrolled loss.
- Define the problem, inputs, outputs and exclusions in writing.
- Set acceptance criteria and the evaluation method before development begins.
- Identify the assumptions the pilot is designed to test.
- Specify tangible deliverables, including working software, evaluation results and architecture documentation.
- Use milestones connected to observable evidence rather than elapsed time or presentation activity.
- Set a decision point: stop, revise, extend or proceed to production.
- Clarify ownership, data handling, third-party dependencies and what can be transferred if the relationship ends.
Do not let a pilot quietly become production. Production requires a separate decision about reliability, security, monitoring, support and organizational readiness. A promising test is evidence for the next decision, not permission to skip it.
Budgeting should follow risk reduction. Spend first on problem definition, data inspection and the smallest meaningful test. Increase commitment only when evidence improves. If you are planning a first engagement rather than recovering from one, use the practical guide to setting a small-business AI project budget.
Where Rohan Hall and OceSha Ventures fit
Rohan Hall is the founder and CEO of OceSha Ventures’ AI-first solutions business. OceSha Ventures builds and operates course creation, branded academies, AI assistants such as Lumi, and business intelligence for businesses and organizations. Lumi’s work includes conversational websites, customer inquiry handling and customer intelligence grounded in information the business has confirmed.
This experience spans AI systems, blockchain systems, scalable architecture and emerging technologies. Rohan previously built and led technology for blockchain interoperability and has worked on blockchain-based supply-chain traceability, verifiable credentials, decentralized identity and W3C Self-Sovereign Identity concepts. He also advised on emerging technologies at Capital Group/American Funds.
For readers who want the larger technological context, Rohan covers AI alongside blockchain, neuromorphic technologies, cognitive intelligence and other emerging fields in The Convergence of AI and the Top 10 Emerging Technologies. The practical lesson is that emerging technology should be understood as an architecture and operating capability—not purchased as a fashionable label.
Reduce the cognitive burden by using one written scorecard in every meeting. Score problem understanding, architecture clarity, relevant evidence, test performance, delivery capability, commercial transparency and operational readiness. The guide to cutting through AI vendor fatigue offers a problem-first way to focus on systems that work rather than repetitive claims.
Use a written scorecard, representative test data and a bounded paid pilot to establish whether the vendor has the architecture, people and operating capability to deliver.
Evaluate the claim before funding the buildFrequently asked questions
Should every AI project include custom model training?
No. The right architecture may use an existing model, retrieval from approved information, deterministic rules, conventional software or a combination. Custom training is justified only when the problem, data and evaluation show that it adds necessary value.
Is using a third-party AI model a warning sign?
Not by itself. Ask the vendor to disclose the dependency, explain what its own team is building around it and address data handling, evaluation, operational reliability and the consequences of provider changes.
How can a nontechnical executive evaluate an AI architecture?
Require a diagram and plain-language explanation of inputs, decisions, models, rules, human steps, external services and outputs. Then have an independent technical reviewer test whether the explanation matches the implementation and contract.
What should happen if the vendor refuses to reveal proprietary details?
The vendor does not need to surrender confidential source code or methods. It should still explain system boundaries, dependencies, evaluation, risks, delivery responsibilities and how performance will be demonstrated. Refusal to provide those basics is a reason to stop.
What is the strongest single red flag during procurement?
A request for a large commitment before the vendor has agreed to measurable acceptance criteria or demonstrated the central technical assumption with representative data.
Can a rules-based solution still be the right answer?
Yes. If the workflow is stable, explicit and predictable, rules may be cheaper, easier to audit and more reliable than machine learning. The vendor should label the approach accurately and price it according to the engineering and business value actually delivered.
The bottom line
The safest response to a costly failed AI project is not to avoid AI; it is to change the burden of proof. Define the job, demand architectural clarity, test unfamiliar cases and release money in stages. Open-source components, rules and external models are all legitimate when disclosed and appropriately engineered. What you should reject is ambiguity: vague intelligence claims, rehearsed demonstrations, inaccessible delivery teams and contracts that require major commitment before technical risk has been reduced.
Rohan Hall
Founder of OceSha Ventures · AI architect and author
Rohan Hall is a technology entrepreneur, AI architect and author with years of technology experience, now focused on practical AI across business, education, government and global impact. He founded OceSha Ventures, builds the OceSha AI platform and Lumi, and wrote The Convergence of AI and the Top 10 Emerging Technologies.
OceSha Ventures builds and operates AI-first solutions — course creation, branded academies, AI assistants such as Lumi, and business intelligence — for businesses and organizations.
Sources
About this page. Last reviewed .
It is based on his verified public professional record.
db102b5c2f1890435c732226d5326544