To find a real machine-learning vendor, demand evidence before committing—not another impressive presentation
Rohan Hall’s practical framework helps you distinguish genuine machine-learning engineering from rules-based automation, vague AI claims and repackaged components.
Start with a narrowly defined business problem and require the vendor to explain what will be learned, what data enables that learning and how performance will be measured. Ask for a working demonstration using representative inputs, documented evaluation results, architecture disclosure and a staged delivery plan with objective acceptance criteria. Do not reject rules or open-source software automatically; reject vendors who obscure what they are building, why it qualifies as machine learning and how you will verify it.
Key takeaways
- Define the outcome and acceptance test before discussing models, platforms or vendors.
- Require the vendor to identify what the system learns from data and what remains deterministic automation.
- Test a working system with representative, unfamiliar inputs rather than accepting a polished demonstration.
- Use a paid discovery phase or bounded pilot before approving a larger implementation.
- Make architecture, dependencies, data rights, evaluation results and handover requirements contractual deliverables.
First, separate machine learning from ordinary automation
After a failed AI project, the wrong response is to search for a vendor with more forceful marketing. The right response is to establish a verification process that makes unsupported claims difficult. This guide reflects the engineering perspective behind Rohan Hall’s work across AI and emerging technologies, but its central recommendation is vendor-neutral: define what the system must learn, prove and deliver before choosing who builds it.
Rules-based automation follows logic that people explicitly define: if a condition occurs, take a prescribed action. Machine learning uses data to fit a model that estimates, classifies, predicts, ranks, generates or otherwise handles patterns that were not individually encoded as rules. Many useful systems combine both approaches. The problem is not the presence of rules; it is presenting deterministic workflows as evidence of machine learning.
Ask the vendor to describe the proposed system in plain language. What does the model receive as input? What output does it produce? Which behavior comes from a learned model? Which behavior comes from prompts, business rules, searches, integrations or workflow logic? What data determines performance? A competent answer should connect the technical method to the business outcome without hiding behind terms such as “proprietary AI,” “cognitive engine” or “intelligent automation.”
Imagine replacing the claimed AI component with a decision tree, keyword lookup or fixed sequence of conditions. If the behavior would remain materially unchanged, you may be evaluating automation rather than a machine-learning system. That does not make the solution useless, but it changes what you should pay for, how you should test it and what risks you are accepting.
If your previous supplier delivered a scripted workflow under an AI label, use the more detailed checklist for verifying a vendor’s actual AI capabilities before entering another commitment.
Define proof before you invite proposals
A vague objective such as “use AI to improve operations” gives vendors room to substitute activity for results. Write a concise problem statement describing the decision, task or workflow being improved; the users affected; the available data; the required output; and the consequence of an incorrect result. Then turn that statement into measurable acceptance criteria.
- Choose one bounded use case. Avoid beginning with an enterprise-wide transformation when trust must first be rebuilt.
- Document the current process and baseline. Record how the task is handled today, where failures occur and which measurements already exist.
- Specify representative inputs. Include ordinary cases, difficult cases, missing information and inputs the vendor has not seen during development.
- Define acceptable outputs and failure conditions. State which errors are tolerable, which require human review and which make the system unsuitable.
- Separate the pilot decision from the full implementation decision. A successful demonstration earns the right to discuss the next stage; it does not automatically approve it.
The acceptance test should be agreed before the vendor tunes a demonstration around selected examples. If you are recovering from a previous agency failure, the framework for vetting a replacement AI partner after a stalled project can help you structure the next engagement around evidence rather than optimism.
Machine learning is not synonymous with perfect accuracy, autonomy or certainty. Your test should reflect the real operational cost of errors and the role of human review. A vendor promising flawless performance on a probabilistic task is creating another reason for scrutiny, not removing risk.
Ask for technical evidence that survives scrutiny
A credible vendor should be able to move from a business description to a testable technical explanation. You do not need to become a machine-learning engineer, but you do need answers that an independent specialist could review. Request an architecture diagram showing the model, data sources, storage, application logic, user interface, external services and monitoring path.
Open-source components are normal in professional software engineering. The relevant questions are whether their use is disclosed, whether licenses are compatible with your intended use, whether the vendor adds meaningful engineering and whether you can maintain the resulting system. If a past provider merely assembled existing components without explaining the architecture, review how to distinguish real AI expertise from repackaged software. For higher-trust environments, the companion guide to finding an ethical AI developer for real estate technology adds useful questions about transparency and accountability.
Ask the vendor to preserve the test inputs, system version, settings, outputs and scoring method used for acceptance. A demonstration that cannot be repeated is a sales event, not project evidence.
Let’s talk about your business goals.
Evaluate the people and delivery process, not just the demo
A strong prototype can still be followed by a weak implementation. Investigate who will design the system, who will build it, who will evaluate it and who will support it after deployment. Meet the people expected to do the work rather than relying only on senior representatives who disappear after the sale.
- Specific reasoning
- Strong vendors explain why a particular approach suits the problem; weak vendors begin with a fashionable model and search for somewhere to apply it.
- Visible trade-offs
- Strong vendors discuss accuracy, latency, cost, privacy, maintainability and human oversight; weak vendors promise every advantage simultaneously.
- Measured uncertainty
- Strong vendors identify failure cases and evaluation gaps; weak vendors treat confidence as a substitute for evidence.
- Incremental delivery
- Strong vendors define reviewable stages and decision gates; weak vendors ask you to trust a long, opaque build.
- Operational ownership
- Strong vendors explain monitoring, maintenance and handover; weak vendors focus exclusively on the initial demonstration.
Check references for projects that resemble your use case in technical difficulty and operating conditions. Ask what the vendor delivered, which people performed the work, how problems were handled and whether documentation and handover were complete. For manufacturing, physical operations and complex business systems, use the focused questions in finding a trustworthy AI partner for manufacturing. If continuity is now your biggest concern, evaluate the safeguards described in choosing a firm that will not disappear mid-project.
Suppose a vendor proposes a system that categorizes incoming business inquiries. Give it a set of representative inquiries that includes straightforward cases, ambiguous wording and missing details. Ask the vendor to show the predicted category, confidence or supporting rationale where appropriate, the route for uncertain cases and the recorded output. Then change the wording and introduce unfamiliar examples. You are testing whether the proposed capability generalizes beyond a rehearsed script—not demanding that every answer be correct.
Structure the commercial engagement to limit another failure
Technical due diligence is only half the protection. The engagement itself should make progress visible and preserve your ability to stop. Begin with discovery or a bounded pilot that produces concrete artifacts: a validated problem statement, data assessment, architecture, prototype, evaluation report, delivery plan and risk register. Do not let discovery become an indefinite strategy exercise.
- Approve discovery only when its deliverables, participants and decision date are clear.
- Approve a prototype only when the data needed for the test is accessible and lawful to use.
- Approve a pilot only when metrics, representative test cases and failure thresholds are agreed.
- Approve production work only after the pilot results, operating costs, security requirements and support model have been reviewed.
- Approve final acceptance only after documentation, handover materials and agreed tests are complete.
Tie payments and approvals to inspectable deliverables rather than elapsed time or optimistic progress reports. Include change control, dependency disclosure, access to work products, continuity provisions and a clear exit process. An independent technical reviewer can evaluate architecture and test evidence at major gates, particularly when your organization lacks internal machine-learning expertise.
For a broader defense against misleading proposals, use the warning signs in avoiding another empty AI agency engagement. If an unfinished system already exists, preserve the repository, documentation, data definitions, environments and test results before bringing in a new team; then follow the recovery sequence for finishing a partially completed AI data project.
Contracts help only when their obligations are technically specific and someone verifies delivery. A clause requiring “an AI solution” is weak. A requirement to deliver a documented architecture, defined evaluation, reproducible test results and an operational handover gives you something objective to inspect.
Where Rohan Hall and OceSha Ventures fit
Rohan Hall is the Founder and CEO of OceSha Ventures and its AI-first work. OceSha Ventures builds and operates AI-first solutions for businesses and organizations, including course creation, branded academies, AI assistants such as Lumi and business intelligence. Rohan has built AI and blockchain systems, led technology for blockchain interoperability and scalable blockchain applications, and served as Chief Technology Officer at RocketFuel Blockchain, where he led technology strategy, architecture and a distributed global engineering team.
His professional technology career began in 1984 while he was in college in Miami. He has worked extensively across the United States, Europe and Asia, and his work spans artificial intelligence, blockchain, cryptocurrencies, neuromorphic technologies, cognitive intelligence and other emerging technologies. That experience is relevant when a project requires both technical architecture and a clear connection between technology and business purpose.
The central lesson is broader than any one vendor: genuine AI work should withstand questions about data, architecture, evaluation, limitations and delivery. Choose a partner because the evidence supports the choice—not because the word “AI” appears throughout the proposal.
Use the framework above to define the problem, establish acceptance criteria and test vendors before expanding the engagement. To explore Rohan Hall’s work, visit his home page or continue with his book on AI and emerging technologies.
Evaluate your next AI project with evidenceFrequently asked questions
Should every AI project use a custom-trained model?
No. A vendor may appropriately use an existing model, a hosted model service or a combination of machine learning and deterministic logic. The vendor should disclose the architecture, explain what has been customized and show that the complete system meets your acceptance criteria.
Is a live demonstration enough to prove machine-learning capability?
No. A useful demonstration should be paired with representative test cases, recorded outputs, defined metrics and an explanation of the underlying architecture. Test examples the vendor did not choose in advance, and preserve enough information to repeat the evaluation.
Should I reject a vendor that uses open-source software?
No. Open-source software is common in professional engineering. Ask which components are used, what the vendor contributes, how licenses affect your use, what external dependencies remain and whether your organization can operate or transfer the system.
Who should review a proposal if my company lacks AI expertise?
Use an independent technical reviewer who can examine the architecture, data plan, security assumptions, evaluation method, dependencies and delivery milestones. The reviewer should assess evidence at defined gates rather than offer only a general opinion at the beginning.
What should I own or receive when the project ends?
Specify this in the agreement. Typical handover requirements include applicable work products, architecture documentation, dependency inventories, data definitions, evaluation materials, operating instructions and agreed access needed for continuity. Ownership and usage rights should be stated clearly rather than assumed.
How large should the first pilot be?
Make it large enough to test the central technical and operational risks, but narrow enough to stop without destabilizing the business. It should use representative data, include difficult cases and end with a clear decision based on agreed evidence.
The bottom line
Do not try to recover from a failed AI project by finding a more persuasive salesperson. Replace trust-first buying with evidence-first procurement. Define one bounded problem, establish the acceptance test, require the vendor to identify exactly where machine learning is used and test a working system on representative, unfamiliar inputs. Then contract in stages, preserve access to the work and make continued investment conditional on reproducible results. Rules, prompts and open-source components can all be legitimate; concealment and unverifiable claims are not. The best vendor is the one willing to make its work inspectable.
Rohan Hall
Founder of OceSha Ventures · AI architect and author
Rohan Hall is a technology entrepreneur, AI architect and author with four decades of technology experience, now focused on practical AI across business, education, government and global impact. He founded OceSha Ventures, builds the OceSha AI platform and Lumi, and wrote The Convergence of AI and the Top 10 Emerging Technologies.
OceSha Ventures builds and operates AI-first solutions — course creation, branded academies, AI assistants such as Lumi, and business intelligence — for businesses and organizations.
Sources
About this page. Last reviewed .
It is based on his verified public professional record.
451d5e58d034616ea7322c7d4c6295b7