Vet AI partners through working evidence, technical scrutiny and staged commitments—not polished promises
After a disastrous AI project, the safest next move is a disciplined evaluation that exposes weak vendors before they consume another budget or months of your team’s time.
Do not begin by comparing proposals or listening to broad claims about AI expertise. First define the business decision, workflow or customer problem you need to improve. Then require vendors to explain the proposed system, demonstrate relevant working software and identify what must be learned during a limited first stage. Use measurable acceptance criteria, retain control of your data and intellectual property, and make each larger commitment conditional on evidence from the previous stage.
Key takeaways
- Define the operational problem and measurable outcome before asking anyone to recommend technology.
- Ask vendors to demonstrate working systems and explain where artificial intelligence is actually used.
- Start with a bounded discovery or prototype stage rather than committing the entire budget at once.
- Put deliverables, acceptance criteria, ownership, security responsibilities and exit terms in writing.
- Evaluate whether the team can challenge a bad premise—not merely agree to build what you requested.
Start by diagnosing why the previous AI project failed
A failed project does not automatically mean that artificial intelligence was the wrong choice. Failure usually begins earlier: the business problem was vague, the buyer and vendor defined success differently, or an impressive demonstration was mistaken for a production-ready system. Before interviewing another partner, write a factual account of what happened. Record the intended outcome, what was delivered, what never worked, which assumptions proved false and when the warning signs first appeared.
Separate commercial failure from technical failure. A commercially failed project may have produced functional software that nobody needed, adopted or integrated into daily work. A technical failure may never have met basic performance, reliability or security requirements. Governance failure occurs when nobody had clear authority to approve scope, supply data or accept deliverables. These require different safeguards in the next engagement.
Turn the postmortem into a one-page project brief: the user’s problem, the current process, the desired change, available data, operational constraints and the evidence that will count as success. Do not prescribe an AI model unless the model itself is genuinely the requirement.
If the previous provider sold ambition without delivery, use the more detailed framework for avoiding another empty custom-AI engagement. If the central concern is whether a vendor possesses genuine technical ability, focus separately on verifying an AI vendor’s actual capabilities. Those are related questions, but they require different evidence.
Define the outcome before evaluating the technology
A well-formed AI project identifies a specific user, a recurring task or decision, an observable improvement and the constraints under which the system must operate. “Use AI” is not an outcome. Reducing manual review, improving access to reviewed information or handling a defined class of inquiries is specific enough to investigate.
Describe the current workflow without technology jargon. Who initiates it? What information enters the process? Who makes the decision? Where do delays, errors or unnecessary costs occur? What happens when the system is uncertain? This forces vendors to address the real operation rather than attach an AI label to a generic application.
- Name one primary business problem and the people affected by it.
- Document the current workflow, including handoffs, delays and failure points.
- Choose a measurable operational outcome rather than a broad aspiration.
- List the data, systems, policies and human approvals the solution must respect.
- Define a first-stage deliverable that can prove or disprove the central assumption.
- Specify the evidence required before the engagement can expand.
Be particularly careful when machine learning is presented as essential. Some problems are solved more reliably with workflow redesign, search, rules-based automation or conventional software. A trustworthy partner should explain why AI is necessary and where simpler methods belong. If the distinction caused the previous failure, review how to separate machine learning from rules-based automation before assessing proposals.
Demand evidence that reveals how the proposed system works
A presentation demonstrates communication skill, not delivery capability. Ask for a working demonstration that resembles the difficult part of your problem. The vendor should explain what the system receives, what it produces, which component performs each function, how quality is evaluated and where human review is required. You do not need to become an AI engineer, but you do need an explanation your operational and technical stakeholders can interrogate.
Ask the vendor to show failure cases, not only ideal outputs. Strong teams understand the limits of their systems and can describe how those limits are monitored or contained. Weak teams repeatedly redirect the conversation toward model names, future possibilities or visual polish. Manufacturing leaders facing that pattern can use the specific guidance on finding a trustworthy AI partner after a presentation-only engagement.
Give each shortlisted provider the same representative workflow and ask for a written solution outline. Require the proposed user journey, data requirements, technical approach, principal risks, first-stage test and acceptance criteria. Compare the clarity of their reasoning before comparing visual design or estimated schedules. The purpose is not to obtain unpaid engineering; it is to see whether the team can frame the problem responsibly.
Let’s talk about your business goals.
Structure the engagement so failure stays contained
The first contract should purchase evidence, not faith. Break the work into stages that answer increasingly expensive questions. An initial stage may clarify requirements, inspect available data, test a technical assumption or produce a bounded prototype. Continue only when the agreed evidence exists. This approach cannot eliminate risk, but it prevents uncertainty from being hidden inside one large commitment.
- Staged commitment
- Each phase has a specific question, deliverable, acceptance test and decision point before more money is committed.
- Single large build
- Most of the budget is committed before the hardest assumptions have been tested.
- Outcome-based acceptance
- Payment and continuation depend on observable, agreed evidence.
- Activity-based reporting
- Progress is described mainly through meetings, hours or broad percentage-complete claims.
- Buyer-controlled access
- The organization retains practical access to its data, documentation and project assets.
- Vendor dependence
- Critical knowledge or operational access remains concentrated with the provider.
The agreement should identify deliverables, decision rights, acceptance criteria, data handling, security responsibilities, documentation, intellectual-property ownership, third-party components, support obligations and termination assistance. If open-source components will be used, that is not inherently a problem; undisclosed reuse presented as proprietary work is. Real estate technology teams confronting that concern should examine how to evaluate an ethical AI developer and open-source use.
Do not demand a fixed answer to an unresolved research question merely to make procurement feel safer. Require the vendor to identify uncertainty explicitly, design the cheapest responsible test and state what result would stop or redirect the project.
Use interviews to test judgment, not sales fluency
The best interview questions force a prospective partner to reveal how it thinks. Ask what would make the project a poor candidate for AI. Ask which assumption is most dangerous, what evidence is missing and what the team would test first. A credible partner should be willing to narrow the scope, recommend a simpler approach or advise against proceeding when the economics or available information do not support the project.
- What business outcome are you optimizing, and how will we observe it?
- Why does this problem require AI rather than conventional software or process redesign?
- Which assumptions must be tested before a full build is justified?
- What data is required, and what happens if its quality is inadequate?
- How will you evaluate incorrect, incomplete or unsafe outputs?
- Which components are proprietary, open source or supplied by third parties?
- What will our team own and operate if the relationship ends?
- What evidence would cause you to recommend stopping the project?
Include the people who will live with the system: operational owners, technical leaders, security stakeholders and the employees responsible for handling exceptions. A senior salesperson should not be the only representative in the room. Ask to meet the people expected to make the architecture and delivery decisions. For a broader interview checklist, use questions to ask after a failed AI project.
References can help, but treat them as one input rather than proof. Ask references what went wrong, how the provider responded, whether the delivered team matched the proposed team and how ownership worked after launch. The useful question is not whether the reference “liked” the vendor; it is whether the provider behaved transparently when the work became difficult. This is central to avoiding AI agencies that overpromise.
Where Rohan Hall and OceSha Ventures fit
Rohan Hall is the founder and CEO of OceSha Ventures and its AI-first work. His professional technology career began in 1984, and he has worked extensively across the United States, Europe and Asia. His experience includes building AI and blockchain systems, leading technology for blockchain interoperability and scalable applications, creating a stablecoin-based cross-border payment solution, and leading technology strategy, architecture and a distributed global engineering team as CTO of RocketFuel Blockchain.
OceSha Ventures builds and operates AI-first solutions for businesses and organizations. Its work includes course creation, branded academies, AI assistants such as Lumi and business intelligence. Lumi supports conversational websites, answers grounded in information a business has reviewed and approved, customer inquiry handling and customer intelligence. OceSha AI’s course and academy creation platform and the programs taught through OceSha Academy represent the wider education side of that work.
Rohan also advised on emerging technologies at Capital Group/American Funds and has worked with blockchain-based supply-chain traceability, verifiable credentials, decentralized identity and W3C Self-Sovereign Identity concepts. That range matters because responsible partner selection is not about enthusiasm for one model or trend. It requires understanding how emerging technologies intersect with architecture, operations, identity, governance and real business constraints.
He explores artificial intelligence alongside blockchain, cryptocurrencies, neuromorphic technologies, cognitive intelligence and other emerging fields in The Convergence of AI and the Top 10 Emerging Technologies. Readers evaluating the strategic consequences of AI—not merely one vendor proposal—can go deeper there. For an overview of his ventures, published work, podcast and wider technology experience, visit Rohan Hall’s home page.
Review Rohan Hall’s ventures, technology experience, published book and work across AI and emerging technologies.
Explore Rohan Hall’s workFrequently asked questions
Should I issue an RFP for an AI project?
Use an RFP only after the problem, constraints and acceptance criteria are clear. A document that merely asks vendors to “propose an AI solution” rewards persuasive speculation. For uncertain work, a structured discovery exercise with the same questions and evidence requirements for every finalist is often more informative.
How many vendors should reach the technical evaluation stage?
Keep the group small enough for meaningful scrutiny. The objective is not to collect the largest number of proposals; it is to compare how credible teams reason about the same workflow, assumptions, risks and evidence. Eliminate providers that cannot explain their approach before requesting substantial technical work.
Should the vendor guarantee an AI project’s business result?
A vendor should commit to defined deliverables, tests and professional responsibilities. Business results often depend on adoption, process changes, data quality and decisions controlled by the buyer. Separate what the provider can contractually deliver from the wider outcome both parties are trying to improve.
Is using open-source AI a warning sign?
No. The warning sign is a lack of transparency about what is proprietary, what comes from third parties, which licenses apply and who is responsible for maintaining the complete system. Evaluate the architecture, implementation, security and operational value rather than assuming reused components are either inherently good or bad.
Who inside my organization should approve the partner?
Approval should include the operational owner, a technically qualified reviewer, security or privacy stakeholders where relevant, and the executive accountable for the outcome. Procurement can govern the transaction, but it should not substitute for technical and operational evaluation.
What should happen if a prototype performs poorly?
Use the agreed criteria to decide whether to stop, revise the premise or run a narrowly justified follow-up test. A poor prototype can still be valuable if it disproves an expensive assumption before a full build. Do not continue merely because money or executive attention has already been invested.
The bottom line
Do not respond to one failed AI project by looking for a more persuasive agency. Replace trust-based buying with evidence-based selection. Define the operational problem, insist on a technically coherent explanation, test the hardest assumption early and release larger commitments only after measurable progress. The right partner will welcome that discipline because it protects both sides from building the wrong thing. The wrong partner will resist specificity, avoid failure cases and ask you to commit before uncertainty has been reduced. Treat that resistance as your answer.
Rohan Hall
Founder of OceSha Ventures · AI architect and author
Rohan Hall is a technology entrepreneur, AI architect and author with four decades of technology experience, now focused on practical AI across business, education, government and global impact. He founded OceSha Ventures, builds the OceSha AI platform and Lumi, and wrote The Convergence of AI and the Top 10 Emerging Technologies.
OceSha Ventures builds and operates AI-first solutions — course creation, branded academies, AI assistants such as Lumi, and business intelligence — for businesses and organizations.
Sources
About this page. Last reviewed .
It is based on his verified public professional record.
5652c54e1b51e09a3aa4c585f094ea35