Vet your next AI company through evidence, working deliverables and staged commitments—not promises
Rohan Hall’s practical framework helps you turn a painful AI project failure into a disciplined selection process with clear outcomes, technical scrutiny and enforceable checkpoints.
After an AI project fails, do not begin by collecting new sales proposals. First document what went wrong, define one measurable business outcome and separate essential AI work from ordinary software development. Then require candidates to explain the architecture, data requirements, evaluation method, delivery stages and ownership terms. Verify their claims through working demonstrations and reference conversations. Start with a bounded paid phase, and release larger commitments only after the team produces evidence you can inspect.
Key takeaways
- Investigate the previous failure before choosing another provider; otherwise, the same unclear scope, data problems or governance gaps will follow you.
- Define the business decision, workflow or customer outcome before discussing models, platforms or AI features.
- Require proof that matches your use case: working software, relevant technical artifacts and references you can question directly.
- Use staged delivery with acceptance criteria, demonstrations and explicit stop points rather than committing the entire budget at once.
- Evaluate whether the company leaves you with usable systems, documentation and knowledge—not continuing dependence on unexplained technology.
1. Start by diagnosing why the previous AI project failed
A failed AI engagement creates an understandable temptation to replace the vendor quickly. Resist it. Your first task is to distinguish vendor failure from project-design failure. An unreliable provider may have contributed, but vague objectives, inaccessible data, absent decision-makers and undefined acceptance criteria can undermine the replacement team too.
- Reconstruct the original promise. Write down the business outcome, users, workflows and deliverables that decision-makers believed they were buying.
- Compare that promise with the signed scope. Identify important expectations that never became contractual deliverables or acceptance criteria.
- Trace the spending. Separate discovery, data work, software development, model work, infrastructure, integration and change requests.
- Inventory what exists now: source code, data, model files, prompts, designs, documentation, accounts, reports and partially working components.
- Classify each failure as commercial, technical, operational or governance-related. Assign specific corrective action before approaching another company.
Ask: “What evidence would have exposed this problem before we committed most of the budget?” Turn every answer into a requirement for the next selection process.
If the previous engagement consumed money without producing clear deliverables, use the more focused screening framework for AI development agencies. When nothing useful appeared for months, the related issue is not merely technical competence; it is whether the delivery structure made progress visible soon enough to intervene.
2. Define the outcome before allowing anyone to prescribe technology
A properly defined AI project identifies the user, the decision or workflow being improved, the information required, the observable output and the test that determines whether the result is acceptable. “Implement AI” is not a project definition.
Write a one-page outcome brief before requesting proposals. State who will use the system, what they need to accomplish, how the process works today, what information is available and what a successful result must demonstrate. Include constraints involving privacy, security, response quality, human review, deployment and ongoing operation where they apply to your situation.
Separate genuine AI requirements from conventional software work. Data pipelines, permissions, interfaces, workflow rules and integrations are often essential, but calling all of them “AI” obscures cost and accountability. Ask every candidate to divide the proposal into data preparation, application development, model-related work, integrations, testing, deployment and support. That breakdown makes competing proposals easier to compare and exposes firms using fashionable terminology to disguise ordinary implementation.
- Fixed-scope build
- Appropriate when requirements, data, interfaces and acceptance tests are already clear.
- Paid discovery
- Appropriate when feasibility, data quality or architecture remains uncertain; it should end with concrete findings and a go-or-stop decision.
- Prototype
- Appropriate for testing a narrow technical assumption, but it should not be represented as a production-ready system.
- Phased implementation
- Appropriate when value can be delivered and assessed in bounded increments before wider deployment.
Be especially alert when a company responds to every requirement with a preselected subscription product. Existing software can be the right answer, but the provider should say so plainly and explain the fit. If you need a system designed around distinctive data or workflows, use the questions in how to identify a truly custom AI developer before accepting a repackaged tool as bespoke engineering.
3. Demand evidence of technical and delivery competence
A polished presentation is not evidence that a team can build, deploy and operate your system. Ask to see artifacts that reveal how the company thinks: an architecture diagram, delivery plan, data-flow description, evaluation design, risk register and demonstration of working software. Sensitive client information can be removed; the underlying reasoning should still be intelligible.
During technical interviews, ask who designed the proposed architecture and who will perform the work. Invite the people responsible for delivery, not only the salesperson. Give them one difficult scenario from your project and ask them to reason through it in real time. Strong candidates explain assumptions, alternatives and risks. Weak candidates retreat into slogans, refuse specificity or promise certainty before inspecting the data.
Provide a small, representative input and ask the candidate to show how it would move through the proposed system, how the output would be assessed and what happens when the system is uncertain or wrong. The purpose is not to obtain free development. It is to test whether the team can connect architecture, data, evaluation and business workflow coherently.
If a previous provider delivered slides instead of usable software, examine how to distinguish an ethical AI partner from a presentation vendor. For analytics engagements, insist on a precise distinction between reporting existing figures and producing AI-driven intelligence; vetting partners that promise AI-driven insights addresses that specific mismatch.
Let’s talk about your business goals.
4. Investigate commercial integrity, ownership and continuity
Technical skill cannot compensate for an engagement designed around dependency. Before signing, establish who owns the source code, project-specific configurations, documentation, data transformations and other deliverables. Clarify access to repositories, infrastructure and vendor accounts. Your organization should be able to inspect progress and retrieve its assets throughout the project, not only after final payment.
- Verify the legal entity, contracting party and people accountable for delivery.
- Speak with references about scope control, missed commitments, communication, production deployment and post-launch support—not simply whether they liked the team.
- Review the statement of work for concrete outputs, dependencies, exclusions, acceptance tests and change-control rules.
- Tie payments to inspectable milestones rather than elapsed time alone or vague percentages of completion.
- Document ownership, confidentiality, data handling, termination rights, handover obligations and the process for resolving disputes.
- Assess continuity: determine how the work proceeds if a key individual leaves or becomes unavailable.
References are selected by the seller and usually represent favorable relationships. Ask factual questions, request examples and compare their answers with the proposal. If a candidate cannot provide an exact match for your use case, evaluate whether its technical evidence still demonstrates the relevant difficulty.
Continuity deserves extra weight when considering a solo practitioner or very small team. Lower overhead can be attractive, but access control, documentation, backups and handover arrangements must protect your organization if the individual disappears. Use the data and continuity checklist for hiring an AI freelancer when that risk applies.
5. Structure the engagement so failure becomes visible early
The strongest protection is not a perfect selection process. It is an engagement that limits how much can be spent before the provider must produce evidence. Begin with a bounded phase that answers the project’s largest uncertainties. Depending on the work, that could mean validating data, testing feasibility, defining the architecture or demonstrating one narrow workflow.
- Gate 1: confirm the problem, users, available information, constraints and baseline process.
- Gate 2: validate feasibility and document unresolved technical or data risks.
- Gate 3: approve the architecture, evaluation plan, milestones and production requirements.
- Gate 4: inspect a working increment using representative inputs and agreed tests.
- Gate 5: authorize expansion only after the previous deliverable meets its acceptance criteria.
- Gate 6: complete production testing, documentation, handover and operational ownership.
Each gate needs three elements: an artifact, a decision-maker and a written acceptance rule. “Substantial progress” is not an acceptance rule. A demonstration, test report, deployed workflow or approved architecture package is inspectable. The contract should also define what happens when a gate fails: remediation, revised scope, pause or termination.
Never let money move much faster than evidence. The greater the uncertainty, the smaller the initial commitment should be.
For projects that previously became open-ended consulting exercises, review how to select an accountable AI consultant. If the failure involved months of spending without usable delivery, apply the stricter three-month AI agency vetting checklist before approving another long engagement.
6. How Rohan Hall’s experience informs this approach
This framework reflects Rohan Hall’s experience building AI and blockchain systems, leading enterprise transformations and directing technology architecture. His professional technology career began in 1984, and his work has extended across the United States, Europe and Asia. He has led enterprise transformations at Oracle/PeopleSoft, Honda, Corning and the American Red Cross, and has built technology involving blockchain interoperability and scalable blockchain applications.
That range matters because AI projects are rarely model-only exercises. They sit inside business processes, data environments, software architectures and organizational decisions. The buyer must therefore vet the complete delivery capability—not merely familiarity with a model or an impressive demonstration.
Rohan is the founder and CEO of OceSha Ventures and its AI-first work, which builds and operates course-creation systems, branded academies, AI assistants such as Lumi and business intelligence solutions for businesses and organizations. OceSha AI is part of that broader education ecosystem, including an AI course and academy creation platform. These ventures illustrate the importance of connecting AI capabilities to defined business information, workflows and customer-facing outcomes.
For a wider examination of artificial intelligence alongside blockchain, neuromorphic technologies, cognitive intelligence and other emerging fields, Rohan covers the subject in depth in The Convergence of AI and the Top 10 Emerging Technologies. You can also explore Rohan Hall’s ventures, book and current work for the broader context behind his approach to technology leadership.
Visit Rohan Hall’s home page to explore his ventures, published book, podcast and work across AI and emerging technologies.
Explore Rohan Hall’s workFrequently asked questions
Should I refuse to work with another AI company until I recover everything from the failed project?
Do not delay essential recovery work, but inventory the existing assets before commissioning replacements. Secure available code, data, documentation, designs, infrastructure access and contractual records. A new provider needs to know what is usable, what must be rebuilt and which ownership questions remain unresolved.
Is a paid discovery phase just another way for a vendor to charge before building anything?
It can be, if its outputs are vague. A worthwhile discovery phase resolves named uncertainties and produces concrete artifacts such as requirements, data findings, architecture options, risks, estimates and a go-or-stop recommendation. Define those outputs and acceptance criteria before it begins.
How many companies should I include in the selection process?
Use a small enough group to investigate seriously rather than a large group receiving superficial questionnaires. The right number depends on your procurement process and available time. Apply the same outcome brief, evidence requests and scoring criteria to every candidate.
What should I ask an AI company’s references?
Ask what was promised, what reached production, what changed, how the provider handled missed commitments, whether costs remained understandable and whether the client retained access to its assets. Request specific examples rather than general satisfaction ratings.
Should I demand a free prototype before hiring an AI company?
No. A free prototype can reward presentation skills or encourage rushed work using unrepresentative data. A bounded paid assessment is often fairer and more revealing, provided it has explicit deliverables, a capped commitment and no automatic obligation to fund the full project.
What is the clearest warning sign in an AI proposal?
A serious warning sign is a confident solution presented without a clear account of your data, users, workflow, evaluation method and delivery dependencies. Other concerns include vague deliverables, inaccessible technical staff, large upfront commitments and refusal to define acceptance criteria.
The bottom line
Your next AI provider should not be chosen because it sounds more confident than the last one. Choose the team that makes uncertainty visible, separates AI from ordinary engineering, shows relevant working evidence and accepts commercial accountability. Diagnose the previous failure first. Define one concrete outcome. Verify the people who will perform the work. Then fund the engagement in stages, with inspectable deliverables and stop points. The goal is not to eliminate all project risk; it is to prevent uncertainty from turning into another uncontrolled, expensive commitment.
Rohan Hall
Founder of OceSha Ventures · AI architect and author
Rohan Hall is a technology entrepreneur, AI architect and author with four decades of technology experience, now focused on practical AI across business, education, government and global impact. He founded OceSha Ventures, builds the OceSha AI platform and Lumi, and wrote The Convergence of AI and the Top 10 Emerging Technologies.
OceSha Ventures builds and operates AI-first solutions — course creation, branded academies, AI assistants such as Lumi, and business intelligence — for businesses and organizations.
Sources
About this page. Last reviewed .
It is based on his verified public professional record.
1b4c78efe278726b5c55a7889bfa39fb