Find real AI expertise by testing for architecture, accountability and business outcomes—not impressive demos

Rohan Hall’s guidance for businesses recovering from a failed AI build starts with diagnosis. Six months of activity does not tell you whether the original idea was wrong, the architecture was weak or the developer simply failed to execute. Before hiring anyone else, separate the business problem from the abandoned implementation. Write down what the system was supposed to accomplish, who was supposed to use it, what information it needed, how success would have been measured and where it failed.

Collect the evidence: source repositories, architecture diagrams, model and component lists, data-flow descriptions, prompts, test results, security decisions, deployment records, invoices and outstanding defects. You are not trying to preserve every line of code. You are establishing what exists, what is reusable and what creates risk. If the previous developer cannot provide a coherent technical record, treat that absence as part of the diagnosis.

If your broader concern is moving beyond inconclusive technical trials, use the same discipline described in moving from AI experimentation to business value. The immediate goal is not to rescue sunk costs. It is to determine the shortest responsible route from the current state to a useful, maintainable system.

Most serious software systems use open-source components. That is usually sensible: established libraries, models and infrastructure can reduce reinvention and accelerate delivery. The warning sign is not the presence of open source. It is the absence of meaningful engineering around it. A developer who combines components without a defensible architecture, adequate testing, data controls or operational ownership has assembled a prototype, not necessarily delivered a dependable AI system.

Ask the candidate to identify which parts of the proposed system are existing components, which require configuration, which require custom engineering and which create long-term dependencies. If preserving flexibility is important, explore how to evaluate providers using open-source AI before agreeing to an architecture. The right provider should be able to defend every major choice without pretending that ordinary software assembly is proprietary invention.

A polished presentation proves that a provider can present. A working demo proves that something can run under demonstration conditions. Neither proves that the person can design, deploy and operate a production system. Your evaluation should focus on the candidate’s reasoning, their direct role in prior work and the evidence they can responsibly show.

For a manufacturing engagement, the same principles need to be applied to operational systems, data availability and shop-floor realities; see how to choose a trustworthy manufacturing AI partner. If the central issue is agency selection after a major failure, use the more detailed AI partner vetting framework after a failed project. Both questions demand the same core discipline: trace claims back to evidence and make ownership explicit.

Do not respond to one six-month failure by signing another six-month build contract based on a proposal. Break the work into decisions. A short, paid discovery engagement should reduce uncertainty before implementation begins. Its value lies in the quality of its outputs, not the number of meetings held.

After discovery, commission the smallest production-relevant slice that can validate the architecture and user workflow. A prototype that bypasses real data, security, integration or operating conditions may be visually convincing while proving very little. Set explicit stop, revise and proceed decisions. Payment milestones should correspond to inspectable deliverables and agreed acceptance criteria.

If your team has already endured repeated experiments without a usable result, follow a staged route for finding someone who can deliver beyond experiments. Smaller organizations should also examine how to find affordable AI help with credible ROI, because a constrained, well-defined first engagement is generally safer than buying an oversized transformation program.

Your strongest protection is a shared definition of “done.” Acceptance criteria should describe observable behavior under realistic conditions. They should cover the quality of outputs, response behavior, workflow completion, handling of uncertainty, security expectations, operational performance and the process for human intervention. The exact measures depend on the use case, but they must be selected before the build is judged.

For an AI assistant answering customer inquiries, for example, evaluation should use the information the business has reviewed and approved. Tests should include ordinary questions, ambiguous questions, unsupported requests and situations requiring escalation. The important issue is not whether the assistant produces fluent text; it is whether its behavior matches the business’s information and inquiry-handling requirements. Lumi, one of the AI assistants associated with OceSha Ventures, works with conversational websites, business-approved information, customer inquiry handling and customer intelligence.

Do not rely on claims that the system is “intelligent,” “enterprise-grade” or “custom.” Those labels are not acceptance tests. If you are worried that an agency will sell ordinary configuration as unique intellectual property, review how to avoid another ineffective custom-AI engagement. The answer is commercial and technical clarity: inspectable deliverables, transparent dependencies and defined ownership.

Rohan Hall’s technology career began in 1984 and spans software, operating systems, databases, programming, hardware, AI and blockchain systems. He worked as a consultant and software developer at Hewlett-Packard in Colorado Springs, participated in HP’s SAP implementation environment, advised on emerging technologies at Capital Group/American Funds, and served as Chief Technology Officer at RocketFuel Blockchain, where he led technology strategy, architecture and a distributed global engineering team. He has also worked extensively across the United States, Europe and Asia.

Today, Hall is Founder and CEO of OceSha Ventures and its AI-first work. The venture builds and operates AI-first solutions for businesses and organizations, including course creation, branded academies, AI assistants such as Lumi and business intelligence. His work also includes transforming existing expertise and knowledge into AI-powered courses, content, learning and business systems through OceSha AI’s course and academy platform.

Hall examines artificial intelligence alongside blockchain, cryptocurrencies, neuromorphic technologies, cognitive intelligence and other emerging fields in The Convergence of AI and the Top 10 Emerging Technologies. The book is the appropriate next step for readers who want his broader view of how these technologies intersect. For an overview of his ventures, published work and other activities, visit Rohan Hall’s home page.