As more companies move from AI experimentation into deployment, a new challenge is emerging: operational reliability.

On stage at Asembia’s ASX26, Infinitus CEO and Co-founder Ankit Jain and Optum Rx CIO Santiago Abraham discussed one of the biggest gaps in healthcare AI today: the difference between a compelling demo and a system that can actually operate consistently inside real healthcare workflows.

Their conversation covered governance, hallucinations, AI evaluation, pilot fatigue, and why healthcare organizations need to think differently about operationalizing AI.

Below is the second in a series of edited excerpts from their conversation. You can read Part 1 here, Part 3 here or watch the full conversation here.

Ankit Jain: How are you thinking about measuring risk when evaluating AI platforms and use cases?

Santiago Abraham: Governance matters in the healthcare space probably more so than in many other industries. My comment there is, if the shopper agent makes the mistake of recommending the wrong thing to you, you added the wrong thing to the cart. If we make a mistake in healthcare with AI agents, there are big stakes at hand.

So really, quality is extraordinarily important in this space. This is an area where, candidly, there’s a ton of creativity. There’s an abundance of opportunities. We have hundreds of use cases already in motion within Optum and so we’re bullish there.

But ultimately the really critical step is: is governance appropriately in place? Do you have the guardrails in place? Do you have responsible use? Do you understand what that looks like?

And then secondly, from a quality perspective, are you managing the quality?

Ankit Jain: There’s also a lot of pressure right now to move quickly. Organizations are launching pilots everywhere. But many companies seem stuck in what I’d call ‘pilot purgatory.’

Santiago Abraham: I think understanding that you’ve got to fail fast – that’s the thing.

If you go into pilot quicksand, that’s the problem. When you go into pilot mode and then you’re there forever – the governance team is saying you don’t have XYZ, you’ve got issues with your data, you don’t have the right RAG behind it, you’re seeing heightened levels of hallucination.

So I think it’s important that you pilot. But picking the use cases upfront is important.

Ankit Jain: One of the challenges a lot of organizations are facing right now is ‘pilot purgatory.’ Everybody wants to pilot AI, but getting from pilot to production is much harder. What have you seen work?

Santiago Abraham: Piloting is important because the AI stuff isn’t like deterministic tools where I want to build X, I spec out X, I build X, I deploy X.

With the AI space, you think an idea is going to work, you think you have the right data sources, you think you have the right SOPs or processes understood, and then you get into it and you may find stuff. 

So I would say it’s important to pilot, but you’ve got to fail fast. If you go into pilot quicksand, that’s the problem.

One of the criteria for picking the use case is understanding the opportunity at scale. Do you understand what the end game looks like if it is successful?

Ankit Jain: One of the trends we’ve seen at Infinitus is that organizations are increasingly asking for transparency into how AI systems make decisions – they want visibility into why systems behave the way they do. 

We recently announced Studio, where we might build an agent for a customer, but then give them a full view into what the agent does and how it makes decisions in a way that previously wasn’t possible. Are you seeing that drive toward transparency as well?

Santiago Abraham: Yeah, I would say so. I think explainability is important.

You have to be able to fine tune the tools and the models. And so the ability to be able to do that, to have that level of configuration, is key.

I would say that you want that with any partner – so great work there.

Ankit Jain: When people talk about AI, there’s often this assumption that autonomy is always the goal. But healthcare is different. How are you thinking about the balance between AI-assisted workflows and fully autonomous systems?

Santiago Abraham: For us, for example, care and care determination decisions have to be with a human at the center with AI assist.

I think every team, every business needs to look at the strategy around AI – what’s the right use case?

Human in the loop is interesting because that’s a model where today you work through an assistant capability, which is really helping you operate with more productivity, more speed, which is awesome.

Where this is going – which is where you’re headed next – is really around human on the loop. So now you’re getting into scenarios where you’ve got an autonomous. Think of it like a junior worker.

And that’s really the game changer in my mind.

This conversation has been lightly edited for length and clarity.