Go back to resources

A clinical risk framework for voice AI: Our taskforce’s first paper

A roadside gate appears on a blue background.

The Healthcare Risk Management for Voice AI Taskforce has published its first white paper, and it tackles a question existing AI governance frameworks don’t yet answer: How should healthcare organizations assess clinical risk in voice agents that talk to patients directly and act on what they hear?

The paper proposes a four-layer risk framework, covering the agent, the data it touches, the patient it serves, and the clinician who has to trust it. It grounds that framework in a formal consensus study: 14 experts across clinical medicine, health systems, payors, patient advocates, and technology builders rated risk across five real-world use cases.

What stands out most is how differently that risk shows up across use cases. A post-stroke care coordination agent, confirming medication fills and screening for red-flag neurological symptoms, is working with a population that frequently experiences aphasia or cognitive impairment, which can degrade transcription accuracy and clinical signal at the exact moment a recurrent stroke needs to be caught. A post-surgical follow-up agent, assessing pain control, wound healing, and diet tolerance seventy-two hours after discharge, carries a different failure mode entirely: misjudging acuity and routing a patient away from an urgent callback they actually need. Same underlying technology, two very different ways for it to fail a patient.

The result is a practical starting point for any organization deploying patient-facing voice AI: where risk actually enters the pipeline, and what to do about it before an agent ever speaks with a patient.

This white paper is based on two manuscripts currently under review at JMIR: “Safety of Patient-Facing Agentic AI: a Consensus Framework for Risk Assessment and Mitigation” and “Safety Risk Prioritization for Patient-Facing Agentic Voice AI in Clinical Care: A Two-Round Modified Delphi Study.” Both are available as preprints.

You can read the paper below, or click here to open it in a new window to save for later.

Loading PDF…

Download agentic-safety-white-paper (PDF)

The latest from Infinitus