When the algorithm hallucinates in the ICU, who’s watching?

Why observability is no longer a “nice-to-have” and is the foundation for responsible AI in healthcare.

The uncomfortable truth healthcare leaders are starting to admit

Walk into any major health system today and you’ll find the same paradox: Artificial Intelligence (AI) capabilities are being embedded into the Electronic Health Record (EHR), Microsoft 365, Workday, ServiceNow, and a growing roster of ancillary applications and research platforms. Yet almost no CIO can answer, with confidence, the three questions that matter most:

• What is every AI agent in my enterprise actually doing right now?
• Can I prove — to a regulator, a board, or a grieving family — exactly why did it make that decision?
• If it starts drifting, hallucinating, or quietly degrading, how fast will I know?

If those questions make you uncomfortable, they should. Because the era of “isolated AI assistants” is ending, and the era of distributed, agentic AI operating across clinical, administrative, and research domains has already begun. Forward-leaning organizations have started calling this what it is: an Enterprise AI Control Plane problem. And at the center of every credible control plane sits one capability that most health systems have significantly underinvested in — observability.

The new shape of risk: from one model for pilots to an orchestration of agents in production

Healthcare is moving from a world of standalone AI assistants to a broader ecosystem of platform-native and enterprise-managed AI agents spanning clinical, operational, administrative, and research workflows. Epic embeds its own agents. Microsoft embeds Copilots. Workday and ServiceNow are racing to do the same. Each vendor is innovating its own silo, and each silo is making decisions that touch patients, claims, schedules, and personal health information (PHI).

That creates a real governance gap that no single vendor can close on its own. Without a shared observability layer connecting those silos, AI capabilities proliferate independently across platforms, leading to fragmented workflows, inconsistent controls, and rising risk. Risk in healthcare isn’t a quarterly earnings miss. It’s a misrouted referral. A missed sepsis flag. A claims denial that violates parity laws. A research dataset contaminated by a drifting model. The blast radius is human.

Note: In closed healthcare platforms, visibility may be limited to the telemetry, APIs, audit records, and events exposed by the platform itself.

Observability enables governance — it isn’t governance itself

I want to be precise about this, because the distinction matters. Observability doesn’t govern anything on its own. What it does is supply the evidence and the context that governance depends on. You cannot enforce a policy you cannot see. You cannot escalate to a human if you don’t know if an agent is drifting. You cannot retire a model if you have no telemetry on its behavior.

This framing isn’t theoretical for me as a former health CIO. One of our healthcare customers published a strategic point of view describing nine capabilities of an Enterprise AI Control Plane: intent routing, policy coordination, identity delegation, observability, agent registry, human escalation, safety guardrails, model governance, and resilience. Observability is the one that makes the other eight possible: it focuses on agent actions, workflow outcomes, system changes, human overrides, and compliance events, creating a unified audit and monitoring layer across the enterprise. That layer feeds governance. It doesn’t replace it. Governance still requires policy, accountability, and human judgment. Observability just makes sure those decisions get made with facts instead of guesses.

That’s profound reframing. Observability in the AI era is not a DevOps concern. It is the evidentiary backbone that clinical safety, regulatory defensibility, and board-level trust are built on.

Observability across the AI lifecycle: three moments that decide everything

1. Development: Catch the drift before it ships

Before an agent ever touches a patient’s workflow, your team needs ground-truth visibility into how it behaves under load, against representative data, and across model versions. That means tracing every prompt, every model call, every token, every dependency — not just accuracy on a benchmark. In healthcare, “it works in the demo” is the most dangerous sentence in the building.

2. Deployment: End-to-end visibility from agent to EHR

This is where many health systems lose visibility. Positioning the Dynatrace OneAgent to ingest logs from Epic might call a FHIR API, which hits an HL7 interface, which triggers a downstream ETL, which writes to a SIEM. At agent speed, no human can follow what’s happening, and that’s the point. How can you operate without observability?

If any link in that chain breaks — or worse, silently degrades — the patient experience and the audit trail both suffer.

Can your current environment ingest EHR telemetry, SIEM/audit logs, clinical activity, FHIR/Interconnect API responses, MyChart activity, and ETL job status — and correlate it with the network, delivery layer, and the 200–400 applications that are connected to Epic?

Any observability strategy that incorporates patient-facing activity must also align with HIPAA requirements, organizational privacy policies, business associate agreements, and minimum-necessary access principles.

Done well, it produces a unified view of real-time log volume across SIEM, clinical, FHIR, HL7, MyChart, and ETL workers; FHIR endpoint distribution; security events such as break-the-glass access attempts; and latency on every Medication Request and Diagnostic Report call. That provides critical operational visibility into the systems surrounding AI-driven workflows. It does not by itself explain model reasoning, prompt behavior, or drift, but it provides the infrastructure context required to investigate and govern AI behavior.

3. Ongoing governance: Auditable, outcome-linked, board-ready

Governance is not a policy document. It’s not a once-a-month committee meeting either. It is a live, queryable system of record. Observability data can link AI activity — tokens, model selection, agent actions — to business context, token spend, and measurable outcomes making deployments auditable while helping teams compare model effectiveness and demonstrate that AI initiatives deliver real business value. That’s the bridge from IT telemetry to executive accountability that meets the expectation of the CIO.

That bridge is about to matter more, not less. Legal experts and compliance practitioners are increasingly flagging AI-enabled documentation and decision-support as potential FCA exposure areas, particularly where organizations cannot explain or validate how AI contributed to a claim. The U.S. Department of Justice (DOJ) continues to prioritize healthcare fraud enforcement under the False Claims Act, while legal and regulatory experts increasingly point to AI-generated documentation, coding, utilization management, and decision-support workflows as potential enforcement areas when organizations cannot explain or validate how decisions were made. ¹ ²

At the same time, the U.S. Department of Health and Human Services (HHS) Office for Civil Rights proposed significant updates to the HIPAA Security Rule in December 2024. The proposed rule would strengthen requirements related to technology asset inventories, risk analysis, system activity monitoring, documentation, and cybersecurity controls across regulated healthcare entities. ³ ⁴

States are also moving ahead independently. Several states have enacted laws restricting the use of AI as the sole basis for healthcare coverage determinations or utilization-review decisions and require meaningful human oversight, physician review, transparency, or disclosure when AI influences a care or coverage decision. ⁵ ⁶

The health systems that can produce evidence on demand, not slide decks but defensible evidence, will be the ones best positioned to satisfy regulators, auditors, boards, clinicians, and patients alike.

The strategic pivot: from observing infrastructure to observing intent

Here’s the shift every healthcare CIO needs to internalize. Traditional observability was about whether the system was healthy. AI-era observability is about whether the decision is defensible. That means correlating four layers in real time.

If those four layers feed a closed loop where real-time telemetry drives automated remediation and continuous optimization, you have AI governance. If not, you have AI hope.

A challenge to my fellow healthcare technology leaders

If you’re a CIO, CTO, or Chief AI Officer in a health system right now, I’d be asking myself these three questions before the next AI steering committee:

• Can I produce a complete, timestamped audit trail of every AI agent decision made in the enterprise over the last 24 hours? If not, what exactly are you governing?
• When the EHR-embedded agent talks to the enterprise copilot, which talks to the ServiceNow agent — who owns the failure mode? If the answer is “we’ll figure it out,” you don’t have a control plane — you have a coincidence.
• Is your observability strategy a budget line item under “DevOps,” or is it a board-reported capability under “AI risk and compliance”? That single org-chart question tells me whether your AI program will survive its first serious incident.
• Can you prove that your AI systems are delivering measurable value for patients, clinicians, researchers, or operational teams? If not, you’re measuring activity, not outcomes.

The bottom line

The healthcare organizations that win the next decade will not be the ones with the most AI agents. They’ll be the ones who can see, govern, and defend every decision those agents make. The category has a name — the Enterprise AI Control Plane — and observability is the foundation it’s built on, not an afterthought bolted on later.

I know which side of that bet I’m on.

FAQ: When the algorithm hallucinates in the ICU, who’s watching?

Q1. Isn’t observability just another word for AI governance?

No — Observability supplies the evidence and context that governance depends on. It doesn’t enforce policy, assign accountability, or make judgment calls on its own. Observability is one of the nine capabilities of an Enterprise AI Control Plane (alongside intent routing, policy coordination, identity delegation, agent registry, human escalation, safety guardrails, model governance, and resilience) — and it’s the one that makes the other eight possible by creating a unified audit and monitoring layer across agent actions, workflow outcomes, system changes, human overrides, and compliance events. Governance still requires policy, accountability, and human judgment; observability just makes sure those decisions are made with facts instead of guesses.

Q2. What does AI-era observability need to be captured in a health system?

It has to go well beyond a single agent or model. In production, an agent might call a FHIR API, which hits an HL7 interface, which triggers a downstream ETL job, which writes to a SIEM — and any silent break in that chain affects both the patient’s experience and the audit trail. Done well, observability correlates signals from EHR telemetry, audit logs, clinical workflows, API activity, patient-facing systems, and integration services. In healthcare environments, that visibility must be balanced with privacy, security, and minimum-necessary access requirements governing protected health information.

Q3. The post talks about four layers — Intent, Model, Infrastructure, Outcome. Why does that framing matter?

Traditional observability asked whether the system was healthy. AI-era observability asks whether the decision is defensible — and that requires correlating all four layers in real time, not just infrastructure metrics.

• Intent: Did the agent understand what was actually being asked (human or system prompt, workflow context)?
• Model: Is the right model being used, and is it drifting (token usage, model version, prompt/response, latency)?
• Infrastructure: Is the plumbing reliable enough for a clinical workflow (compute, network, dependencies, FHIR/HL7 endpoints)?
• Outcome: Did the agent help, hurt, or get overruled (business result, override rate, compliance event)?

If those four layers feed a closed loop where real-time telemetry drives automated remediation and continuous optimization, an organization has AI governance.

Q4. Why is this becoming a regulatory issue and not just an IT or clinical-safety issue?

Because the bar for “prove it after the fact” is rising on three fronts simultaneously. Federal enforcement agencies have flagged that AI use in healthcare creates new False Claims Act exposure for documentation, claims processing, and care decisions that can’t be explained after the fact. HHS’s long-pending HIPAA Security Rule modernization is expected to turn asset inventories, AI-inclusive risk analysis, and audit logging into explicit requirements rather than “addressable” suggestions. And third, a growing list of states (and nations) are not waiting for the federal (national) government. Some states now require a licensed human in the loop and disclosure whenever AI touches a coverage or care decision. Health systems that can produce evidence on demand will be the ones that clear all three bars.

Q5. What are the questions a healthcare CIO, CTO, or Chief AI Officer should be asking before the next AI steering committee?

• The post frames three diagnostic questions that double as a starting checklist for any organization assessing its own readiness:
• Can you produce a complete, timestamped audit trail of every AI agent decision made in the enterprise over the last 24 hours? If not, what exactly are you governing?
• When the EHR-embedded agent talks to the enterprise copilot, which talks to the ServiceNow agent — who owns the failure mode? “We’ll figure it out” means you have a coincidence, not a control plane.
• Is your observability strategy a budget line item under “DevOps,” or is it a board-reported capability under “AI risk and compliance”? That single org-chart question signals whether your AI program will survive its first serious incident.

1. U.S. Department of Justice. False Claims Act Settlements and Judgments Exceed $2.9B in Fiscal Year 2024 (January 15, 2025). [justice.gov]

2. Mintz. Artificial Intelligence and False Claims Act Enforcement (February 2024). Analysis of healthcare AI oversight, claims processing, coding, and human-review risks under existing FCA frameworks. [mintz.com]

3. U.S. Department of Health and Human Services, Office for Civil Rights. HIPAA Security Rule Notice of Proposed Rulemaking (December 27, 2024). [hhs.gov], [federalregister.gov]

4. HHS OCR Proposed Rule Summary. Requirements include technology asset inventories, risk analysis, security documentation, and enhanced monitoring controls. [assets.ctfassets.net], [coephi.org]