As healthcare organizations integrate AI into clinical care, executives are balancing the need to give their clinicians support in care management and coordination with a proper understanding of AI governance. And they’re starting to voice concerns about relying too much on AI.

The 2025 AI in Clinical Care Mastermind underwent a metamorphosis. The program, which began with a handful of CIOs and Chief AI Officers in 2024, now includes Chief Medical Officers, who are closer to the front lines where the technology is being introduced. As a result, roughly 15 health systems took part in the program, which culminated in the fall forum alongside the HealthLeaders CMO Exchange in Park City, Utah.

The main pressure point among the group? The potential negative effect of AI.

While healthcare organizations rush to launch new AI tools, they’re struggling to keep up with governance. 

"AI governance is a different twist on digital health governance in the sense that AI tools are not static and they are not predictable," says Lee Schwamm, MD, SVP and Chief Digital Health Officer at Yale New Haven Health. "AI tools are not like an electrocardiogram machine, where you connect it to the patient, then you get a consistent and predictable output."

But rather than setting up a separate AI governance structure, Yale New Haven Health has determined that AI governance needs to be an enhancement of the organization's existing approach to digital health governance. The health system has also set up an AI Implementation Advisory Committee.

"Today, about 15% of our digital health applications are overtly enabled with AI, but a year from now, that figure could be 60%," Schwamm said. "So rather than setting up two parallel governance structures, we have committed to creating a thoughtful model for how we govern tools that have AI."

"We are taking the approach of integrating AI governance directly into the quality and safety framework that we use as an institution," adds Mark Kandrysawtz, MBA, SVP and Chief Innovation Officer at WellSpan Health. "We track and monitor AI solutions in the same way we would track the operational output of our human team members. Instead of AI governance being a separate group, it is now part of how we manage innovation and transformation overall."

ProMedica has resisted the impulse to "reinvent the wheel" when it comes to AI governance, according to Brian Miller, MD, Chief Medical Information Officer and Vice President of Telehealth.

"There are nuances with AI tools such as consent, risk, and the Health Insurance Portability and Accountability Act," Miller said. "So, our compliance, risk, and legal teams have improved their knowledge for these factors to manage the nuances. But our governance processes for AI tools generally mimic what we have always done when we look at any tool that helps us."

Another key aspect of AI integration is change management. Not all clinicians are ready – or willing – to embrace AI. Some are old-fashioned, preferring to do things as they’ve always done it because it works for them, while others are leery of the hype surrounding AI and the concerns that others have aired.

“I’m still getting pushback that this is a waste of time,” adds Erik Summers, Chief Medical Officer at the Medical University of South Carolina (MUSC). He says doctors are quick to reject technology that interferes with their routines and are just as likely to find work-arounds.

Corey Cronrath, DO, MPH, MBA, FAAPL, FACOEM, CMO of the Mental Health Cooperative, says his organization tried an ambient listening tool a few years ago and was forced to discontinue it because their doctors found it to complex and intrusive. He said it took almost three years to bring their doctors back on board – critical time lost in an age where hospitals are losing patients and clinicians if they aren’t demonstrating that they’re using AI.

“A really good [AI] tool can fool you into making you think it’s thinking for you,” notes Tipu Puri, MD, PhD, CMO of the University of Chicago Medical Center. That’s why it’s crucial that clinicians and others understand the basics behind predictive analytics when they apply AI to clinical care.

Eventually, Blum notes, AI tools will evolve into thinking tools, requiring even more continuous monitoring.

Kandrysawtz says AI also has the potential to improve public health.

"For example, AI has the ability to boost outreach to the community at scale not only in the area of notifications about infectious diseases but also engaging and educating communities about issues such as cancer screening and prevention," he says. "AI can not only communicate information to communities but also answer their questions on a scale that is not possible with a human workforce."

Chances are, if someone has thought of a new way to use AI in healthcare, it’s being tested out somewhere. That includes tools that can interpret diagnostic images and tests, ambient AI for nurses, the Emergency Department and the ICU, even AI-enabled cameras that can monitor patients in the ED or in their hospital rooms.

"In clinical care, the area for new AI tools that has the biggest potential for impact is in the provider experience," Miller said. "We plan to drive a much better experience for our providers with an AI scribe tool, which focuses on the patient-clinician encounter." 

"One AI tool will look at the patient's history and medications that the patient is taking as well as summarize the patient's chart," he adds. "We are also looking at AI tools that can help providers after a patient encounter in areas such as coding and Hierarchical Condition Categories capture."

"We are expanding our relationships with existing digital diagnostic partners," Kandrysawtz says. "Over the past three years, we have been able to measure the impact of about a dozen AI solutions that we have in place for digital diagnostics. That has given us confidence to embrace new solutions more rapidly in the digital diagnostics space."

Blum says some larger health systems with the resources will eventually develop their own AI tools for clinicians, rather than buying something from a vendor. For example, he says, his health system is developing an AI tool to identify pressure ulcers, a specific use case that would cost a lot of money for a vendor to address.

Tom-meka Archinard, MD, MBA, FACEP, SVP and CMO at University of Maryland Capital Region Health, says AI is also being applied to critical areas outside of clinical care, such as in identifying and addressing social determinants of health and other issues outside the hospital that affect patient care. AI has the potential, she says, to crunch large amounts of data better and faster than humans, enabling healthcare leaders to better understand where their patients need help and funnel them to the right resources.

As Thomas Balcezak, MD, MPH, EVP and Chief Clinical Officer for Yale New Haven Health, puts it, AI will eventually become so accepted and commonplace that we'll forget how much trouble we had putting it in place.

The HealthLeaders Mastermind series is an exclusive series of calls and events with healthcare executives. Please join the community at our LinkedIn page. Please join the community at our LinkedIn page.

To inquire about participating in an upcoming Mastermind series or attending a HealthLeaders Exchange event, email us at anorris@healthleadersmedia.com.