April 30, 2025
By Grace Totman, Co-Head of Capstone’s Healthcare Practice
Capstone believes that scrutiny of artificial intelligence (AI) in healthcare will increase state-level efforts to restrict its use, notably in utilization management (UM). This will create risks and opportunities for payors, providers, and vendors depending on state exposure and the nuances of the restrictions. Beyond UM, we believe interest in regulating the use of AI in healthcare will also continue to grow.
- Capstone believes interest in regulating AI in healthcare is growing, especially among state lawmakers. While both federal and state discussions are underway, state-level action is more immediate, as the number of bills introduced is growing rapidly. Federal lawmakers, however, largely remain in fact-finding mode. The Trump administration’s rollback of Biden-era AI regulations signals continued federal deregulation, likely encouraging further state-level activity.
- Most policymakers attention has centered on the use of AI in utilization management (UM), particularly when AI is used to deny or delay care through prior authorization and claims management. Several states have proposed requiring humans to make any decision that is not an approval, or banning AI in UM altogether. We expect this use case to remain the focus in the next year, with resulting policies creating complexity for payors, providers, vendors, and investors. For example, a proposed Minnesota bill would ban the use of AI in UM while an enacted Illinois bill allows AI in UM but requires a clinical peer—a licensed provider with clinical expertise in the specialty—to make any adverse decisions.
- Capstone believes the combination of AI-focused human requirements and increasing like-provider type mandates will drive greater payor reliance on UM vendors that serve as labor arbitrage without removing humans from the workflow.
A DEEPER LOOK
Health plans’ use of algorithms is under scrutiny. Congress, executive agencies, and the media increasingly question whether automated processes are replacing clinical judgment—leading them to emphasize the need for greater transparency.
In addition to AI regulation in UM, Capstone is closely tracking broader efforts to regulate AI in healthcare.
Timeline of Scrutiny
Concern about the use of AI in healthcare has significantly ramped up in the post-pandemic period. In March 2023, two investigative reports alleged that large payors—UnitedHealthcare, Elevance, Cigna, and CVS Health—utilized AI algorithms to inappropriately deny care to seniors in the Medicare Advantage (MA) program. These reports kickstarted much of the concern that exists today.
Exhibit 1: Timeline of Scrutiny of AI in Utilization Management
| Event Date | Description |
|---|---|
| April 2022 | HHS’ Office of the Inspector General (OIG) released a report showing that MA organizations deny prior authorizations (PA) that meet medical necessity criteria. |
| March 2023 | STAT News investigated UnitedHealth Group for limiting care to seniors with algorithms. |
| March 2023 | ProPublica published a report on Cigna’s Procedure-to-Diagnosis (PxDx) tool being used to batch deny claims without opening patient records. |
| April 2023 | The MA final rule clarified that MA plans may not use Milliman Care Guidelines (MCG) or InterQual criteria to restrict care access beyond traditional Medicare laws. |
| May 2023 | Republicans on the US House Committee on Energy & Commerce asked Cigna for more information on its PxDx tool. |
| July 2023 | HHS’ OIG released a report on Medicaid managed care organizations using PA to deny care more than MA plans. |
| July – September 2023 | Cigna was sued three times (in California, Connecticut, and Delaware) for using PxDx. |
| September 2023 | Sen. Bill Cassidy (R-LA) released an AI white paper, expressing interest in ensuring AI is not used to deny patient care. |
| November 2023 | Over 30 US House Democrats sent a letter to the Centers for Medicare & Medicaid Services (CMS) urging the agency to increase oversight on MA plans’ use of AI to deny care. |
| November 2023 | STAT News updated the UnitedHealth Group story, adding reports that it pressured staff to follow algorithm determinations, not clinical judgment. |
| October 2024 | US Senate Permanent Subcommittee on Investigations released a 54-page report on MA plan use of AI to deny PAs. |
| November 2024 | CMS issued CY 2026 MA and Part D proposed rule, including AI guardrails—with impacts to PA. |
| March 2025 | CMS Administrator Dr. Mehmet Oz agreed with members of Congress that humans should ultimately make coverage denial decisions, not AI. |
| April 2025 | CMS finalized its CY 2026 MA and Part D rule without the proposed AI guardrails despite near-unanimous support in comments. |
Source: Capstone analysis
Federal-Level Action
Despite scrutiny, federal-level efforts have lagged. President Biden’s AI Executive Order, the subsequent HHS Strategic Plan for the Use of AI in Healthcare, and CMS’ MA and Part D proposed rule—all of which happened prior to President Trump’s inauguration—have been rescinded or had AI guardrail provisions stripped, respectively.
While we believe the Trump administration has an interest in and will likely regulate AI in healthcare in some capacity, it has taken a “pro-innovation” approach and rolled back most Biden-era efforts that it believes “hinder AI innovation and impose onerous and unnecessary government control over the development of AI.” Capstone believes this strategy lends itself to federal inaction—at least near term.
AI Executive Order and HHS Strategic Plan
Biden Action:
In October 2023, President Biden issued the Executive Order (EO) on the Safe, Secure, and Trustworthy Development and Use of AI (EO 14110). The EO was designed to create a framework for broad AI regulation, charging multiple federal agencies with establishing standards, governance, and strategic plans for AI regulation. Specifically, the EO required HHS to develop a strategic plan that addresses the “development, maintenance, and use of predictive and generative AI-enabled technologies in healthcare delivery and financing.” Capstone previously covered the EO from the perspective of Big Tech.
Following the AI EO’s direction, HHS released a now-rescinded strategic plan in January 2025. In the strategic plan, HHS acknowledged that AI algorithms deployed by health plans for UM and PA fall outside of FDA medical device regulation and ASTP/ONC certification criteria for predictive decision support interventions (DSIs). While the agency described the benefit AI in UM can have—reducing delays in patient care—it listed “increasing the oversight and enforcement of existing federal laws and regulations, such as those prohibiting denying medically necessary, covered services or discrimination in access to federal benefits,” as a near-term priority.
Trump Action:
As soon as President Trump was inaugurated, EO 14110 was rescinded and replaced with a new EO on Removing Barriers to American Leadership in AI. The Trump EO charged agency heads with suspending, revising, or rescinding actions taken pursuant to EO 14110. As such, HHS’ strategic plan was taken down from its webpage. Capstone believes the Trump administration is interested in limited regulation for AI and unlikely to coordinate a unified regulatory plan for the technology. HHS’ current limitations in regulating payor administrative technologies, wherein authority lies in ensuring complying with coverage rules, it is unlikely there is regulation banning use. In fact, in an FAQ related to provisions on UM requirements within CMS’s 2024 MA and Part D final rule, the agency notes AI technology is permitted to assist in coverage determinations if it bases decisions on individual patient circumstances.
MA and Part D Rulemaking
Biden Action:
In November 2024, the Centers for Medicare & Medicaid Services (CMS) issued its proposed MA and Part D rule, which included guardrails for AI use to ensure equitable service delivery and forbid discrimination.
Trump Action:
Despite CMS Administrator Dr. Oz previously agreeing that humans, not AI, should have the final say in utilization management at his confirmation hearing, the agency did not include the AI guardrail provisions in the final rule. This comes amid reports that the administrator told federal staffers that AI models may be better than human physicians, and that patients may prefer an AI avatar. Capstone believes the federal government will continue to delay regulating the application of AI in UM until the administration creates a unified regulatory plan.
State-Level Action
Since 2023, state interest in creating the AI regulatory framework has grown, with the total amount of bills introduced growing rapidly. We believe this is only going to continue given relative inaction at the federal level.
Exhibit 2: State-Level AI Legislation Growth
Source: National Conference of State Legislatures
The primary focus of the health use bills in 2025 is legislating payor use of AI, specifically as it relates to UM and PA. The bills that were introduced relating to this area focus on 1) UM and PA determinations and 2) increased transparency on payor AI use.
In 2024, California adopted Senate Bill 1120, which mandates that a qualified human individual review UM medical necessity determinations. Illinois enacted H2472, which allows accredited algorithms (i.e., Utilization Review Accreditation Commission, National Committee for Quality Assurance) for medical necessity determinations, but mandates that only clinical peers can make adverse decisions. At the same time, states are also increasing “like provider type” mandates, which necessitate that payors have providers of a similar or same licensure type review appeals. For example, a neonatologist needs to review a neonatal prior authorization appeal. We believe complexities in UM compliance supports greater payor reliance on UM vendors who specialize in the area.
Bills introduced in 2025 fall into two buckets—states that allow AI in UM except in adverse decisions (e.g., Massachusetts SD 268), and states that outright ban AI from being used for PA decisions (e.g., Minnesota HF 2500).
Seemingly inspired by HHS’ Office of Inspector General reports on PA denials, states want information on payors’ use of AI, with bills like Maryland HB 697 requiring payors to submit quarterly reports related to the use of AI—including information about denials, overturned denials, and more. Other bills, like New York AB 1456, go even further, requiring payors to submit AI-based algorithms and training data sets being used for UM to state departments.
What’s Next
State-level legislation impacting the use of AI technology for UM is likely to continue being approved, with effective dates as early as this year. Direct impact on payors, providers, and vendors will vary based on state exposure.
Federal-level regulation is less likely, as it would require coordination from the Trump administration that is actively rolling back Biden-era AI actions. Legislation is unlikely this year given other congressional priorities.
Capstone intends to cover regulation of AI in healthcare in a more in-depth way going forward.
