How do we prevent automated efficiency from turning into clinical neglect? As healthcare organizations increasingly hand over administrative and diagnostic tasks to machine learning, we face a fundamental question: Can algorithms optimize clinical workflows without systematically inflating costs and denying necessary patient care? In North Carolina, this tension is moving from academic journals directly into legislative chambers as lawmakers attempt to draw a firm line between human clinical judgment and automated decision-making.
When Proxy Data Distorts Clinical Reality
In corporate insurance and hospital administration, artificial intelligence is no longer a futuristic concept. According to a comprehensive report by WRAL News published on May 21, 2026, state legislators are currently debating House Bill 565, which would prohibit insurers from using AI as the sole basis for denying healthcare claims or prior authorization requests. The underlying concern is that without human oversight, algorithms naturally optimize for variables that do not align with patient health.
This phenomenon was clearly demonstrated in a landmark 2019 study published in the journal Science. While public headlines simplified this finding as "AI displays racial bias," the actual research revealed a more complex structural flaw: the algorithm was not programmed with racial prejudice, but it used "future healthcare costs" as a proxy variable for "health needs." Because systemic factors meant less money was historically spent on Black patients, the algorithm erroneously concluded they were healthier than white patients with identical clinical profiles. When researchers adjusted the model to evaluate actual health conditions rather than financial expenditure, the racial bias disappeared, highlighting how easily an unmonitored proxy can distort clinical outcomes.
The Hidden Financial Incentives of Machine-Generated Codes
The clinical side of healthcare faces a parallel risk in the form of "upcoding," where AI tools recommend more complex, expensive diagnosis codes to maximize insurance reimbursements. State Senator Amy Galey recently pointed to a Rice University report showing that hospital service prices have more than tripled since 2000, warning that AI billing tools are already driving the appearance of more severe diagnoses without any matching changes in actual patient treatment. When algorithms are trained on historical billing data, they quickly learn that higher codes lead to larger payouts, creating an automated incentive loop.
As Alessandra Bazzano, a professor at the UNC Gillings School of Public Health, points out, AI does not invent these behaviors; it amplifies existing systemic pressures. If a hospital's reimbursement structure rewards clinical severity, an unmonitored algorithm will rapidly accelerate billing toward those high-cost codes. To counteract this, House Bill 565 mandates that providers and vendors cannot use AI to recommend or generate billing codes that result in upcoding unless a treating provider reviews the documentation and confirms it is supported by the patient’s medical record.
Limitations to Consider in Legislative Enforcement
While the bill's intent to keep humans in the loop is clear, enforcing such regulations in a highly digitized environment presents major practical challenges. State Senator Gale Adcock, a practicing nurse practitioner, noted during committee debates that upcoding is already prohibited under existing laws, suggesting that the new proposal still needs significant work to be enforceable. The core difficulty lies in detection: how do regulators prove that an insurance denial was "solely" decided by an AI, rather than a human reviewer who simply rubber-stamped an algorithmic recommendation?
Furthermore, data from the National Association of Insurance Commissioners shows that most health insurers already integrate AI into daily operations, from fraud detection to automated prior authorizations. Demanding that busy clinicians manually audit every automated recommendation could worsen professional burnout. A DemandSage survey indicates that physician AI adoption has surged to two-thirds of practitioners, up from 38% in 2023. Stripping these tools away entirely, or burying them in administrative red tape, could stall genuine clinical efficiency, which is why Senator Natalie Murdock has emphasized the need to involve hospitals, insurers, and providers in shaping the final text.
Balancing Regulation with State-Funded Innovation
The debate is further complicated by North Carolina's dual role as both a regulator and a major financial backer of medical AI. Even as lawmakers push for billing restrictions, the state-backed public-private partnership NCInnovation has committed over $19 million to commercialize university-developed AI healthcare projects. These range from AI-assisted maternal ultrasounds at UNC-Chapel Hill to virtual reality nurse training at Winston-Salem State University.
The next critical step for researchers and policymakers is to establish standardized audit protocols that measure "algorithmic drift"—the tendency of AI models to shift their optimization goals over time. As Kandyce Brennan, co-chair of the AI task force at the UNC School of Nursing, notes, transparency is the only way to determine whether AI is genuinely supporting clinical decision-making or simply automating cost-shifting. Future research must focus on designing "explainable AI" systems that provide clear, human-readable rationales for their decisions. Monitoring the progression of House Bill 565 through the state legislature will provide a crucial signal of how governments intend to police these digital gatekeepers without stifling life-saving clinical innovations.
