Validating the Epic Sepsis Model — How a Prediction Running in Hundreds of Hospitals Actually Performed

The Epic Sepsis Model (ESM) is a prediction model that the US electronic health record vendor Epic Systems built into its software to give early warning of sepsis in hospitalized patients. On 21 June 2021 Andrew Wong, Karandeep Singh and colleagues at the University of Michigan published an independent external validation of it in JAMA Internal Medicine. They reported a hospitalization-level area under the curve (AUC) of 0.63, and found that at a threshold within Epic's recommended range the model did not flag 67% of patients with sepsis. Because the model was already running in hundreds of hospitals, and performed far below the figures its vendor had shared, the study is repeatedly cited as a warning about deploying medical AI first and checking it later.

It is often described as a failure of medical AI, but the ESM was not deep learning. According to the paper it is a penalized logistic regression, a comparatively simple statistical model. What was at issue was not the novelty of the method but how widely it was used without independent validation.

What the model was

The paper says Epic developed and validated the ESM on data from about 405,000 patient encounters at three health systems from 2013 to 2015, and that it was in use at hundreds of US hospitals as part of Epic's EHR. Because it is proprietary, only limited information about its performance was public, and to the authors' knowledge no independent validation had been published.

MedCity News reported that the ESM computes a risk score every 15 minutes from vital signs, laboratory values and other EHR data. Each hospital chooses the score at which an alert fires: a lower threshold produces more alerts, a higher one risks missing patients.

The paper opens by noting that early detection and appropriate treatment of sepsis have been associated with a significant mortality benefit in hospitalized patients. That is where the hope for early-warning systems came from. It also observes that EHR vendors had largely filled the "implementation gap" left by research models that never reached the bedside, by building warning systems into their products.

The study and its results

The researchers looked back at adults admitted to Michigan Medicine between 6 December 2018 and 20 October 2019. ESM scores were calculated for every hospitalization, but stays on the units where alerts were switched on were excluded to avoid biasing the evaluation.

A score of 6 is within Epic's recommended range. In the authors' words, the ESM "did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue." Of the 2,552 sepsis patients, the ESM flagged only 183 (7%) who had not received timely antibiotics, that is, cases clinicians had not already acted on; the authors read this as low sensitivity compared with contemporary clinical practice.

On the gap with Epic's figures, the authors pointed to how sepsis timing is handled. Counting predictions made after sepsis has begun makes a model look better: a sensitivity analysis that included predictions up to three hours after onset raised the AUC to 0.80. An earlier study had found that ESM alerts came a median of seven hours after the first lactate measurement, suggesting the alerts reflected sepsis already apparent to clinicians.

The authors' limitations and Epic's reply

The authors listed their limitations: the validation was done at a single academic medical centre, though with a large and relatively diverse cohort; they used a composite sepsis definition; and sepsis definitions themselves are still debated.

In an emailed statement to MedCity News, Epic said the study "did not take into account the analysis and required fine tuning that needs to occur prior to real-world deployment of the model." The company also maintained that the model helped clinicians intervene early, that its full formula and inputs were available to administrators, and pointed to a preprint from University of Colorado Health researchers who found Epic's model more accurate than their existing early-warning score. Wong, for his part, told the same outlet that Epic's model was trained using hospital billing codes, so that it was "essentially trained to identify which patients will be billed for sepsis, not which patients actually develop the clinical criteria for sepsis."

What it asked of hospitals and regulators

The paper's conclusion reached beyond one hospital. The authors wrote that the spread of proprietary models "has led to an underbelly of confidential, non–peer-reviewed model performance documents that may not accurately reflect real-world model performance", and that ease of integration within the EHR and loose federal regulations had let hundreds of US hospitals adopt such algorithms. They called on professional organisations to make formal recommendations about their use.

The cost on the ward ran both ways. Missed cases weaken the point of an early warning; too many alerts produce what the authors called alert fatigue. At the threshold studied, the positive predictive value was 12%, so most flagged hospitalizations were not sepsis. Wong told MedCity News that hospitals should run their own internal validation, and that proprietary models must be validated externally "before adopting them out of ease or convenience."

Regulation followed. The HTI-1 rule, published in the Federal Register by the US Department of Health and Human Services on 9 January 2024, requires certified health IT to let a limited set of users see and record "source attributes", categories of technical performance and quality information, for predictive decision support interventions. The rule's cost-benefit discussion says some predictive interventions are applied to tens or hundreds of millions of people despite crucial insufficiencies in performance, and cites this paper as one example. The rule itself acknowledges substantial uncertainty about whether transparency will actually lead to better models being used.

A follow-up: predictions that echo doctors' suspicions

In 2024 another Michigan team, led by Jenna Wiens, published a follow-up in NEJM AI. According to the university's release, across about 77,000 hospitalized adults the model looked strong (0.87) when all predictions during the stay were counted, but fell to 0.62 when limited to data recorded before patients met sepsis criteria, and to 0.53 before a blood culture was ordered. The researchers suggested that some of the data the model relies on may unintentionally encode a clinician's suspicion of sepsis, by which point the prediction is unlikely to help.

For AI sold as regulated medical devices, see IDx-DR in the US and EndoBRAIN in Japan. For a randomised trial of AI's effect in practice, see the MASAI trial; for another gap between promise and practice, see the sale of IBM Watson Health. The wider story is on the AI and society timeline.