Clinical trials test new treatments on thousands of volunteers and take years to complete. But what if we could approximate some of those trials using data that’s already been collected in real-world settings? No, it’s not magic–it’s modeling.
The challenge was posed to Eric Polley, Associate Professor in the Department of Public Health Sciences at the University of Chicago (then an assistant professor of biostatistics at the Mayo Clinic) by the US Food and Drug Administration: using electronic health record (EHR) data, could researchers predict the outcome of an ongoing clinical trial—before it was published? The goal wasn’t merely to guess right, but to explore what’s feasible and what isn’t when trying to replicate trial conditions using real-world data.
At the 2025 Mt Hood Diabetes Challenge, an international simulation model competition and conference hosted this year by the UChicago Center for Chronic Disease Research and Policy, Polley will deliver a plenary lecture on Wednesday, June 25 on his work using observational data to emulate clinical trials. “It sounds a little crazy,” he admits, “but the FDA wasn’t expecting to completely replace clinical trials. They wanted to understand the boundaries—what’s possible, what’s not, and what best practices look like.”
To begin, Polley starts with the trial’s protocol: inclusion and exclusion criteria, treatment arms, outcome measurements, and follow-up structure. Then they attempt to map each element to corresponding data points available in routine clinical care.
Sometimes that mapping is straightforward—especially in more pragmatic trials. Since many of these outcomes are documented in the health records of patients taking the drugs, it is possible to extrapolate the known data as if the patients were participating in a trial. Other times, it’s impossible. For example, trials may measure outcomes more frequently than patients typically visit doctors, or they may rely on specialized lab tests not routinely used in practice.
“We’re not asking, ‘Does this drug work?’” explains Polley. “Most of the time we’re comparing drugs that are already approved. Maybe Drug A was approved versus placebo and Drug B was also approved versus placebo—but they’ve never been compared head-to-head. So, using observational data, we can try to emulate a comparative effectiveness trial. For example: Which drug has fewer adverse events? Which is more effective in a given subpopulation?”
Polley and his collaborators have now built a cohort of about 7 million patients with type 2 diabetes using combined Medicare and private insurance claims. “With that, we can go beyond emulating real trials and start proposing hypothetical ones,” he says. “We design an idealized clinical trial—define the patient population, treatments, and outcomes—and then see if it’s feasible to emulate using the available data. For example, we might look at rates of cardiovascular events and compare GLP-1s versus SGLT2s.”
A major challenge in working with observational data is the lack of randomization. Patients don’t get assigned drugs by chance—they and their doctors make decisions based on a wide array of factors. To address this bias, Polley uses propensity score weighting to estimate the probability that each patient would have received one treatment over another based on their characteristics. Patients are then weighted according to the inverse of that probability, creating a pseudo-population that mimics randomization.
Machine learning can help with adjusting the weighting. “In these large data sets, estimating treatment propensity scores correctly is hard with traditional statistical methods. If we use more flexible models—we’re more likely to approximate the true probabilities,” he says. Polley teaches these methods in a machine learning for public health course at the University, where he also leads the data science concentration in the Master of Public Health program.
Yet even when results are promising, there are limitations. Claims data lack information on A1C levels or weight changes, which are crucial for evaluating diabetes treatments. Still, with careful design and the right statistical tools, trial emulation offers a way to answer pressing comparative effectiveness questions faster and more cost-effectively than traditional trials.
Eric Polley delivers the plenary lecture “Emulating Diabetes Clinical Trials using Medical Claims Data” at the Mt Hood Diabetes Challenge on Wednesday, June 25 at 1:30pm as part of a session on New Approaches to Modeling. REGISTER HERE.
Abstract:
There are many clinical questions where we do not have randomized clinical trials to answer questions on comparative effectiveness or safety. Examples include shifts in the intended population, limited collection of relevant outcomes, or concerns about long term effects not collected during clinical trials. The Target Trial framework is a communication guide for causal inference methods when attempting to emulate clinical trials using observational data (e.g. retrospective medical claims records). I will introduce the framework and demonstrate how it can be used to determine feasibility of available data for clinical questions. Examples from a series of diabetes trial emulations from electronic health records and Medicare claims data will be presented.