A recent publication reports on an analysis of how study-specific fitness-for-use was addressed amongst pragmatic clinical trials that utilize real-world data (RWD), and makes recommendations to develop best practices.
To learn more about current practices, Duke University and Duke Clinical Research Institute (DCRI) faculty conducted a two-phase study that began with qualitative interviews to inform survey development. This was followed by the execution of the survey to trials that employed RWD. The results appear in the Journal of the American Medical Informatics Association’s ‘Evaluating fitness-for-use of electronic health records in pragmatic clinical trials: reported practices and recommendations.’ The study—part of broader research on the use of electronic health records (EHRs) data in a large cardiovascular clinical trial (NCT03872401)— was funded by the U.S. Food and Drug Administration (FDA) to support the 21st Century Cures Act mandate to use RWD to support regulatory decision making.
RWD and real-world evidence (RWE) are increasingly being used in regulatory decision-making. Defined as data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources, RWD may be extracted from EHRs, claims and billing activities, and registries. These are mostly considered as secondary data sources, so the underlying dataset must be judged to be ‘fit-for-use’ before leveraging it in clinical studies, including pragmatic and observational studies.
Despite taking place during the height of the COVID-19 pandemic, the response to requests for survey participation was strong. Of 15 invited trial teams, seven interviews were conducted. Of 31 invited trials, 15 responded to the survey. Most respondents had prior experience using RWD (93%). The major concerns expressed about EHR data were data reliability, missingness or incompleteness, variation in data quality across study sites, and the presence of implausible or incorrect values. While many pragmatic clinical trials carried out fitness-for-use activities such as data quality assessments, fewer than a quarter did so before choosing a data source. Fitness-for-use activities, findings, and resulting study design changes were not often publicly documented. Overall costs and personnel costs were barriers to fitness-for-use assessments.
“Based on experiences from pragmatic trials that used real-world data, we identified opportunities to bridge the gap between current approaches and ultimate best practices,” said Sudha Raman, PhD, lead author and faculty in the Duke Department of Population Health Sciences. “We suggest that future study teams should transparently define and articulate how study-specific RWD fitness-for-use will be assessed; conduct fitness-for-use assessments before the trial starts; and share the results of these assessments and how any challenges are overcome.”
“This new study illuminates decision making processes and identifies key concerns about real-world data in pragmatic trials, including endpoint ascertainment from electronic health record data,” said Emily O’Brien, PhD, second author and DCRI faculty member. “Prespecified assessments of fitness for use will form the basis of a body of evidence to inform best practices and decision-making around the use of RWD.”
While the studies examined were not for regulatory purposes, this new research highlights the importance of transparency and attention to detail during the decision-making process for future RWD/RWE trials. This documentation will be required increasingly as the FDA refines its approach to evaluating studies conducted with RWD.
Other authors of the paper include: Bradley G Hammill, DrPH; Adam J Nelson, MBBS, PhD; Laura J Fish, MPH, PhD; Lesley H Curtis, PhD; and Keith Marsolo, PhD.