At some point in the future, NIH may halt funding for clinical trials deemed too underpowered to produce meaningful findings or that fail to meet enrollment goals. To do this, the agency would have to adopt so-called “stopping rules,” which is among the recommendations of a task force that spent two years studying NIH’s “stewardship” of clinical trials.
The recommendations were presented to the NIH Advisory Committee to the Director (ACD) during its most recent meeting; members did not vote on them. However, acting NIH Director Larry Tabak signaled his support for the concept.
NIH officials will need “the discipline to stop supporting underpowered boutique studies, which, quite frankly, many investigators around the country have built and sustained their careers on,” Tabak said. “We’re going to have to strike a better balance because the resources have to come from somewhere.”[1]
Before offering the recommendations, Debara Tucci, M.D., co-chair of the task force, presented data on NIH trials, some of which was gathered from ClinicalTrials.gov.[2] The committee also discussed the problem of timely filing of trial results on the website,[3] and the confirmation process for the new NIH director nominee.[4]
Tucci, also director of the National Institute on Deafness and Other Communication Disorders, noted that “about 40% of NIH’s annual budget supports clinical research.” More than $6 billion of NIH’s $18 billion annual expenditures for “clinical research goes to clinical trials.” NIH is funding fewer clinical trials testing treatments and drugs, with increased support being devoted to “prevention” and “behavioral” studies, according to Tucci’s presentation.
Treatment Trials See Big Drop
Looking just at NIH data for those trials on ClinicalTrials.gov, in 2005, the purpose of more than 70% of NIH-funded trials was “treatment,” according to bar graphs Tucci presented. Although treatment trials remain the largest type that NIH funds, by March of this year, the percentage had tumbled to approximately 46%.
Meanwhile, the percentage of basic science trials has increased from about 3% in 2005 to about 9% this year. Prevention studies increased slightly during this period, from about 12% to 15%. Health service research saw a jump from about 1% to approximately 8%.
The decline in treatment trials was concerning to ACD member Wafaa El-Sadr, M.D.
“I’m a little bit worried that [we] may be drifting away or shift[ing] away from that very fundamental role for NIH and really answering the big questions,” she said. “NIH is the only real entity that supports these types of trials.” El-Sadr is the founder and director of the International Center for AIDS Care and Treatment Program and a professor at Columbia University Mailman School of Public Health.
Tucci acknowledged she found the shifts “surprising,” saying, “That’s the reason I presented those data.”
Call for More ‘Implementation’ Studies
ACD member Brian Mustanski, director of the Institute for Sexual and Gender Minority Health and Wellbeing at Northwestern Medicine, said NIH should examine the “ethical and regulatory challenges” and provide “best practice and guidance” regarding pragmatic research and implementation trials.
Mustanksi also co-directs the Third Coast Center for AIDS Research and the Center for Prevention Implementation Methodology funded by NIH. “In my own field of HIV research, there’s been a lot of attention and growth to implementation research to try to get innovation into the field, into the hands of people who need them more quickly, domestically, globally.” These trials face a number of challenges and don’t always fit the standard regulatory frameworks, he said.
Tucci responded that implementation trials are a “topic of big consideration across NIH…that we’re all talking about,” adding, “I don’t think that we have come to the point where we have the answers on these.”
Michael Lauer, M.D., NIH deputy director for extramural research, said some units within NIH “are quite interested in implementation trials” and noted that the National Heart, Lung and Blood Institute “set up a Center for Translational and Implementation Research. There’s actually a reasonable portfolio of implementation trials across some of the institutes.”
As Tucci explained, the “key principles of clinical trial oversight and conduct” are to:
-
“Protect and support participants
-
“Ensure that research is meaningful to participants and public
-
“Ensure effective research/trial design and management to maximize benefit to public and conserve NIH resources
-
“Promote transparency, trust and communication of all clinical research conduct and results”
Currently, NIH lacks “standard metrics to monitor clinical trial conduct and progress,” and it needs to adopt processes to ensure trials are successful, said Tucci.
NIH Should Assist in Design, Enrollment
Stopping rules should be part of a “continuous review” of trials that NIH conducts after studies are funded, said Tucci, but NIH also needs to ensure trial design is appropriate, the task force found.
“There are lots of different definitions of success of clinical trials,” Tucci said. “The ultimate measure of success, of course, is do the findings substantially advance and improve patient care? Do they advance public health policy and scientific understanding? But there are also more granular metrics for following the progress of a trial, including whether the trial meets certain milestones for enrollment targets and enrollment rates, whether the primary endpoint is reached, and whether the scientific hypothesis can be addressed at the end of the trial.”
Tucci continued, “One particularly challenging process can be when to decide when to stop a clinical trial that’s underperforming, and that’s unlikely to reach the intended enrollment target, in order to maximize responsible use of NIH and participant resources, including time, money, and public trust. Studies that don’t meet enrollment targets are unlikely to be able to adequately answer the scientific question that they were designed to address. The task force felt that this is a particularly important issue to address across the agency.”
Tucci said NIH should implement the concept in two ways.
“One, how the studies that are submitted are reviewed. So, how do we make sure that the studies as they’re conceived of are not underpowered, that they’re conceived in a way that they’re likely to be able to address the question that they’re supposed to address? And then secondly, how do we monitor them in an ongoing fashion during the conduct of the study, and make sure that we can…help them with enrollment so that they have all of the resources brought to bear across NIH to help them enroll adequate numbers in the trials?” Tucci said.
No-Cost Extensions Could Be Limited
When enrollment goals can’t be met, “we need to have stopping rules,” Tucci said. “We need to have ways of saying, in a consistent fashion across NIH: ‘If you are not going to meet your enrollment target, we are stopping funding right now, and we’re not doing the five-year thing with the no-cost extensions for many, many months to allow you to try to meet your enrollment targets.’”
She did not offer specifics, such as how long enrollment would have been open before NIH would consider shuttering a trial and what other factors would be involved in the agency’s assessment.
Tabak pointed out that small isn’t “always bad” but said NIH must be more circumspect in its funding decisions in this regard.
“Knowledgeable” people “have pointed out to me that you can do a spectacular trial with very small numbers of people, but you need the right analytic approach…. But we need to, I think, do a better job going forward,” Tabak said. NIH must be willing to cease supporting trials that “just are too underpowered to give us any meaningful, actionable data,” he said.
Such a change, he argued, would free NIH to support more “bigger picture” studies.
Institutions may “unknowingly” contribute to the problem by failing to formally recognize “the people in the middle who do so much of the work” of a large trial, he added. “There is a tendency for those folks to strive….to have their own trial, one that they can call their own, and they tend to be…of this smaller variety.”
“Institutions can help us help them by giving better credit to the large multidisciplinary teams that are needed to do the really big trials, so that everybody who participates is getting appropriate professional credit,” Tabak said. “I don’t mean to be overly critical of institutions, but they do have a role here.”
‘Crappy’ COVID Trials Decried
Tucci agreed with Tabak that “there are small trials…pilot trials that need to be done with small numbers to test out ideas initially. But then in a larger trial that’s…conceived to be adequately powered from the beginning, if it’s not adequately powered, how do we just cut it off? I think the stopping rules are really, really important.”
She said that “during COVID, the so-called crappy small trials came up time and time again. And we definitely want to change that trajectory.”
ACD member Alexa Kimball, M.D., president and CEO of Harvard Medical Faculty Physicians, was supportive of the stopping rule recommendation. “The resources required to do these [trials] well are enormous and incomplete, underpowered, under-recruited studies not only are not a good use of those resources…if we don’t get an answer, [patients] participated for naught. So, the imperative around this is tremendous.”
El-Sadr addressed the quandary investigators face. While “small studies are important, they’re insufficient” in terms of having an impact. Investigators may be told large trials are too expensive to conduct, she said.
“It’s kind of a balancing act—how do we encourage the ambitious studies that are really fundamental” and build “on the critical small studies that are needed to inform the bigger ones. I think we [conduct the] smaller ones, but somehow, we shy away from taking the plunge to do the bigger ones,” said El-Sadr.
Other Wide-Ranging Recommendations Offered
The task force also recommended that NIH “consider whether new or enhanced policies are needed to increase diversity, equity, inclusion, and accessibility in the recruitment and retention of clinical research participants. This may include strengthening review of inclusion plans, updating language used in inclusion policies, new data collection and data improvement efforts, and centralizing approaches for recruitment, which is an ongoing challenge in many of our funded clinical trials,” said Tucci.
Additionally, NIH also should “promote participants as partners in clinical research and clinical trials” and “embrace and implement new research methodologies, strategies, and resources,” she said.
Tucci noted that the National Cancer Institute established “new guidelines to streamline and modernize clinical trials,” including limiting data collection for some late-phase trials. Officials “also seek to prioritize the use of the electronic health record data for clinical trials.”
Other agencies and clinical trial groups have “similar goals,” said Tucci. “We all recognize that clinical trials, as performed in the past, are often too unwieldy, too complicated, and too expensive.”
In closing the “very rich discussion,” Tabak said the task force and ACD members “really brought up some very important things for us to consider.” He did not address next steps for the task force or its recommendations.
Tabak also offered thanks at the end of what he said he expected was his last meeting with the ACD—his 44th—“now that we have a nominee for the permanent NIH director.”
The last 24 ACD meetings he attended were “in my role as either principal deputy or acting NIH director,” Tabak said. “I’m very fond of telling trainees that you don’t have to be the smartest person in the room to succeed you just have to be in the room with smart people. The privilege of working with all of you and those who preceded you on this committee has allowed me to always be in a room with extraordinary people. And I’m really, really grateful to all of you.”
The ACD’s next scheduled meeting is Dec. 14-15.