After releasing a report requested nearly three years ago, Sen. Elizabeth Warren said the Food and Drug Administration (FDA) and HHS “should clean up the industry to keep patients safe.”[1] The industry to which the Democratic senator from Massachusetts was referring is institutional review boards (IRBs) and by patients, Warren meant participants in research studies.
The Government Accountability Office (GAO) found gaps in how FDA and the HHS Office for Human Research Protections (OHRP) oversee all types of IRBs; it also examined consolidation by two large, for-profit IRBs.[2] GAO made four recommendations, including that the agencies “examine approaches for measuring IRB effectiveness in protecting human subjects and implement the approaches as appropriate.” How to assess whether IRBs are doing a good job—and first defining what that means—has bedeviled human research protection officials for years. No methods are universally accepted or widely in place today, despite numerous efforts.
Now that task has fallen to the HHS Secretary’s Advisory Committee on Human Research Protections (SACHRP), which took a first stab at it during its most recent meeting.[3] Yet, as of this month, SACHRP, whose charter calls for it to have 11 members, was expected to be down to just four due to terms expiring and a lack of new appointments.[4]
“This issue of IRB effectiveness is one that SACHRP has grappled with before, and it remains a difficult” issue, Julie Kaneshiro, acting OHRP director, said at the meeting. She added that OHRP was “really looking for your help and learning from all of you who have been doing some deep thinking about this over the many years.”
As with most of its recommendations, these are being drafted by SACHRP’s subcommittees and then will be presented to SACHRP for approval. In this case, David Forster, chief compliance officer for WCG, and Susan Kornetsky, director of clinical research compliance at Children’s Hospital in Boston, are coauthoring them.
Forster cochairs SACHRP’s Harmonization Subcommittee, while Kornetsky is a member of its Subpart A Subcommittee. Forster and Kornetsky facilitated SACHRP’s discussions at the meeting.
As Forster pointed out, the assignment is twofold: defining and measuring IRB effectiveness. What SACHRP will need to decide is “what definition of IRB effectiveness is the most important to focus on and measure,” Forster said, adding there may be multiple measures.[5]
GAO’s specific recommendation is:
“The Secretary of Health and Human Services should ensure that OHRP and FDA convene stakeholders to examine approaches for measuring IRB effectiveness in protecting human subjects and implement the approaches as appropriate. These could include effectiveness measures; peer audits of IRB meetings and decisions; mock protocols; surveys of IRB members, investigators, and human research participants; or other approaches.”
OHRP Seeks Defintions, Approaches
In response, the specific questions OHRP asked SACHRP to address are:
-
“What constitutes effectiveness in protecting research participants? This could be defined in terms of avoiding harms, ensuring subjects exercise informed consent, protecting subjects’ rights and welfare, treating subjects equitably or fairly, or achieving greater consistency in applying the regulation, or something else. Depending on what is being protected, the IRB’s actions could differ and measures of effectiveness would vary accordingly. What definition of IRB effectiveness is the most important to focus on and measure?
-
“SACHRP is one HHS ‘stakeholder’. What other stakeholder groups should HHS convene as part of examining approaches for measuring IRB effectiveness? What factors make an entity an appropriate stakeholder?
-
“GAO provides several potential effectiveness measures. How do these approaches differ, and what are their benefits and limitations? What approaches should HHS and stakeholders prioritize? Are there other approaches should HHS and these stakeholders consider for measuring IRB effectiveness in protecting human subjects?”
Turning to the question of effectiveness of IRBs in protecting research participants, subcommittee members made the following points:
-
“IRB oversight of research is grounded in three ethical principles, respect for persons, beneficence and justice
-
“These principles at times conflict when applied to a given protocol (e.g., desire to enroll all language speakers versus need to have the consent process in a language understood by both research staff and subjects)
-
“The principles are codified in the Common Rule, in criteria for approval and elements of consent”
In an overview of how IRBs function today, Forster noted that “the IRB system was not designed with effectiveness measures in place” and said operations are “more of a flow-through process.” The Common Rule doesn’t require IRB members to meet publicly, although some states do, and federal requirements for meeting minutes are “minimal,” he added.
As noted in its recommendations, GAO suggested some measures of effectiveness: peer audits of IRB meetings and decisions; mock protocols; and “surveys of IRB members, investigators, and human research participants,” according to Forster’s slides.
For its part, OHRP provided SACHRP with a few suggestions of its own, Forster said, namely how well they avoid harms, ensure participants exercise informed consent, protect participants’ rights and welfare, treat participants equitably or fairly and achieve “consistency in applying the regulation.”
Common Standards Reflect Compliance
Other “possible” standards of IRB effectiveness Forster presented are:
-
Compliance with regulatory requirements
-
Compliance with the Belmont Principles
-
Compliance with accreditation standards
-
IRB member education assessments
-
IRB staff education assessments, such as certified IRB professional
-
Research staff and institutional compliance with the IRB’s requirements
-
Public trust in the IRB system
-
Human research participant satisfaction with research and IRB research oversight
-
Investigator satisfaction
The subcommittee’s suggestions for measuring effectiveness match up with these. For example, accreditation status and investigator certifications could be tallied. Turn-around times, error rates, unanticipated problems and investigator noncompliance are among possible measures of IRB effectiveness, according to the subcommittee.
The subcommittee provided SACHRP with the following list of stakeholders to address the second question.
-
Individuals who volunteer to participate in research
-
Patient advocacy groups
-
Institutions that conduct research, from Academic Medical Centers to single practitioner sites
-
Institutional officials
-
IRBs
-
Investigators
-
Research staff
-
Agencies—all Common Rule agencies, including FDA
-
SACHRP
-
Commercial research funders, such as sponsors and clinical research organizations
-
Bioethicists
-
The Consortium to Advance Effective Research Ethics Oversight (AEREO)
-
Association for the Accreditation of Human Research Protection Programs
-
Consultants
SACHRP agreed to add “communities that have been historically harmed and exploited by research” to the list of stakeholders, following a suggestion from SACHRP member Consuelo Wilkins, M.D., senior associate dean for health equity and inclusive excellence and professor of medicine at Vanderbilt University Medical Center.
Less Process, More Outcomes Metrics
Before beginning their discussion, SACHRP heard presentations by GAO officials and from Emily E. Anderson, associate professor at the Neiswanger Institute for Bioethics, Stritch School of Medicine at Loyola University Chicago; Ann Johnson, director of the University of Utah’s IRB; and former SACHRP member Holly Fernandez Lynch, assistant professor of medical ethics and law at the University of Pennsylvania.[6]
Lynch discussed AEREO’s efforts to collect and analyze data on IRBs’ effectiveness and the concept that IRB decisions and operations should be “reasonable.”
Wilkins said she had concerns about this approach, stating that reasonableness is based on “group norms” and might not protect people from harms. Instead, effectiveness could be measured by “looking at the outcomes of studies. What were the adverse events? Were people harmed? Were studies completed? Did they meet accrual goals? Did they meet diversity accrual goals? How many complaints did people have?”
Other things to consider would be “measures around participants’ perception and satisfaction, what’s the uptake of results in the community, whose privacy was violated. I think there are lots of different ways” to assess IRB effectiveness, Wilkins said.
“My approach typically would be to…create the logic model with these outcomes that are expected if something were effective and see what the measures are that exist for those,” Wilkins said. “I’m not against having some process measures be there, but they should be aligned with a causal pathway that would be expected to change the outcomes to protect people. There should be evidence those things would actually protect people or that there would be less harm.”
Members also said it would be important to assess whether IRBs are making correct decisions about research that is exempt or subject to expedited review. As the discussion expanded into IRBs’ contributions, SACHRP chair Douglas Diekema, M.D., offered his perspective.
“With 23 years as an IRB chair, I have had abundant opportunity to review situations related to noncompliance at my institution, and many other things,” said Diekema, professor of pediatrics and adjunct professor of bioethics and emergency medicine at the University of Washington School of Medicine. “It’s very clear to me that investigators just do not and never will have a handle on the regulations that allows them to remain compliant.”
Diekema added he is “fairly convinced we could still not be enrolling very many patients…for whom English is not their first language if our IRB were not making that a fairly important requirement.”
IRBs, HRPPs Interconnected
Several members said SACHRP will need to decide whether to develop recommended standards that are minimums or best practices.
After figuring out “what constitutes effectiveness,” SACHRP members “should not be limited in our thinking about the measures that exist,” Kornetsky said, echoing others’ comments.
Another consideration is whether members should confine their recommendations to IRBs or, more broadly, address effectiveness measures for human research protection programs (HRPPs). Although discussions at the subcommittee level have “veered around wildly,” members ultimately said IRBs should be the focus, Forster said. But “this is your recommendation,” he said of SACHRP members.
SACHRP member Robert “Skip” Nelson suggested that SACHRP focus on IRBs but not to the exclusion of HRPPs.
“OHRP has a hold on IRBs in a way that’s different than HRPPs, so my own view is that I would probably respect that the charge is [about] IRBs,” said Nelson, senior director of pediatric drug development in the Child Health Innovation Leadership Department at Johnson & Johnson. “But, in doing so, we need to set the overall context for IRBs being embedded within an HRPP. We need to recognize if we propose certain measures, whatever that might be, whether quantitative or qualitative, we should identify which are really focused on the IRB versus the HRPP.”
The recommendations also should “address the context of independent IRBs, which may not be embedded in an HRPP. You can have a very effective IRB and a disaster HRPP, and participants aren’t protected,” Nelson added, “or you can have a not-so-great IRB and wonderful HRPP, and wonderful investigators, and maybe they are protected.”
OHRP ‘Flexible’ on Timing
SACHRP spent approximately three-and-a-half hours on the topic, after which time Forster said he wasn’t certain how SACHRP wanted to proceed. Members then agreed that subcommittee members should develop bullet points or key messages so SACHRP can weigh in and provide clearer direction on possible recommendations.
Members will need to develop a “consensus on what stays in, what goes out,” Kornetsky said. Noting that some ideas the committee floated are “divergent,” this step will be “the hard work.”
Diekema asked Julia Gorey, SACHRP executive secretary, when OHRP expects the recommendations. “I think OHRP is flexible,” Gorey said. “We certainly recognize this is a huge task, a big lift.”
SACHRP’s next meeting is July 19-20. Its third and final meeting of the year is scheduled for Oct. 18-19. Unlike the others, this meeting is expected to be held in person. Recommendations on IRB effectiveness might be ready for SACHRP approval at the October meeting, Diekema said.