At Cornell University, institutional review board (IRB) members meet with the chief information security officer and a liaison to the general counsel’s office. Their regular attendance has been “really critical,” said IRB administrator Vanessa McCaffery. “Even though the IRB is tasked with thinking about participant protection, we also don’t want to approve anything that’s going to potentially cause a legal issue for participants or for the institution.”
McCaffery recommended adding experts such as these to IRB meetings during the Q&A period following a recent talk on artificial intelligence (AI) at the annual meeting of Public Responsibility in Research & Medicine (PRIMR).[1]
“Part of my job is to realize what I don’t know,” McCaffery said at the meeting, adding that she’s been learning about AI on her own and is “kind of the AI nerd on the IRB staff right now.”
In comments after the meeting to RRC, McCaffery said she was particularly “thinking about the potential legal and security implications for human participants and for researchers that come with use of new tech such as AI transcription tools and other countless, rapidly proliferating tools and technologies.”
Like McCaffery’s, many of the concerns raised by both the speakers at the PRIMR talk and audience members relate to ensuring the privacy and confidentiality of research participant information when AI is used, with questions about how HIPAA rules apply—or don’t, given they were implemented long before AI began being used.
But she’s also not alone in trying to understand AI and its implications for IRBs. For now, universities and other institutions employ a kind of DIY strategy in the absence of federal or even state regulations and guidance on AI. IRB officials have a double duty: analyzing how investigators whose protocols they’re reviewing are using AI and considering AI technologies that might actually help their own operations and oversight.
During the talk, Donella S. Comeau, M.D., a neuroscientist specializing in clinical AI development, presented elements of various governance structures for generative AI in a health care or research setting and for IRB oversight of AI.[2]
Regardless of whether there are laws, regulations, policies or guidance, governance “must address unique features of AI,” said Comeau, who is also a neuroradiology attending at Beth Israel Deaconess Medical Center and vice chair of the Mass General Brigham (MGB) IRB. As described on her slides, these are:
“Explainability: It is often difficult to know why an AI does what it does, so regulating it and auditing it in the event of an injury may be difficult.
“Non-directed behavior: AI doesn’t need explicit instructions and can develop its own behavior, which makes it difficult to delineate regulations that would cover all potential AI behaviors.
“Emergent behavior: Non-directed behaviors also mean that AI often behaves in unexpected ways, which can create unanticipated problems.”
Expanded IRB Authority, Knowledge Required
Currently, AI research doesn’t generally get special consideration by IRBs. Most is “treated like standard research,” Comeau said, and may be categorized as minimal risk or even exempt, reducing IRB oversight.
Another issue is lack of transparency. “We often don’t get complete disclosures to the IRB, and part of this is related to proprietary concerns.” IRBs, she added, need “real autonomy” and “authority” to require researchers to change protocols to mitigate any possible harmful effects.
The Belmont Principles also apply to AI, but they will need to be expanded and adapted, said Comeau, who also advocated for “mandatory bias training for AI researchers.”
“We have to enhance the capacity of IRBs to review AI models. And we need to standardize our review processes [and have] protocol templates and consent forms,” she said. “We have to educate researchers, ourselves, the public, and this needs to be a real broadened engagement. We cannot be siloed here, and all of us have to stay informed and keep abreast of the changing regulations and norms in order to safeguard [against] the potential adverse effects of AI.”
Added Comeau: “What we are looking for here is the next step in our good clinical practice, that AI…would have rules that we could follow, and that hopefully these guidelines would be expanded before harmful or unethical events occur.”
For IRBs themselves, AI could be “very important” and “very helpful, especially for things like consent, consent at different levels of cognition, consent for people of different languages,” Comeau said. AI, for example, might be the “only way” to accomplish certain goals, she added, such as ensuring data sets or participants in a clinical trial are really diverse.
During the Q&A portion of the talk, a question was asked: “Should a researcher let the IRB know that they’re using a generative AI to help them make a manual that participants are going to get in an intervention?”
Jack Gallifant, M.D., a postdoctoral student at Massachusetts Institute of Technology (MIT) and one of the speakers, replied that “there’s a lot of use of these sort of approaches,” which demands transparency. “At the end of the day, even [with] that single researcher making that document, it’s the institution who’s going to bear the brunt of the blame if it goes wrong. They’re going to need to declare that.”
Another audience member wondered whether human subjects regulations need to be adapted to address AI because, in most cases, data sets are assembled without consent due to deidentification.
Comeau acknowledged this was a difficult question that might prompt a discussion of who is the subject and who might be harmed. “These are all things that we are thinking about. One of the things that people are talking about is requiring that, if you are going to have data sets, they be fully consented.” But this “is not something that we can require of everyone,” such as those in industry, she added. Where the IRB has oversight “in medical research, we can start to say that our researchers, when they build models, may need to have consent.”
Mirror Collaborations Forged to Fight COVID-19
Another idea is the concept of “federated models, where the data remains with you, but you allow access to it,” Comeau said. “These are the things we need to ask for. These are the things that we need to push our machine learning and data science colleagues to build for us.”
In response to a question about resources and concerns for small IRBs, which “do not typically review complex research protocols,” Comeau said MGH is working with Harvard, MIT and others “to develop open-source tools,” given “we are in a very unique situation where we have a lot of expertise in the field of AI available to us.”
Comeau suggested that tackling the challenges of AI in research requires the same collaborative approach that was a hallmark of COVID-19. “A lot of this comes down to collaboration; it comes down to unified policies. We really need people who have various expertise, inputs, influences, experiences to build these tools that we need,” she said.
In addition to Comeau and Gallifant, speakers included Mark Dredze, professor of computer science at Johns Hopkins University. He answered a question asking whether there are heightened privacy and security concerns related to AI.
“AI systems are no different from a lot of the other online tools you have,” Dredze said. He noted that “people become really concerned about sending patient data to an online tool…then [they] use Gmail to forward it around.”
The answer is the same as with “any other information system,” according to Dredze. “Do you have the rights, permissions and the security in place to send it wherever you want it to go? I think most of us understand that we might have an internal SharePoint that we can use, but we can’t put it on Google Docs. There are those kinds of restrictions in place. And the same thing is true with any AI system.”
Could Internal Protections Slip After Development?
Dredze said he does not use ChatGPT but a version that is “HIPAA compliant” and thus can handle “patient data.” He added, “we have plenty of understanding about data privacy and security, and we just need to sensibly apply it here, as well.”
As a follow-up, Benjamin Silverman, M.D., senior IRB chair for MGB, said that “many academic medical centers [are doing] exactly what you said, which is to use internal instances of generative AI algorithms that are HIPAA-compliant to do our work.”
But Silverman added that “much of that work, I would say to your point about authorizations, is done on large, unconsented data sets where we’re making a waiver of authorization for consent” that also applies under HIPAA.
Silverman noted, “we do these analyses on our secure internal servers, but the goal at some point is that those algorithms will be either made open source or commercialized, et cetera.” When this happens, “should we, or do we need to worry about essentially retained private information within those algorithms when they are taken out of our hospital servers and used more broadly?”
Dredze said AI does change the dynamic because “historically, we’ve been more confident [in] our ability to keep the data used to develop the models private when the models went public. And that is increasingly not the case.”
As a result, “We need to be very, very cautious,” Dredze said, noting, “at Hopkins, I think we’re very cautious.” Adding that he didn’t “want to name names,” Dredze said he has “seen other institutions do things that I don’t think our IRB would’ve been okay with. We’re trying to figure out where the middle ground here is, and it’s really hard to know.”
‘You All Are Problem-Solvers’
Geoff Lomax, associate director for medical affairs and policy at the California Institute for Regenerative Medicine, noted the discussion harkened back to those “we were having decades ago with the utilization of genomic data and research.” At that time, “you all as a community, or we all as a community, did a lot of painstaking work to develop a framework [that] can be utilized responsibly,” he said.
After there were unethical or abusive uses of genetic data, “legal and policy frameworks evolved,” he said, including passage in 2008 of the federal Genetic Information Nondiscrimination Act. Lomax asked whether a similar law might be passed for AI and whether the speakers see “policy frameworks sort of emerging in a way that they give you confidence that will end up in a similar place.”
Comeau responded that she “hopes so,” adding, “I don’t know that I necessarily see that path now. I don’t really see a unified scientific movement…I definitely see groups who are starting to talk about this, but I don’t see it as something where we are all on board; we’re all thinking about this.”
It “worries” Comeau that the current approach to AI issues, in her view, is similar to climate change, with “a lot of people who are out there and giving us the truth and talking about the things we should worry about.”
But the discussions aren’t resulting in “practical solutions. And so that’s why I’m really encouraging you all,” Comeau said. “You all are problem-solvers. You all develop policies. And, so if you start from your foundation and you connect with these other groups, I think that we can build that.”
Policies may come more from a “grassroots” movement, with “the people in the trenches, the people who are really there to take care of the health of people and to safeguard them [having] to take the lead here. Because I think sometimes the leaders aren’t able to unify people to do it,” Comeau said.