For the first time since 2012, the HHS Office of Research Integrity (ORI) issued[1] an annual report, but for those used to previous updates or looking for useful details, it may prove to be a disappointment. Covering fiscal year (FY) 2020, the report consists mostly of graphics and presents essentially a snapshot of activities. Missing are trend data and any information from past years’ actions. ORI has a unique and crucial role in the research world: It investigates research misconduct, defined as fabrication, falsification or plagiarism, in the billions of dollars of studies supported by NIH and other Public Health Service (PHS) agencies.
The new report also contains no analysis of the agency’s findings of research misconduct and omits other categories of information included in past reports, such as the number of allegations investigated by PHS-funded institutions. Those interested in ORI’s activities and findings also can’t do their own analysis as ORI removed all of the previous annual reports and has no plans to put them back, RRC has learned.
Agency officials also told RRC they intend to release some additional data when analyzed; in other instances, they said the information is unavailable.
But Alan Price, a former ORI official who is now a research integrity consultant, told RRC the new document fails to fulfill the historical purpose of the annual report—helping to educate research integrity officials (RIOs).
Even in its skeletal form, the surprise posting of the report in February may signal more of a return to normalcy for ORI. Despite its critical job of overseeing integrity in the government’s largest research portfolio, the agency has been battered over the last decade by leadership vacancies and allegations of malfeasance by a former director who was removed after two controversial years in the post.
March 2020 marked the first time since 2016 that ORI’s four leadership positions were filled by permanent appointees.[2] At that time, Elisabeth Handley was named director after serving as interim director beginning in July 2019, and ORI announced the appointment of Karen Wehner as director of the Division of Education and Integrity (DEI), a post that had remained open for four years. Alex Runko has been the director of ORI’s Division of Investigative Oversight (DIO) since July 2019. The fourth leader is Wanda Jones, ORI deputy director and associate director for research and scientific integrity. Jones has led the agency during times of vacancies before being named to her positions permanently in 2019.
ORI: New Format Common in Government
ORI, or its precursor agency, published calendar-year annual reports beginning in the 1990s, according to Price. The last one that ORI issued before the FY 2020 version covered 2012. That report, which ORI provided upon request to RRC, is 74 pages and includes 10 tables—with some data going back to 1993.
According to responses to questions from RRC, ORI leaders “chose not to compile annual reports” following the 2012 version and put together the new one after they “made the annual reports a priority in part to meet the needs of the community.”
All the previous annual reports were for calendar years, making comparisons with past data problematic. The new report switched to a FY “to provide consistent reporting with our programs and budgets which operate on a fiscal calendar. Trends are fairly similar across any 12-month period, so this move reduces confusion over the numbers, and ensures we are citing one set of numbers,” ORI said.
The new report is graphics-heavy and, at fewer than a dozen pages, far shorter than previous ones. RRC asked ORI about the new format and what the basis was for what was included and what was omitted in the new report.
“Individuals consume data and information differently. This is a consistent trend with how public-facing products across the federal government are displayed,” the agency said, adding the report “comports” with 2018’s 21st Century Integrated Digital Experience Act.
RRC also asked why the previous annual reports disappeared. “ORI removed the annual reports during a website update,” the agency said, without noting when this occurred. “At this time, ORI does not have plans to re-post the Annual Reports pre-dating FY 2020. ORI intends to continue processing and analyzing the raw data related to case and allegation closures from FY 2012 through FY 2019. When the analysis is complete, ORI will publicly report the results.”
The agency added that information “about the specific case findings, regulations, processes, case studies, and other content of that type found in the old reports remains publicly available on our website.”
However, not all of the information in previous reports is posted, nor is it complete. For example, ORI removes misconduct case summaries, which are listed by the year in which they were concluded, once the individual’s sanction period has ended.
Report Context, Content Lacking
The new report provides no description of the agency, its history or mission. Some of the data are presented in a confusing way, particularly because there is little text to provide context or explanations. The report proper begins on page three of the 11-page report, in a section titled “ORI Investigative Oversight.” ORI states that in FY 2020, it had 35 “cases continued,” received 204 allegations, had 34 “cases opened,” and 72 “total closures.”
Although not specified clearly in the report, the 72 were not all actual investigated cases that ORI handled. Twenty-seven of the 72 were closed at the allegation stage, according to a footnote in the middle of that page.
Of the remaining 45 cases that were closed after being investigated by ORI, there were, as noted previously, 11 misconduct findings. There were eight cases of “no misconduct” and 26 cases that ORI declined to pursue (DTP). The new report does not define these categories, and the 2012 report did not use the phrase DTP.
However, Jones, writing in ORI’s fall 2018 newsletter[3] as the agency’s interim director, explained that these are cases for which institutions have made a finding but which ORI determines it will not advance to further action—essentially, it drops them on the federal level. Federal misconduct findings can result in investigators being debarred from receiving federal funding or being placed under supervision. ORI can also order that retractions of offending publications be undertaken. Cases that are dropped, according to Jones at least, are a source of frustration for ORI and do not mean that an institution cannot move forward with sanctions based on its own policies or that the individual at issue was cleared of wrongdoing.
“ORI describes closures four ways: (1) with findings as prescribed under 42 C.F.R. Part 93, (2) DTP, (3) no misconduct, and (4) administrative closures,” Jones wrote in 2018. “A no misconduct closure may still have issues of concern, as do many DTP closures. We do not always view these as ‘exoneration’ of the respondents. In fact, an ORI finding or settlement does not affect institutional findings or administrative actions based on an institution’s internal standards of conduct, as specified at 93.319(b).”
Past Reports Were Highly Detailed
The 2012 report said “DIO’s review process involved opening 41 new cases, closing 35, and carrying 45 cases into 2013. The number of open cases is the highest number in 16 years.” In 2012 there were 51 open cases.[4]
Also in 2012, ORI received 423 allegations; of these, 116 came directly to the agency and 19 were initially made to NIH. The 2012 report also said the total number reflected “an increase of 56 percent over the 240 allegations handled in 2011, and well above the 1992-2007 average of 198.”
As noted, the new report states that in FY 2020, ORI received 204 allegations, but it provides no information about how they came to the agency. Asked by ORI to define the source, the agency gave no specifics but said allegations come “via a variety of sources. Most allegations were received from complainants or institutional RIOs, via the ‘Ask ORI’ resource mailbox, and other federal agencies,” ORI said.
When asked by RRC how these numbers compare to the 2013-2019 years, ORI said officials are “currently analyzing the data and will make it available to the public once the analysis is complete.”
Nature of Compliance Reviews Unknown
Similarly, the new report uses only graphics to address ORI’s compliance review activities in FY 2020, stating that 32 cases were “handled.” Fifteen cases were carried over from a previous year; 17 new cases were presumably opened, although this is not stated; 12 cases were closed; and 20 cases were carried over into FY 2021. No information is provided as to what these cases were about specifically or what the outcomes were of the closed cases.
In contrast, the 2012 report contains lengthy descriptions for nine cases for which ORI conducted a compliance review, which is separate from a misconduct investigation. ORI explained in the 2012 report that “institutional policies and procedures are routinely examined as part of DIO’s oversight review of institutional inquiries and investigations. Any shortcomings in the process of addressing allegations of research misconduct are identified, and recommendations for corrective action frequently follow,” it said.
Compliance reviews were initiated in 2012 for instances of retaliation against individuals who raised misconduct allegations and in a case where ORI determined an institution had “improper sequestration of evidence, possible conflicts of interest, inadequate interview procedures, and failure to pursue additional leads.” With ORI’s help, the institution developed a corrective action plan, which included reporting and monitoring by ORI of how it handled cases over a two-year period.
Institutional Data Were Omitted
Related information is also lacking. Unlike previous reports, ORI did not include data from institutions on their own misconduct actions, details that are valuable for both RIOs and others to learn what their colleagues are experiencing, reveal the volume of misconduct problems nationwide and give insight into how ORI responds to institutional reports.
As the 2012 report explained, institutions “are required to report annually their research activity for the prior year to ORI.” In that report, ORI noted that “6,714 funded institutions reported 323 allegations, inquiries, or investigations,” and said this reflected “what institutions submitted in their 2011 Annual Report, which is submitted to ORI in 2012.”
RRC asked ORI why this information was not included in the new report. “At the end of calendar year 2019, 5,126 institutions had filed active assurances,” ORI said. “Unlike our case data, assurances accrue on a calendar year basis, so 2020 filings are in process. Institutions have until April 30th to report their calendar year 2020 numbers, and we expect to complete analyses early in 2022.”
However, it is reasonable to assume that ORI could have included the most recent data it had for this category rather than leaving it out entirely.
‘Where’s the Substance?’
Price worked for ORI for 17 years, holding positions of scientist-investigator, division director and DIO associate director, among others, before starting a consulting business in 2006. His duties included working on annual reports.
“I was surprised and grateful that [the annual report] was at least back and visible” online, said Price, adding that he “missed” seeing the reports from 2013 until now.
But he was “struck” by the new FY 2020 format, saying “maybe more people might read a short version but then they may say, ‘Where’s the substance, the analysis?’”
Price noted the absence of trend data, including basic finding totals per year. An annual report “doesn’t have to list all the details” about cases with findings, he said, but he would like to see, along with the individual’s name, degree, institution, type of finding—be it fabrication, falsification or plagiarism—whether retractions or corrections occurred or were requested, and what administrative actions were imposed.
“I think the newer RIOs, in particular, would benefit from the longer explanations of ORI’s handling processes and examples and data that [were included] for years,” he said. “There are constantly new RIOs coming on board,” and some have no background in handling misconduct.
Contact Price at resmisccons@researchmisconductconsultant.com.