Editor’s note: After this story was published, we learned that we mistakenly counted some studies as violators of the reporting law because they were not included on a list of exempt studies provided to STAT by the National Institutes of Health. Before publication, NIH had described this list as a complete accounting of exempt trials. But an agency official subsequently told us that organizations running studies completed before Jan. 18, 2017 — when new ClinicalTrials.gov regulations went into effect — could delay reporting the results without seeking a formal exemption, if the drug being tested had not been approved by the Food and Drug Administration. For such studies, requesting a formal “certification for delay” was optional, the official said. Unfortunately, this means that for many older trials, there is no practical way for STAT or anyone else to verify which ones are exempt, but this story and the accompanying data visualization remain the most accurate assessment possible. Based on this new information, we have created a separate list of trials that sponsors say were unapproved and thus exempt, and have linked to any documentation provided by the sponsors. STAT has not verified these claims. Sponsors can submit information here. Thankfully, this transparency loophole has been partly fixed under the new rules, and for trials completed since Jan. 18, 2017, you can check this TrialsTracker website to determine whether the results are overdue. (The details of that site’s methodology are here.) You can also read a detailed description of the methodology STAT used to analyze the ClinicalTrials.gov database here.
The reporting of clinical trial results to a public database — mandated by a 10-year-old federal law — has improved sharply in the last two years, with universities and other nonprofit research centers leading the way, according to a new STAT analysis of government data.
Overall, trial sponsors had disclosed 72 percent of required results to the federal ClinicalTrials.gov database as of September 2017. That compares with 58 percent just two years earlier.
Unreported research data can compromise the safety of human research and slow the pace of innovation, and the increase coincided with sustained attention to the problem by advocates for transparency in science, the media — and even then-Vice President Joe Biden.
Especially striking were the increases at major research universities, which conduct the vast majority of academic trials: Their reporting jumped from 58 percent to 77 percent of trials. The National Institutes of Health and top recipients of its funds did even better, a sign that leaders in academic medicine are finally taking seriously the 2007 law that requires reporting the results of most human studies of medical treatments. Drug and device companies improved more modestly.
For its analysis, STAT examined nearly 18,700 clinical trials whose results — based on publicly available data — should have been posted to ClinicalTrials.gov between 2008 and Sept. 11, 2017. The investigation follows a similar report by STAT two years ago, which found that prestigious medical research institutions were among the most flagrant violators of the reporting law.
Many of the biggest gains were at research institutions singled out for woeful reporting in the earlier STAT investigation, including Memorial Sloan Kettering Cancer Center, the University of Pittsburgh, and Stanford University. Such formerly routine violators improved, on average, from reporting the results of just 35 percent of studies to 76 percent.
“We put everyone on notice that we consider this an extremely high priority,” even for busy and sometimes-recalcitrant faculty, said Dr. Mark Cullen, senior associate dean for research at Stanford Medical School.
STAT was the first to examine the reporting track records of individual institutions, and the improvement by the worst offenders suggests that naming names drove better reporting, experts said.
That investigation and assessments by other groups “shows the power of public accountability … for fixing what is probably the single most important flaw in the fabric of evidence-based medicine,” said Dr. Ben Goldacre, a fellow at the University of Oxford in the U.K. and a founder of projects to press for openness in clinical research.
Clinical trial reporting violations, 2008-2017
The NIH is the largest funder of academic biomedical research in the U.S., and it also runs the clinical trials database. In a written response to questions, the agency attributed some of the improvements to clarifications made in its final rule for ClinicalTrials.gov, which took effect in January 2017, and to NIH efforts to educate researchers.
The findings weren’t all positive. Results for 4 in every 10 trials were reported after the legal deadline — one year following a trial’s completion or termination. (Some of these late trials, however, may have been exempt from reporting because the drug being tested had not been approved by the Food and Drug Administration, or it was being tested for a new indication. NIH provided STAT a list of trials that had been granted a reporting delay, but after this story was published, it said the list might not have included all exempted studies.)
Valeant Pharmaceuticals, Albert Einstein School of Medicine, the University of Nebraska, and the University of California, Irvine, for example, continued to flagrantly violate the law. Each has failed to report even 20 percent of its trials within the legal deadlines.
In written statements, Valeant said it was working to provide results to ClinicalTrials.gov, or seeking approvals for delays from the Food and Drug Administration, and Albert Einstein said it would hire a specialist to “ensure that our researchers will come into compliance.” Nebraska said it had dedicated resources to compliance and vowed to “prioritize this issue going forward,” and UC Irvine said it had recently taken “numerous steps to improve compliance” and planned to “meet or exceed the best practices” going forward.
Millions of patients and doctors use ClinicalTrials.gov to find trials, compare the effectiveness of approved and experimental drugs and devices, and survey sometimes-deadly side effects. The legal requirement to submit results began in 2008. It stemmed from many cases in which companies allegedly hid negative clinical evidence about highly profitable drugs — such as links to suicidal thoughts by teenaged users of GlaxoSmithKline’s blockbuster antidepressant Paxil.
Designed as a universal, free, and open repository of human studies, ClinicalTrials.gov was meant to compensate for deficiencies in the scholarly literature. Journals publish results selectively and in hard-to-compare formats, and often charge subscription fees that patients and doctors can’t afford. Results on ClinicalTrials.gov also honor the sacrifice of trial volunteers and help researchers plan safer, more promising follow-up studies, advocates say.
National and global public health leaders reacted to STAT’s 2015 findings with anger, as did Biden, who threatened to slash federal research funds to law-breaking recipients of government grants.
Jennifer Miller, a New York University professor who rates pharmaceutical companies’ transparency for trials on approved drugs, credited the acceleration in compliance partly to pressure by U.S. and global health authorities, as well as to journalists, scholars, and independent watchdog groups “measuring and benchmarking the process. And what gets measured often gets done.”
A new priority on disclosure
In 2015, drug and device makers, on the whole, easily outperformed universities in reporting results. The difference narrowed sharply by 2017, although both categories showed improvement overall.
Many universities and nonprofit research organizations identified as major offenders by STAT in 2015 backfilled large gaps in ClinicalTrials.gov, belying prior complaints that the system was unduly complex or burdensome.
In 2015, Stanford had submitted results for only one-third of trials due — partly, an official said, because professors lacked the time and money to comply. By 2017, it had submitted results for 85 percent of its trials.
The university learned from STAT’s earlier investigation that “we had a serious problem,” Cullen said. Soon after, it hired six full-time employees to train and coach faculty for compliance with ClinicalTrials.gov and FDA rules.
Memorial Sloan Kettering went from providing just 20 percent of its required results in 2015 to 98 percent as of 2017. Dr. Paul Sabbatini, one of the center’s top clinical research executives, credited the addition of two dedicated reporting specialists after STAT’s coverage of his institution’s poor record.
“Everyone [now] understands that this push towards transparency is important,” he said. “We hopefully would have come up to speed anyway, but shining a light on it made it happen in a much easier and faster way.”
Stellar results came from a few drug companies, including Belgium-based UCB — the top performer of any leading organization. It was among a handful of top research entities that posted all required trial results. And unlike most others, UCB did so within the legal time limit in the vast majority of cases. But these good performers were offset by lagging drug makers such as Valeant, whose record actually worsened from 2015 to 2017.
Unreported results, unknown costs
While there has been progress, the thousands of trials with still-undisclosed results can have serious public health implications.
For example, Dr. Andrew Chang of Albert Einstein has conducted seven trials examining how to calibrate the use of opioids among hospitalized patients — timely and potentially vital information as the nation confronts an intractable addiction crisis. His results are up to seven years overdue.
Chang’s experiments tested approaches managing acute pain — such as providing opioids on demand versus according to a set time frame, and at what dose opioids can cause blood oxygen to fall to potentially harmful levels.
Two studies are freely accessible. Others were never published, or the publication status is unclear because it was not possible to match cryptic published abstracts to trial registrations on ClinicalTrials.gov. The full articles in these cases were protected behind paywalls — an obstacle the reporting law was meant to solve.
Chang and representatives of Albany Medical College, where he also works, did not respond to repeated requests for comment. Albert Einstein declined to comment on Chang’s work.
In 2008, Dr. Joan Daughton of the University of Nebraska began a trial to test the safety and effectiveness of the long-term use of the stimulant Ritalin in 4- and 5-year-old children suffering from attention deficit hyperactivity disorder. But Daughton withheld results from the long-completed trial, even as concerns about Ritalin have grown. Last year, a study by other researchers showed significant, worrisome changes in an important neurotransmitter in the brains of adults who had taken Ritalin as young children — suggesting that such patients could be “more vulnerable for risk-taking behavior” and drug abuse as adults.
Daughton, now at Children’s Hospital & Medical Center Omaha, declined to respond to questions about her research findings and failure to disclose the data to ClinicalTrials.gov or in a scholarly journal. After being contacted by STAT, she entered the study data into ClinicalTrials.gov; they have not yet been publicly posted, pending quality review by the NIH.
‘A really toxic notion’
Two years ago, the NIH and the FDA said they hadn’t penalized violators of the ClinicalTrials.gov reporting provisions because no final rule was in place.
The law gives the agencies authority to levy financial penalties — withholding grants, or in the case of drug companies, imposing fines of up to $10,000 a day for late results. In theory, the FDA could have brought in more than $40 billion from corporate violators for violations as of September 2017, STAT calculated.
That final rule has now been adopted, but in effect, it retroactively forgives any violation prior to its reporting requirements going into full force on Jan. 18 this year. And the agencies have not expanded their enforcement staffs to monitor compliance.
The new rule clarifies definitions meant to help researchers navigate data requirements. It also requires entry of certain previously optional information, such as why some studies are stopped prior to completion, descriptions of experimental interventions, and trial start dates.
In response to questions, FDA spokeswoman Lauren Smith Dyer issued a written statement that the agency would integrate ClinicalTrials.gov compliance and enforcement into its current monitoring program, which conducts more than 100 inspections of research organizations annually. The agency will not add compliance staff for that extra work, she said.
The NIH said in its statement that if a grantee fails to report trial results as required, “NIH may take an action up to wholly or partly suspending or terminating the grant.” Like the FDA, it will add no staff to police ClinicalTrials.gov reporting, and currently has no dedicated compliance staff.
The NIH faces an additional challenge. Beginning this year, the agency’s internal policies require all researchers it supports to report results on trials beyond those for drugs, medical devices, and vaccines. These include observational studies, examinations of nutritional supplements, and early, Phase 1 safety trials. As of 2017, nearly 800 such experiments had been completed for more than a year and would have been subject to this new reporting requirement.
Goldacre called the lack of new enforcement resources “extraordinary, and extremely concerning.” He said it also lets researchers off the hook for unethical practices.
“It’s a really toxic notion that’s taken root in the minds of many senior academics, trialists, and university administrators, that their trial reporting obligations should be set by the law rather than the very simple, unambiguous, and obvious obligation that you must report the results of all trials,” he said.
Goldacre’s research group at Oxford plans to launch a real-time tracking website on Jan. 18 to flag trials whose results are past due.
“In the absence of meaningful work from the FDA to police compliance,” he said, “the only thing left is public accountability.”
Correction: The story has been updated to note that an NIH list of trials exempt from reporting results to ClinicalTrials.gov may not include all studies that qualify for exemptions. The graphics with this story have been updated to correct the data for Beth Israel Deaconess Medical Center in Boston. It had the largest improvement in reporting clinical trial results between 2015 and 2017, a 493 percent increase. In addition, six trials were wrongly attributed to Shire instead of Baxter.
Scott A, Rucklidge JJ, Mulder RT. Is mandatory prospective trial registration working to prevent publication of unregistered trials and selective outcome reporting? An observational study of five psychiatry journals that mandate prospective clinical trial registration. PLoS One. 2015;10(8):e0133718.
How does one convert the data you provided in the link? In its current format, it’s useless.
In the spirit of transparency, release your entire database in Excel publicly. Hovering over thousands of trial ‘circles’ to amass the information is onerous.
All of the data is already publicly available here https://apps.statnews.com/ct17big/public/data/ct17.json
I’ve released this clinicaltrials.gov data set for free use on https://trialtap.com
Might be more useful if you don’t care to play with the JSON data that Natalia sent and instead just want to use excel.
This is a complete download from clinicaltrials.gov. The public should be able to access the exact data set that the authors used in this article.
Again asking the authors to post the exact data set in excel.
In the spirit of transparency, please do not encourage/promote usage of any closed source (proprietary) commercial software. Python and R are free and open source.
This “article” reports to relate to “legal compliance” yet seems to ignore many aspects of the law.
This “article” disregards that many studies that are registered voluntarily on CT.gov do not need to have results disclosed per the LAW (as there were not applicable trials) and in these cases results from these studies are instead posted on other registers, company or institution websites, or are published in journals. Also study results posting is not required by LAW until approval (actually 30 days after approval) for a given indication, this too is ignored by this “article”. By my estimate (and those of other researchers such as Jennifer Miller) companies are more like 98% compliant with the law while this article is reporting 30-50 compliant in many cases. In fact this article is attacking those companies and institutions that historically went “above and beyond” legal requirement by registering additional studies voluntarily (or to meet journal requirements)…
In an era of fake news and pressures on RESPONSIBLE free press this is dangerous sensationalism based on lack of rigor to actual legal requirements and FACTS. If this article is indeed about companies and academic institutions LEGAL compliance then it should have the scientific rigor to evaluate vs the legal requirement. Please focus on the law and look more deeply in those areas where of course we should all be concerned.
All those below applauding this article may need to dig a little deeper and hold your applause.
We filtered the data to remove any trials whose findings were not required by law to be reported.
Please provide an example (NCT#) of at least 1 trial presented on the data visualization that was not required by law to be posted.
There are also some non-public factors at play. For instance if the Sponsor delayed results posting, they may not be considered late, but STAT wouldn’t know that. I would imagine a disclaimer should at least be footnoted…
Yes, Brice, some datapoints were not available for download from ClinicalTrials.gov – like data that would have allowed us to exclude from the violator’s list, studies that had received an official reporting extension. To eventually receive this data, STAT filed a separate request with ClinicalTrials.gov. We are preparing a more detailed explanation of our data analysis and a step-by-step guide to using our full dataset with links to some additional spreadsheets, which will be publishing shortly.
Thank you, STAT, for this tremendous public service. The power of the free press…squared!
This is journalism at its very best. Congratulations!
It would be fantastic if you could also link the full results in Excel or similar format for downloading.
Good research and good article. Are there, or do there need to be, fines, sanctions, repayment of grant dollars with interest, or even stiffer penalties? Taking government research funding and not delivering the required reporting sounds like a breach of contract or even a form of graft. This does not pass muster in the corporate or nonprofit sector, and should not pass muster in government.
Comments are closed.