Former NEJM Editor Criticizes Publication and Peer Review of ARISTOTLE Trial

Editor’s note: The following guest post is reprinted with permission from the blog of Alison Bass. Bass is the author of Side Effects: A Prosecutor, a Whistleblower and a Bestselling Antidepressant on Trial, and was a longtime medical and science writer for The Boston Globe.

New heart medication study was too flawed for publication, former journal editor says

by Alison Bass

At a Harvard event last night honoring the former Sen. Grassley investigator Paul Thacker, someone in the audience wanted to know how the topic of Thacker’s talk — Dollars for Doctors: Who owns your physician? — was related to the soaring cost of medical care in this country.

As Thacker noted at the outset of his talk, Medicare and Medicaid are now a larger portion of the federal budget than the Pentagon and health care now equal 23 percent of all federal expenditures. Part of the problem, he noted, is the fact that our medical system is stacked towards the widespread adoption of expensive new drugs over older generics even when the new agents are not necessarily safer or more effective than cheaper drugs.

It was left to Dr. Arnold Relman, professor emeritus at Harvard Medical School, to showcase a fresh-off-the-page example of how studies that are funded by drug makers and conducted by researchers who have financial ties to the industry present skewed research results that favor expensive new drugs over generics.

His case in point: The New England Journal of Medicine published a study last week concluding that a new anticoagulant known as apixiban (brand name: Eliquis) was superior to the generic drug warfarin in preventing stroke and deaths in patients with atrial fibrillation (abnormal heart rhythm). The study was funded by Bristol Myers Squibb and Pfizer, which jointly manufacture Eliquis, and featured a lengthy roster of authors, many of whom have extensive financial ties to the drug industry (in the form of speaking and consulting fees). At least three of the authors were Bristol Myers Squibb employees, as the fine print at the end of the study disclosed.

The issue Relman, a former editor of he New England Journal, raised was how these financial conflicts may have influenced the way the paper itself was skewed in favor of the new drug. He noted two major omissions in the discussion section of the study. One was the fact that the anticoagulant showed no efficacy over the much cheaper warfarin generic in the 7,000 patients recruited in Europe (this was a multi-center trial involving 18,000 patients from the U.S., Latin America, Asia and Europe). Two, 35 percent of the patients on warfarin were not taking a therapeutic dose of the drug, which, he said, could explain why they had a higher rate of blood clotting and stroke than patients taking the new anticoagulant.

Yet neither of these key limitations were mentioned in the study’s discussion, a glaring omission, according to Relman. He said that the journal itself was remiss in publishing the study without mentioning these limitations.

“This study was not well peer reviewed,” Relman said. “Neither [Dr. Marcia Angell, also a former editor of NEJM] nor I would have accepted this paper for publication.”

Yet as a result of its publication, many heart doctors will now be steered toward prescribing a much more expensive drug when the cheaper generic would do just as well in many cases.

The study, Relman said, is a prime example of why the disclosure of conflicts of interest (which most leading journals now require) is not enough to curb bad or biased science. He suggested that medical institutions simply prohibit their faculty from doing research on drugs when they are receiving lucrative speaking and consulting payments from industry. (Research shows a clear link between a company funding a trial and favorable the results for that company’s drug — see here and here.)

“The real solution is for medical institutions to get rid of these unethical practices,” Relman said. “Disclosure is not a solution.”

Thacker agreed. The Physician Payment Sunshine Act, which he helped shepherd into law and requires drug companies to disclose all payments to doctors, should be considered a first step, he said.

“It’s a way to get a handle on just how widespread the problem of financial conflicts of interest are, so we can start to make real changes,” he said.

If the documented harm to patients from such conflicts of interest is not enough of a reason, perhaps the unsustainable costs to our overburdened health care system will eventually tip the balance toward change.

Editor’s Note: Click here for additional discussion about this episode.


  1. Vikas Saini, M.D. says:

    Agree with Relman’s overall position, and the inadequate discussion of the results points to a bias in the INTIAL feeling and tone towards the new drug. Often these aspects require so much space that the discussion occurs in later publications and in talks, etc.

    On the other hand, the chain of logic implied by this post has many points where varying paths can emerge and not all of them necessarily lead to overtreatment and excess costs.

    For example, it’s still very possible that apixiban will be much better than warfarin and better than the other direct thrombin inhibitors and DEPENDING ON PRICE could be a net savings to the system.

    Given what a pain it is to manage warfarin to the target INR, I’m hoping apixiban pans out. For the record — I usually don’t prescribe new drugs in the first year on the market!

    The logical solution to the conflicts issue — given that so much of academic medicine survives on drug company funding — is for pivotal trials to be funded by the companies into an escrow account, but the protocols designed by FDA and independent investigators, and executed by independent clinical trials operations.

  2. Edward J. Schloss MD says:

    I don’t find Relman’s comments very compelling. Having not heard his talk, I’ll respond only to what’s written above.

    1) Relman notes no advantage of apixiban over warfarin in a European subgroup taking the generic form of that drug. He criticized the authors of the trial for not citing this observation. Taking a post hoc subgroup and trying to draw conclusions from this is a classic medical study mistake. This can only be hypothesis generating, not conclusion forming. In addition, there should be some form of a reasonable explanation for the subgroup effect before any hypothesis is drawn. I suspect what is observed here is a statistical fluke, much like the well publicized statistically significant effect of zodiac sign seen in the ISIS-2 thrombolytic trial years ago

    Had the authors of the trial wished to do so, I suspect they could have easily plucked out post hoc subgroups that showed excess benefit of apixiban opposite to what Relman found. Had they done so, a responsible reviewer would have removed this from the paper. Relman’s criticism appears no more valid.

    2) Relman suggest that the Warfarin treatment arm may have exhibited a reduction in efficacy because 35% of patients were not on a therapeutic dose. This criticism appears to refer to the TTR (time in therapeutic range) measure of warfarin dosing. All who wish to respond to Relman’s comments must understand that achieving 65% TTR with warfarin is actually quite good and likely better than what is achieved in the “real world.” Please look up the healthy dialogue on this topic regarding Rivaroxaban and the ROCKET trial on this blog. All would agree that a 65% TTR has been considered “skillful” use of warfarin by the FDA panel. This should be applauded, not criticized.

    I’m all for generics and use them regularly. I’m all for practicing cost effective medicine. I agree with Relman and Thacker on this point.

    Don’t these folks know, however, how badly we need an alternative to warfarin? Although cheap at the pharmacy, this drug has an enormous amount of indirect costs and problems associated with it. A day doesn’t go by in the office that I don’t have to beg my patients to take this drug, lest they remain exposed to the risk of a stroke. Patients absolutely hate warfarin. Beating up on pharma for developing a good alternative to warfarin is like beating up on Santa Claus.

    It feels like “open season” lately on doctors regarding industry relationships. Developing pharmaceuticals and managing a research trial requires skill and time. Qualified individuals will need to be compensated or they will decline to do the work. If we move toward a policy of barring any industry relationships (transparent or opaque) from all researchers, the quality of the output will suffer. Industry cannot develop treatments without input from clinicians who see patients. Clinicians, conversely, cannot develop treatments without industry. No one is going to do this work for free.

    I entirely understand that there have been unreasonable excesses in doctor/industry conflict of interest and that these should be exposed and discouraged. I applaud the work of these blogs and other news media in this effort. Let’s however also celebrate the good in doctor/industry relationships. We all need these strong ties to advance the science and bring good products to market.


    Edward J. Schloss MD
    Cincinnati, OH
    P.S. I have no conflicts to disclose.

  3. Point of clarification: actually, the authors do mention the geographic subgroups in the discussion section. They state: “The results were consistent in subgroups according to geographic region.” I agree that this statement may be misleading because, as shown in figure 2, in the Europe subgroup there were 75 events in the apixaban arm and 77 in the warfarin arm, so basically the same in the two arms. In the other 3 geographic subgroups there was a much larger difference between the two arms. The p value for interaction was 0.44. The authors should have pointed out these varying results. However, I agree with Dr. Schloss that this is probably a statistical blip.

  4. Amen.
    The high cost of medicine is generated to a great deal by us, the docs. The young docs have never heard of older effective drugs, all age docs need an MRI before trying to figure out anything, and all old people from nursing homes are treated like they’re the President.

  5. Ethan J. Weiss says:

    I agree Larry. I think Relman could have cited a lot of other more compelling examples, and I agree with Dr. Schloss (and Marilyn) that the examples above were not terribly important. I also think his choice of warfarin as a generic drug is flawed and neglects how difficult a drug warfarin is for patients and doctors. He neglects to allow that thousands of patients who may have refused or been unable to take warfarin may be helped by the eventual approval of this drug. The TTR issue was very well addressed in your recent reporting on the FDA hearings for rivaroxaban.

    One question I do have about apixiban and the other Xa drugs that has not been adequately addressed by any of the investigators is how we will handle the inevitable concurrent use of anti-platelet agents. That is, with the publication of APPRAISE-2 this past summer, we learned that addition of apixiban to patients on aspirin and or clopidogrel led to significantly more bleeding events (in an ACS setting). Given that many of the patients who would be taking apixiban for Afib might also have stents, this is going to be an important issue going forward. Since the use of anti-platelet agents was quite limited in Aristotle and the other new trials (roughly 1.5%), this important safety question needs to be addressed and should have been discussed in the manuscript (the low use of DAPT). The question is whether the FDA is going to ask the manufacturers to do so. I bet not.

  6. Relman’s comments (if accurately quoted) suggest that he is very sadly out of touch with trials methodology and the content area.

    1. Its not realistic to expect a benefit independently in an 8000 patient subgroup – if such a benefit was expected the trial would have been planned to include only 8000 patients, thereby saving a lot of money.

    2. INR control in the trial was much better than usually encountered in community practice

    3. We are dependent on pharamceutical industry to fund trials (lets face it, world governments cannot afford to do them) and the trials are subject to massive scrutiny.

    Thank you NEJM for publishing this excellent trial

    (i have been closely involved in pharma trials but was not in any way involved in the ARISTOTLE trial)

  7. Uhhhhhh, how ’bout the reduction in all cause mortality with apixiban versus adjusted-dose warfain, Arnie? Surrogates are one thing, but all cause mortality cannot come from any author’s bias. I suspect Arnie needs an oral Xa inhibitor in order to prevent his NEXT stroke!


  1. […] long ago the former NEJM editor Arnold Relman said that NEJM should not have published a recent paper, the ARISTOTLE study, which compared […]

  2. […] is  “no longer possible to believe much of  clinical research published”   and   “The real solution is for medical institutions to get rid of these unethical practices. Disclosure…   Your doctor will claim that his or her treatment recommendation is based on solid […]

Leave a Reply

%d bloggers like this: