This fact was driven home for me with a recent publication. Several weeks ago an article was published in Critical Care Medicine entitled "Etomidate is associated with mortality and adrenal insufficiency in sepsis: A meta-analysis."
The point of this post is not to debate if etomidate should be used to intubate septic patients. Etomidate very well may kill people with sepsis. I just don't know from the data currently available. Using this meta-analysis as an example, the goal is to point out two important areas where we could stand to sharpen our literature evaluation skills.
Point #1: Choose (and interpret) your titles wisely.
It is an overwhelming task to skim through several journals' Table of Contents each month. In a specialty such as Emergency Medicine, many relevant articles appear in non-EM journals making it even more challenging. It's tempting to think we know what an article concluded, based solely on its title.
This point particularly applies to those who publish. But readers also should use extreme caution if only reading titles and abstracts. Given that the last several articles on this topic found that etomidate did not increase mortality when given as an induction agent to septic patients, I was quite surprised to see this bold title declaring etomidate is associated with mortality. We're all so busy that it would be very easy to simply see this title and assume it to be true, without ever reading the article. That is very dangerous medicine, in my opinion. And, this principle extends far beyond this one meta-analysis.
I've already seen etomidate avoided in a hypotensive, septic patient based on this article. I've also heard colleagues giving a quick summary of the article to students and residents saying this article "confirms what we already knew." What?!? When did we definitively "know" this? I still can't believe a highly regarded journal such as Critical Care Medicine would allow this article to be published with this title.
Point #2: The meta-analysis is not the end-all-be-all of publications.
We've all sat through some sort of literature evaluation class back in school. When the meta-analysis was described to me as a student, I remember thinking how awesome it was. Let me get this straight... people way smarter than me are going to take all of the articles published on a given topic, perform some fancy (way over my head) statistics, and give us an evidence-based conclusion? Sign me up. Coming out of pharmacy school, I pretty much thought meta-analyses were the cream of the crop when it came to the published literature. How wrong I was.
I shouldn't have to go back and analyze each of the articles the authors used, but that is exactly what I did in this case. Here is what I found:
With regard to mortality, 5 trials were included. The 4 smaller ones mostly demonstrated that etomidate did not increase mortality compared to other agents. However, the one larger trial encompassing 499 of the 865 total patients (58%) did show an increase in mortality. It was published by Cuthbertson, et al in Intensive Care Medicine in 2009.
Let's take a closer look at this ICM study.
It was such a large contributor to the meta-analysis outcome, it seems important to understand what that trial was all about. Despite the authors calling it an a-priori sub study of the CORTICUS trial, it was actually a post-hoc analysis looking at etomidate's association with mortality. You can read the two published commentaries to the Cuthbertson study by Pallin and Andrade, which each highlight several major issues with the data in this trial.
The bottom line is that the trial by Cuthbertson was highly flawed and really doesn't give us any insight as to etomidate's contribution to mortality. In fact, one of the biggest critiques was that physicians in the CORTICUS trial were instructed to avoid etomidate due to its propensity to suppress cortisol production. So, when physicians did use it, there was likely a reason for it (ie, the patient was hemodynamically unstable and they didn't have many other good induction agent options). Therefore, etomidate was probably given to the sicker patients already more likely to die from the start.
If you dig even deeper, you'll find that the Cuthbertson group used two logistical regression models. One showed a nonsignificant increase in mortality while the other showed a significant increase. Of course the statistically significant one was reported in the abstract. The bottom line is that if you use bad data to construct a meta-analysis, you'll end up with a bad meta-analysis.
So where does this leave us?
In part, it means we have to remain as skeptical as ever when reading published articles. We already know titles and abstracts don't give the full picture. Taking into account reporting biases, funding sources, and even authors' personal/professional agendas, it seems we can't always rely on the peer-review process to uphold the highest standards of integrity. The best journals out there aren't immune. One reason I love Free Open Access Meducation (FOAMed) is that the peer-review process is instant and no holds barred. If you post something that is inaccurate or controversial on Twitter or a medical education blog, you will get called out on it. The best part is that the ensuing conversations inevitably lead to knowledge sharing and learning. Isn't that what research is supposed to be about after all?
Dr. Joe Lex said it best on Twitter:
I couldn't agree more.
References:
- Chan CM, et al. Etomidate is associated with mortality and adrenal insufficiency: A meta-analysis. Crit Care Med 2012;40(11):2945-53. [PMID 22971586]
- Cuthbertson BH, et al. The effects of etomidate on adrenal responsiveness and mortality in patients with septic shock. Intensive Care Med 2009;35(11):1868-76. [PMID 19652948]
- Pallin DJ, Walls RM. The safety of single-dose etomidate. Intensive Care Med 2010;36(7):1268-70. [PMIS 20405278]
- Andrade FM. Is etomidate really that bad in septic patients? Intensive Care Med 2010;36(7):1266-70. [PMID 20405279]
Thanks for this thought-provoking post. I'm Peds Emergency Medicine faculty and the evidence for etomidate and adrenal suppression in kids is even (much) less air-tight. Nonetheless, the AHA in the 2010 resuscitation guidelines recommended against it. It's not that Ketamine is a bad drug for RSI induction, but dogmatization of hot button issues in our culture has certainly spread to medicine.
ReplyDeleteWow, that indeed is unfortunate. Here's the no-etomidate in sepsis statement:
Deletehttp://circ.ahajournals.org/content/122/18_suppl_3/S876.full
Bryan-this is a wonderful post, but I am not sure I am seeing this issue the same way. The authors were quite judicious and appropriate in their article title. They implied that etomidate is associated with increased mortality. They did not state it caused increased mortality. The association is probably true, but it may not be a causal link. If the authors had stated etomidate results in increased mortality, then I could understand your ire.
ReplyDeleteThe second point is the purpose of a meta-analysis such as this one. I have to imagine that the authors saw the same thing most of us did--in almost all of the studies that did not generate sig. p-values on etomidate in sepsis mortality, the trend almost always was on the increased mortality side. When you see a recurrent trend like that, then a meta-analysis is appropriate to pool the data. You can no longer say 4 of the 5 trials were negative--that is why the meta-analysis was done in the first place.
The take home message from this study is not to sway practice or to say that we must stop using etomidate in sepsis, nor do I think the title reflects that. The take home message is that those who believe etomidate is safe should not believe it. Those who believe etomidate increases mortality should not believe it. Both groups should look at this article and be in a state of clinical equipoise. This equipoise will allow a well-crafted, adequately powered randomized trial, which if done properly and if it is not ruined by IRB restrictions will actually answer the question of whether etomidate causes increased mortality in sepsis.
Thanks, Scott, for your comments. I guess we'll have to agree to disagree on this one.
DeleteThe take home point regarding the title was that titles such as this invite conclusion-forming bias without even reading the article. For a busy practitioner, why bother critically evaluating the article if it's right there in the title. In my opinion, the choice of using "associated" or "causes" really is about semantics. They are clearly different, but many readers may draw the same conclusion regardless of which is used. Although the authors do acknowledge some limitations, they state in the conclusion, "that clinicians must be more selective about administering etomidate to facilitate sedation for intubation of the paper clearly indicate." That seems like an implication for possibly causality that could sway practice. I’ve already seen first hand when it has.
I do agree that the question warranted a meta-analysis. This looks like one that maybe should have been scratched early in the process considering the lack of good data available.
Of the 5 trials evaluating mortality, here is what we know:
Trial 1: 106 pts - mortality lower with etomidate (38% vs 43.7%)
Trial 2: 76 pts - mortality higher with etomidate (41.5% vs 34.3%)
Trial 3: 122 pts - mortality lower with etomidate (36% vs 43%)
Trial 4: 62 pts - mortality higher with etomidate (91% vs 84%)
Trial 5: 499 pts - mortality higher with etomidate (42.7% vs 30%)
Trials 1-3 are pretty much a wash when added together. Trial 4 is obviously different than the rest with its very high overall mortality rate. Trial 5 by Cuthbertson clearly affects the overall numbers. If you remove this trial with its many major issues, you pretty much have no "association" with mortality.
From a mechanistic standpoint, if etomidate is "associated" with mortality why didn’t patients who received hydrocortisone + etomidate fare no differently than those receiving etomidate alone (in the CORTICUS trial)? Further, etomidate suppresses cortisol production for 48-72 hours. Why has there been no difference in mortality demonstrated within that time frame? It is always at least 10 days after etomidate administration when patients seem to die more.
I am still an advocate for ketamine for intubating septic patients, but not because etomidate is “associated” with mortality. I hold firm in my conclusions that choosing titles is of vital importance and that bad data in = bad data out. Your conclusion is right on. On this topic, at least, no one should believe the data that supports their stance. Looking forward to future debates.
Excellent discussion. I must admit that I would have been swayed by the word "association" to mean "causal" effect if Bryan hadn't done all the hard work in reviewing this article for me.
DeleteBryan-the difference between association and causation may be semantics in the common vernacular, but the meanings are incredibly divergent in the worlds of research and EBM. This is one of the first things our residents are taught when we give EBM lectures.
DeleteI believe you flipped the mortality in the one you listed as trial 3, which I believe is this trial (PMID:20828877). It was etomidate that had the higher mortality (43%). Of note this trial was an RCT. So 4 of the 5 trials showed a higher mortality with etomidate. The only one that didn't was a non-randomized trial that had no multi-variate analysis for the all of the confounders that you nicely elucidated in your post.
Are you a bit less doubting of EBM?
Scott-completely agree with you about the divergent meanings between association and causation. My feeling, and experience, are that although they are vastly different, many graduated medical professionals (pharmacists, nurses, physicians, etc.) aren't trained to identify these differences. This is why title choices are so important. To you and me, associated vs causes represent two very different meanings. Without the proper literature evaluation training, however, associated might as well mean causes.
DeleteThat being said, very interesting point you bring up about Trial 3. I took my stats from Table 1 of the meta-analysis. The meta-analysis table states that etomidate had a lower mortality than the comparison group (36% vs 43%), although not statistically significant. However, if you pull the original study, those numbers are actually reversed. In the original study, etomidate showed a nonsignificant increase in mortality compared to midazolam (43% vs 36%).
Not sure what to do with that second point, now. It may restore a tiny bit of my faith in EBM, but further proves my point of the need for in depth analysis of anything published (even in highly regarded journals). I only pulled the Cuthbertson trial for further analysis. At least it looks like the meta-analysis authors used the correct numbers in their pooled relative risk calculations (not what is stated in the table). Although the results of the small Tekwani 2010 RCT favored midazolam, there weren't enough patients to show a statistically significant result.
The Cuthbertson trial still clearly had a major influence (58% of patients) on the meta-analysis conclusion with its significant flaws. I hope the proper trial is performed at some point since etomidate does seem to cause issues, even if a clear reason hasn't been established.
Happy Thanksgiving.
Hi Bryan,
ReplyDeleteExcellent comments, interestingly enough I just blogged about this article few minutes ago. Fortunately, we came to the same conclusion.
I don't believe this meta-analysis writes the last word on the Etomidate and sepsis saga; just as Scott said, it invites for a well design RTC with adequate power to finally answer the question. In the mean time, when I see my next severe sepsis patient, I will think about it (20 seconds max) and then choose the agent that best fits the patient's profile. It may be Etomidate or Ketamine or whatever.
Great to hear @Aramd. Would love to see your post. What's your blog's URL?
DeleteBryan, great read. Two thoughts come to mind: 1) Professional schools and residencies must provide effective training in critical reading and biostats, and 2) To make the most of FOAM the pool of active participants must be increased.
ReplyDeleteAgreed on both points. I think there are actually a lot more participants reading blogs and lurking on Twitter than we realize. The trick is to make them more engaged and active on these public forums.
DeleteThe most important point you made is that readers must be critical of the articles they read, and they must have the necessary skills and tools to evaluate them - otherwise don't bother.
ReplyDeleteIn this particular example, the first step is to understand what the purpose of systematic reviews is: it is to scan and summarize available evidence on a given topic. Just because someone performs a meta-analysis, it doesn't mean it is the highest quality evidence. Users of literature need to know how to assess the methodological quality of a systematic review before even delving into their summaries and conclusions...just like how some individual studies are more methodologically sound, there is huge variation in the quality of systematic reviews. (for a good tool use AMSTAR for assessing your next systematic review)
Second, even the most methodologically sound systematic reviews can only report on what the current evidence is. The authors did nothing wrong here - they took what crappy evidence there is to date, and summarized their findings as best they could. The problem is, any conclusions made in a systematic review is only as good as the original studies. Garbage in, garbage out.
Having said that, you can complain all you want about how crappy the evidence is, but the fact remains: as far as we can tell based on what limited data we do have, there seems to be associated mortality with etomidate use in sepsis. It is just as negligent (or perhaps even more so based on available data) to suggest etomidate is safe in this scenario.
Sometimes we just do not have the perfect clear cut data and answers we want, but it has to start somewhere. Users of literature MUST obtain the training and tools needed to understand such nuances, before attempting to quote "up to date evidence".
Thanks for your comments. I agree your points in the first 3 paragraphs. In response to your comments in paragraph 4, I want to be clear that I never implied that etomidate is safe in sepsis. To quote a point from my post: "Etomidate very well may kill people with sepsis. I just don't know from the data currently available."
DeleteThe largest trial (Cuthbertson) in support of etomidate's effect on mortality was so flawed with some important confounders, it is probably not reasonable to conclude etomidate definitively contributed to the results. The trial used etomidate only in the sickest patients more likely to die from the start. If that hadn't been the case, the trial may very well have shown no increased mortality from etomidate.
Further, the current theories we have as to why it increases mortality don't seem to hold up to the evidence. From a mechanistic standpoint, if etomidate is associated with mortality, why didn’t patients who received hydrocortisone + etomidate fare no differently than those receiving etomidate alone (in the CORTICUS trial)? Further, etomidate suppresses cortisol production for 48-72 hours. Why has there been no difference in mortality demonstrated within that time frame? It is always at least 10 days after etomidate administration when patients seem to die more.
I hope the proper trial is performed at some point since etomidate does seem to cause issues, even if a clear reason hasn't been established.
I also agree with the tweet. Most of evidence are based on data. Various kinds of data needed to prove a theory, that is why it is right to say "Data Based" rather than "Evidence Based".
ReplyDelete