So I'm going to do one other talk. It's a discussion of meta-analyses, and we're going to talk about not only what they are and how to evaluate them, but we're going to discuss how to do them. And then right after the break, in an hour or two, we're going to actually do the meta-analyses, and you'll, I think, be persuaded that there's no real magic in these in the sense that they're easy to do. You have the tools in your laptop now on how to do them. Okay? So we're going to review these a little bit. So here's what I'm planning to talk about, and you stop me at any point if I'm not covering stuff or not being clear, but we're going to talk about these systematic reviews and meta-analyses, that's SR and MA, and we're going to talk both about how to appraise other people's systematic reviews and how to do them ourselves, and we'll look at some threats to their reliability the same way we have for single studies, okay? And we're going to go about it the same way that we just did for single randomized control trials, always keeping in mind this hierarchy of evidence, and a lot of people put this meta-analysis at the top, okay? I'm not sure that that's uniformly agreed to, but there's certainly powerful ways to answer questions. Therapy is important, and we've already said this this morning and before. This is not the only feature we use to make decisions and recommend therapy, okay? Our own experience isn't thrown out the window, okay, and neither is patient judgment or values or those things. It's really important to know that the practitioner of evidence-based medicine advocates integrating all of those things. It's not evidence-only medicine. So with that in mind, systematic reviews and meta-analyses are almost the same. They're concise summaries of our best evidence, and they use this very rigorous method that's explicit and transparent so that you can reproduce what other authors have done to collect, appraise, and then synthesize a result from the evidence about a specific question in a way that minimizes both bias and random error, and we'll talk about how that's done. But if they're done right, they're supposed to look at the whole truth, all of the evidence that we have about a particular question, and the only difference between a systematic review and a meta-analysis is that the meta-analyses introduce some quantitative techniques to help us, and that's what we'll do when we do the hands-on stuff, okay? So you fill this table out for me. So with regard to the question, how does a systematic review differ from this sort of narrative review, which I'm using as an example of a book chapter? How are they different with regard to the question? Yeah, exactly. PICO versus FOCUS is how they differ with review theory. So for example, you might have a question about, does prophylactic anticonvulsant medication prevent first seizures in patients with subarachnoid hemorrhages or after craniotomy? That's a very good FOCUS question for a systematic review. You might see a chapter of anticonvulsant management in the neurosurgery patient. So a broad question, a FOCUS question. How about sources and search for the evidence, differences and similarities? So a narrative review, they don't generally tell you. They select the articles. They're these chapters with 200 references, but you don't quite know what the selection or the filter or the criteria were. But it's very comprehensive and explicit in a systematic review. Selection of articles, basically the same answer, right? Very explicit in a systematic review and potentially biased in a chapter, because you have an idea about what the answer to a question is, and you can select the evidence to support that. It's like writing high school English papers, that five-paragraph kind of paper that you have. How about the appraisal of the evidence in a systematic review or in a chapter, let's say? Yeah, just the way that we just did, exactly. And the synthesis is quantitative, if you can, and qualitative within very rigid guidelines in a systematic review. And as a result, the inferences are occasionally evidence-based in a chapter, but always evidence-based in a systematic review. So the whole process is really similar when we're looking at single studies and meta-analyses. And I'm going to suggest that we think about meta-analyses, and one of the groups actually has selected a meta-analysis for their project. Think about a meta-analysis as a clinical trial where your patients are the individual articles or the evidence that you're going to collect for the meta-analysis, and that helps clarify what we're doing. And you can do these meta-analyses for therapy and for diagnosis and prognosis and risk factors that are applicable to any type of question. Okay? So just to kind of review, these systematic reviews compared to a narrative like a chapter limit bias. They're transparent, rigorous, and reproducible. If you do a meta-analysis and I do one on the same question, we ought to get the same answer. So you can check my work. They're comprehensive. Which of these is the most important thing, the thing that meta-analyses really do the best, the reason, in fact, that we do them at all? I mean, all of these are good things, but what thing does meta-analysis give to you that just looking at the individual articles don't? It is comprehensive, but I could look at the articles without meta-analyzing them. I could be comprehensive. It increases the precision. That's what meta-analyses do. They don't make the studies any better. Right? If they're bad studies and you do a meta-analysis, they're still bad studies. They increase your estimate of effect size. They narrow your confidence interval. They increase the precision. Those are all synonyms. And like clinical trials, right, I asked you to think about meta-analyses as clinical trials. They're at risk for random error, bias, and confounding at every step, right? So we'll talk about that, but systematic reviews and meta-analyses have some particular risks of bias that aren't relevant to single studies, and those in particular are publication bias and heterogeneity, and we'll talk about those. And like single trials where we use CONSORT, there are a whole bunch of published, easy-to-access ways to both evaluate the meta-analyses and to produce your own, both the strategy for getting the data and the quantitative aspects. PRISMA is an example. There are a whole bunch, and we'll talk about those, okay? But meta-analyses only do that, and primarily they increase the precision of your effect size estimate. How much does early anticonvulsant therapy in patients with first seizures reduce the risk of seizures from broad to narrow? It increases the precision. Doesn't improve the quality of the evidence that goes into them, okay? But if you do these properly, then you have a document that provides you the best evidence, and if you do them quantitatively, you actually get a number, a number of association, odds ratio or relative risk. You can certainly translate those into number needed to treat that we can make use of in the clinic. So, how do you actually do these? Well, the easiest way is to find someone else who's done them, and you can find these in big databases of meta-analyses, maybe the easiest way. But Medline also provides meta-analyses, and if you're using OVID, there's actually a meta-analysis button that makes it really easy. It's almost as easy if you're using PubMed, this clinical queries page will get you right to meta-analyses, and so you can access those with two clicks instead of one. And remember, try and look at other databases beside PubMed if you're doing something formal, like a practice guideline for the AANS. Okay, so how do we actually do these? Okay, this is like the instructions for the small group session. So we'll get started the way we get started with any of these projects by asking a really focused question using the PICO format. And remember, we're thinking about meta-analyses as clinical trials with our patients as the articles that we're putting in. Okay, so you have to have pre-established inclusion and exclusion criteria, the same way you would say patients greater than 18 years of age with a, you know, with, I don't know, with a performance status of whatever. You've got to do that with the articles. Why? Because you see what you get and then pick the articles. Why do you insist on having established selection criteria? I mean, it's the same reason you do that in a clinical trial. Yeah, exactly. Bias is a risk at every step, and this is a huge potential for bias. Studies when you're trying to combine them are necessarily going to be different. This is the second big risk in these meta-analyses, right? So the patient groups are always a little bit different. The age may be different. The criteria for performance, the type of disorder, the interventions may vary a little bit. The outcome measures frequently are not exactly the same. And so the studies are always heterogeneous. And so you also have to anticipate that they're not going to be the same and explain how you're going to deal with that. How much heterogeneity are you going to permit before you say, I can't combine these studies? Okay, and you need to know all of that stuff ahead of time. And then you do your exhaustive literature search, and these are the components people advocate. So not just Medline, but other databases, talk to experts, look at a list of randomized control trials to see whether, you know, clinicaltrials.gov to see whether maybe you're missing some randomized trials that have been done. Many people will accept published abstracts. That's a little controversial, but you've got to look through the list of references in your selected articles. You have to be really exhaustive about this. And then apply those inclusion and exclusion criteria that you've pre-established to select these final eligible articles, and then critically appraise them, like we just finished doing. Why is it important to make the selection and the critical appraisal, by the way, in duplicate? Male Speaker Because it's just like a diagnostic test. You need to make sure that everybody's measuring the same thing, but it's the same plot. Exactly. And how are you, if you and I both appraise the articles, how are we going to decide how frequently we agree? What's the statistical way to do that? And you can, by the way, even though our tool looks at kappa with two evaluators, you can apply kappa to three or four. So it's doable. So it'll reassure us that Nick and I are seeing the same thing. And you should have a strategy when we disagree. How are you going to resolve the disagreement beforehand? We're going to discuss it and resolve our differences. We're going to get a third person to break the tie. You should have that all established ahead of time. And you can do these for therapeutic questions, like we've been discussing, or for diagnosis or prognosis. Meta-analyses apply to any type of question. So it's just reviewing what I just said about two people doing all of these steps. Because like any clinical trial, meta-analyses are subject to random error and bias, and we want to minimize those at every step. And then if you can, we want to quantitatively evaluate the evidence, and we're going to now talk about what quantitation means. And we're going to talk about the math. So here's what a meta-analysis looks like. This is data from the early AIDS trials, the antibiotics. And these are the two ways that you see meta-analysis displayed. This is all of the individual studies and the meta-analysis. And this is a technique called a cumulative meta-analysis, where there's the first study and you're adding the second one. Here's the total number, and you're adding another one. The number's gone up. What observation do you see about the — maybe it's the easiest to see here, but also there — about — well, tell me what this is. What is this picture describing? What's the diamond and what's the arms on either side? Yeah, exactly. This is your point estimate and your confidence interval. Do you see a consistent association between the size of the confidence interval and the size of the study? Yeah. Are you surprised about that? No. And that's why the smallest is the summary statistic at the bottom. And that's why the cumulative meta-analysis looks like it does. Bigger the size of the study, the more precise the estimate. What did I say wrong? So I know from sitting over there that you can't always hear what Dr. Barker has said. So in this particular example, a big reason that the summary statistic is not as small as the CONCORD study is the heterogeneity of effect estimates and precision of the trials that go into the meta-analysis, and it also has to do with the technique that people use to do the meta-analysis and a few other things. But by and large, big studies are the way that you get confidence intervals smaller, increase precision. Another example that really shows the same thing. This is streptokinase for acute MIs. So we perform the meta-analysis, as we've discussed, to try and get a better assessment of the true effect size. And the studies that we put in vary in precision, and we sort of have to weight them so that we... So why, for example, wouldn't you just... I mean, we have all these numbers. If you bothered to look up the articles, you could see how many patients met and didn't meet an endpoint, and you could just stick them all, add them up, stick them into your two-by-two table. Why wouldn't you do that? Much simpler. The math is certainly much simpler. Male Speaker 3 Yeah, so a patient from a tiny study shouldn't probably be weighted the same as a patient from a monster study, okay? You need some way to credit studies for their size, because bigger studies, generally have a better estimate of effect size. And so that's what meta-analyses do, and there are a bunch of strategies for doing those. The two common ones are the top two, and those are the only two I think we'll talk about today. If you're interested, we can discuss some of the other ones, but these are the ones that you'll see when you read papers and that you'll use probably this afternoon when you're doing your own. So here, we're going to come back to this, but just for a second, look at the — this is what's called a fixed effects model of doing meta-analyses. It's the one that we'll provide you in the tools. This is called a random effects model. When you get down to the bottom, you get pretty much the same result. Look for a minute, and you're going to see the same slide again. Look for a minute and compare side to side, and get in your mind what — do you see some differences there between the two strategies? The data is the same. Any differences that stand out to you? Yeah, yeah, that's really subtle and good observation. So this is a more conservative estimate of the effect size. There's something that goes along with that too, another subtle observation. The confidence intervals of the random effects cross the midline more often. And what does that — well, you know what? Think about that one. I'm going to give you three or four slides to distill that. That's good. And while we're doing that, I'll talk about the fixed and random effects models. So there are two sort of ways to think about how you would put studies together. A fixed effects model makes the assumption that there's one true effect size for the specific question in the universe, and each of the studies that you're putting together in your meta-analysis kind of estimate that one true effect size. And if you could do an infinitely large study that included all of the patients who have data relevant to your question, you would get precisely that effect size estimate. But because you can't, because we have to sample, the studies are all different. They don't give you the same precision and effect size. And so a fixed effects model will weight the different estimates based in some way on sample size, inverse variance, some way to weight credit sample size in the different studies. And so what a fixed effects model is recognizing is this within-study random error. Is that fair? We're sampling from a pool of possible groups of patients to answer a question. Each of the studies is a different sample. They're going to have some random variation of the effect size. And so we've got to take that into account in our meta-analysis. And it doesn't really address this issue of heterogeneity, of between-study variation. That's the fixed effects model. The random effects model is a little bit different. The assumption here is that each study in your random effects model is estimating a different but true effect size, because each of the studies has slightly different populations and slightly different interventions and slightly different outcome measures. And so each study is estimating something a little bit different, but you still want to put them together. We still need direction in taking care of our patients. And so when you combine those, you now have the source of error that's related to randomness, because you're doing a selection of all the possible patients, and this between-study variability, because each study is asking a slightly different question, estimating a slightly different or maybe a very different effect size. And so the strategy that the random effects model uses accounts for both of those. And what it really does is it does the fixed effects model, this sort of weighting based on the larger or smaller study, and then it goes back and looks at how much of this difference between studies is there. And it undoes, to some extent, the weighting that you've just done based on size, so that if there's a lot of variability between studies, you undo it completely, and it's basically like just sticking numbers into the two-by-two table. And if there's really almost no variability, then you don't undo it at all, and those are the two ranges. It's a fixed effects model, random error, and as a result, as you observed, the random effects model kind of weights smaller studies more highly, and it's more susceptible to publication bias, which we'll take a look at, but less susceptible to heterogeneity, and as a result, produces these wider or more conservative confidence intervals. So that's exactly what you observed. Here's some pictures just to see. Same question, okay, by phosphonates for hip fractures to prevent hip fractures. So here's the fixed effects calculation, the random effects, okay, really gets sort of the same answer in this particular case, but with a more conservative estimate of effect size, but they can, the models can sometimes actually differ in terms of their instruction to you. So random effects, fixed effects, very different answers. This is back to the one that we showed you originally, sort of following from the more conservative and less conservative estimates. If you were using this model, you would have stopped your, and you were following this along cumulatively, you could have stopped your investigation a couple of years earlier than you would have if you were using the random effects model, because it's a more conservative estimate. Anyway, that's just a kind of a summary of the two side by side. As I say, we use for our own tool, fixed effects model. People will sometimes use the random effects model. Usually there's not a big difference, but you should be aware of the circumstances where it is important. Oh, you know, the other cool thing about that study is, this is actually a way of, we can't usually, we said we can't measure bias precisely. This is at least giving you the maximum quantity of heterogeneity. You can actually measure in that sense, how big could it possibly be, the difference between those two. I think that's kind of cool. And then there's some unique biases or threats to reliability with meta-analytic techniques. One of them is publication bias. And you probably know this really familiar to you. What is publication bias in general? So positive outcome studies, more likely to be published. What other studies either are more likely or less likely? English language studies, more likely to get published. So small, negative, non-English language studies tend either not to get published, or if they do get published, there's a big delay, a bigger delay in their publication. And there are a bunch of strategies to assess that in meta-analyses. The one that's kind of the coolest in a way, they all have cool names though, but the one that's the coolest, and the one that you see a lot, is this thing called a funnel plot. And it's probably worth spending a minute deciding what this is all about. So what you do basically, this is the summary estimate of the effect size, that down-pointing arrow. And then we've just plotted the different studies, the risk ratio on the x-axis, and some measure, in this case, the actual sample size. And then you try and fit a triangle, a symmetric triangle, that includes most of the studies and is centered at this estimate. Why is it a triangle? Why isn't it some other shape? What properties of what you're doing there make this a triangle? Why isn't it a funnel? Why do they call it a funnel plot? But why is it that shape? Yeah, yeah, exactly. The larger the sample size, the less exaggerated the effect size is, and the smaller the study, further out. So this is a neat way of identifying outliers, for one thing. And it also is a really good visual way of identifying the potential for missing studies. So remind me again what type of studies you think are potentially not going to be included in your search for articles, or haven't been published? And where would that be on the picture here? Yeah, where that hole is? That's what a funnel plot does for you. This funnel plot would make you concerned that there may be some publication bias, that you're missing some small negative studies. Just other examples. We actually have one of these that we'll look at in Journal Club. So anyway, there are other strategies. And they have such cool names that I figured it was worth just doing two slides. This fail-safe strategy asks the question, how many undetected negative studies would you have to add into your meta-analysis, make up and add in, to change the result? Okay, and we would ideally like that to be not a small number. If a small number of those trials change your result, then you're concerned about the possibility of publication bias. And similarly, this file-drawer test, you sequentially remove studies from your meta-analysis, and you'd like to find that no one or two studies have a huge impact on the outcome of your meta-analysis. If they do, again, you're worried that your estimate is tenuous. And then the other one, heterogeneity. We've talked enough about this so that you remember that studies differ in all sorts of features. And if they're different enough, you worry about whether it's honest, fair, and permissible to combine them. So let's think about some combination rules. Because the studies are, you know, they're inevitably going to be different, okay? But we still want the best estimate of an effect size. So what do you think about this rule? Comfortable pooling if the studies are on the same side of... Do you like that rule? So would you pool these studies? Yeah? Comfortable with these? How about these? Same distance apart? Or why so? So pooling these or no? I'd probably be worried about this, right? They look like they're estimating... They're grouped, and they look like the two studies and the three studies are estimating some different effect, perhaps asking some different question. Different patient population, different outcome measure, different intervention. Different enough that they're giving you a very different estimate of effect size. How about these? Pretty comfortable with these, right? Although one might say that a meta-analysis was superfluous in this really tightly grouped set of studies. How about these? Same distance, right? Same thing as that. Probably this is where a meta-analysis would be most useful. It looks like they're all estimating the same effect size, visually anyway, but it's so imprecise that you don't know. So this is a good perhaps. But I think you can't focus exclusively on where in the graph of benefit or harm. How about taking into consideration the confidence intervals? So we were a little bit divided about this one. How about now? So now, geez, these confidence intervals completely overlap. So apart from what you might say about the studies themselves... Yeah. But this is something that a meta-analysis would reduce those confidence intervals, give you more precision on your estimate. And you can't really make a good case that they're estimating something different or estimating something at all. So anyway, there are other tests for heterogeneity. This one, the Cochrane collaboration has now adopted this, and so it's become a little popular. But there's this idea of a test for heterogeneity that uses a p-value. So the null hypothesis here is that the underlying treatment effect is the same in all of your studies, and you do this statistical test. And so a low p-value means what? Against your null hypothesis, you get a low p-value. Is that what you want to see or not? I guess it depends what your interest is. But a low p-value suggests the chance is not the best explanation for the variability that you're seeing. And so you may not be able to or want to combine those. Almost all of the math behind meta-analyses is not as secure as the math behind two-by-two tables and analyzing individual studies. There's a lot of uncertainty. I don't want to say fudging. And so, for example, it's reflected here by calling an important p-value something that we would never consider important in a single study. And in general, the math is less well-defined, especially as you get out into some of these other things about heterogeneity. So it's a problem. You should recognize it and you should have a strategy for dealing with it, but it's not always avoidable. And here are the options. You can ignore it if it's not substantial and just go ahead and analyze. You can account for it with a different model of meta-analysis. You can say, gee, it's not possible to do a meta-analysis yet with the data that's available. And then there are some more sophisticated kind of analogous to multivariate analysis strategies that you can use. And these are the ones that are still not well-established, okay? So on the one hand, heterogeneity is always there. It's a threat to the reliability of your meta-analysis. You need to recognize and account for it, but we still need to have information that makes use of all the data that's available. There are lots of ways to address that. So pretty much finished. So a good meta-analysis does a lot of good things for you. Most importantly, it gives you a more precise estimate of the effect size, and because the numbers are big, it allows this sort of hypothesis-generating approach to subgroup analysis that you can't frequently do with single studies. But it doesn't do everything. It doesn't supersede good clinical judgment, and it doesn't make studies better if they're not good, okay? So bad studies don't get improved by doing a meta-analysis. Bad studies are still bad studies.

meta-analyses

evaluation

rigorous method

evidence-based medicine

limitations

publication bias

effect size estimation