I'm going to start off with discussing observational studies. A lot of people see this and they think that this is really simple, not worth their time, very basic, but in fact it's the majority of what you see in the published literature. Prize-controlled trials are difficult to do and time-consuming and expensive, and well-done observational studies provide a lot of what we read month-to-month in the journals. So it's worth looking at them. So this is what we're going to talk about. We're going to talk about different types of interventional studies and where they fit in that evidence hierarchy and how we can use them, how we abuse them, and then at the end where it leaves us. So you might not be surprised to know that we can divide observational studies into a couple of different categories, and they differ from interventional trials in this real important way that an observational trial, just like it sounds, is when we look at the world as it is and see what's going on. An interventional study requires you to perturb the system and see what the result of the perturbation is. So that's the basic difference between these studies. And then here's the whole list. So just as we said that there's a hierarchy of clinical trials types, so interventional trials more reliable than observational trials, within each of those categories there's a hierarchy. So I guess this probably should get reversed, but way down at the bottom are single case reports and case series and kind of moving up through the ranks to cohort studies and ultimately to randomized control trials. And we'll talk briefly about all of them. I just wanted to mention, and this is not just relegated to historical importance, sort of at the end of the course we'll talk about this, a guy 50 years ago, more, called Bradford Hill, was thinking about how do I know what evidence is strong and what evidence is weak? And he did this in the setting of some famous research that you might be familiar with. He and another guy, last name Dahl, were the people who told us all sorts of things we take for granted now, that smoking is associated with lung cancer, that alcohol is associated with breast cancer, that asbestos is associated with mesothelioma, just all of these sort of very basic epidemiologic tenets that smoking is associated with heart disease. That was all them. And they came up with this list of criteria for how can I believe in association. And the four on the top are the ones that I would recommend to you to just keep in mind the strength of the association. We called it magnitude of effect a few minutes ago. Consistency, do studies over and over again tell you the same thing? Biological plausibility, okay, and we're going to circle all the way around at the very end of the course to that because Bayesian advocates will call that, call that what? Biological plausibility. That's pretty much like your pretest probability or your prior. We'll get back to that. And to some extent, this temporality, does the proposed cause precede the proposed outcome? So that's important stuff. And you may ask if randomized controlled trials are so great and observational trials have so much problems associated with them, why do them at all? And the reason, of course, is that randomized controlled trials may be unethical. You can't, for example, randomize people to smoking to find out what harmful effects it produces. And in a lot of cases, they're impractical. They take a long time. They cost many millions of dollars to do. And we just can't always get randomized controlled evidence. So we need alternatives like this. And they're actually often pretty good. This is just from a few years ago, a compilation of observational trials, cohort and case control studies. And the identical randomized controlled trials, the observational ones are the black dots and the randomized controlled trials are the open dots. And in fact, at least for these particular questions, not only do the observational studies pretty well estimate the randomized controlled trial effect, but there's less scatter, right? They're all pretty much on the almost consistently on the same side of the unity of the no difference. You can see that maybe better from the actual numbers if you calculate risk ratios and odds ratios. But observational studies will often give us a result that later randomized controlled trials bear out. But not always. There's some really outstanding misses. You think of some outstanding misses? I mean, the obvious, the one that people in public health talk about all the time is hormone replacement, right? Thought from some huge 30,000, 40,000 patient cohort studies that that was a good thing to do. And it turns out that the randomized controlled trials showed us that it was a terrible thing to do. In the neurology world, a couple of recent trials in ALS, minocycline, patients were asking for minocycline. It's an antibiotic. You can get it off the shelf. It turns out that it's harmful in ALS despite the cohort studies that suggest it. So there are lots of examples. Within the observational studies then, and that's what we're going to focus on for the next 10 minutes, there are two types. There's descriptive observational studies. You look at things that are already there. And then there are these analytical observational studies. And this difference is not only important because of the type of study, but this is the question that IRBs ask. If you've got a study plan, if you have a question in mind, right, then you need the – I mean, you always need the IRB, but then they need to intervene. The IRB is involved in approving research where you have a question in mind, right? If you have an idea and say, I'm going to go back home and look up all the cases of this or that, the IRB will usually give you an expedited review, right? But if you have a question in mind before you start looking for the data, then that becomes research and requires a lot more supervision. So this is a relevant distinction for that reason as well. We're going to go through all of these studies real quickly, and you're going to devise the study. And I'm going to – this may be the last time that we choose a non-neurosurgery example, this is a cardiology example or a neurology example if you prefer. So this is just to remind me, it's an echocardiogram, and the little outline there is the mitral valve. It's showing mitral valve tenting, and it's to remind me to tell you that patients with Parkinson's disease are more likely to get cardiac valvular disease. And so I'm going to start you off – that's an observation. So treating a patient with Parkinson's disease with a dopamine agonist, so something like bromocriptine, one of those drugs, and the patient develops valvular heart disease. I've never seen that before, I've never seen that association in the literature, so I think it might be important to tell people about. And so I publish it as a what type of observational study, a case report. How would you make that more impressive? So instead of a single anecdote, you have a whole sack of – yeah, so how would that sound? What would that sound like? Stick to this question. What would that sound like? Do a case series that sounds like what I just described for the case report. Great. Great. So I look back in my practice, I have a big Parkinson's disease practice, and I've found 100 patients with valvular heart disease, and 60 of them are getting dopamine agonists. Is that more impressive data? More reliable, stronger evidence? Why is that more reliable? Bigger number, yeah. The chance of random error is reduced. What's the problem with this data? Why is it inadequate to make a decision? Yeah, yeah, exactly. So you don't know, for example, how many patients developed valvular heart disease who didn't get a dopamine agonist, or how many people who are getting a dopamine agonist did not develop valvular. So you have no comparison. So it's stronger evidence, but it doesn't take you very far. You don't see this type of study quite as much in medicine, but what are we talking about when we say an ecological study? We see this a lot in international medicine. There's a person in our department who does these kind of studies with hydrocephalus. Yeah, yeah, exactly. So in my example that I've chosen, let's say, for example, that you look at dopamine agonist use in Senegal, where there's low use, and in the United States, where there's high use, and then look at the outcome that you're interested in, valvular heart disease, and you wouldn't be surprised to see that there's less valvular heart disease in Senegal and more in the United States. So that's a big number of patients, to suggest the association. What's the problem with that type of study? Why is that not providing you strong evidence? It's big numbers. It's like countries worth of numbers. Yeah, good, and more specifically, you don't know that the people who are having valvular heart disease are the same people who are using dopamine agonist. It's a population-level study, right? So good. How about this final—oh, that's to remind me that there are some ecological studies that I believe. It accounts for my weight. I've been trying to prove that this is true by following that advice. It's kind of cool, though, isn't it? We'll talk more about these kind of studies. What's a cross-sectional study? This is the final one of these observational studies that we don't see as much of, but you do. A lot of public health people looking at gun control now, for example, look at these cross-sectional studies. It's a really good public health tool. Here's an example. I live in Dauphin County in Pennsylvania, so you could, if you're interested in this valvular heart disease problem, go around and ask about the prevalence, the existence of valvular heart disease—that's the illness we're interested in—and then look at all sorts of exposures. You can look at dopamine agonist use, but also age and race and socioeconomic status and gender, and then look at those two numbers together. The problem, of course, with that is that you're looking at prevalent disease, not new cases, so there's no real temporal association between the exposure and the outcome. But it's still a really good way of looking at disease and population, so public health people do that all the time. But this is what we do most of the time. If you look in—I did this a couple of months ago for the AANS meeting—if you look at the journals, you'll see that the majority of studies consist of case control and cohort studies. So what are these? What's the difference? How would you set these up? So we're talking back into the world of evidence-based medicine. Set up this question for me using a PICO format. Inpatients. Inpatients with Parkinson's disease. Now think intervention or, more broadly, exposure, because that way it works for risk factors and diagnostic tests. So inpatients with Parkinson's disease who receive a dopamine agonist—that's the exposure—compared to those who don't. And the outcome? Yeah, so something like that, if we condensed it down to a single sentence. And the point of this, of course, is to estimate the association. Is there an association and how strong is it? That's what the point of doing this exercise is. So if we were going to do a case control, what distinguishes a case control study? What's the feature that just, no matter what else is going on, tells you that you've got a case control study? What are those? What happens? In what way? Because that's the right answer, but from whose perspective? The answer was it's retrospective. I would say it's retrospective from the view of the patient, but we're probably saying the same thing. So case control studies start with the disease of interest or the outcome of interest more broadly, right? So in this case, a case control study, you would look at patients with Parkinson's disease who have valvular heart disease. And then you'll go backward in time, that's why I say it, so backward in time to look at exposure. Were they or were they not exposed to dopamine agonists? And then you'll look at a separate group of Parkinson's disease patients without valvular heart disease. And again, go backward in time to look at their exposure. And then you'll compare the exposures between the two groups. So from the perspective of real time, you're going backwards, right? You start with the disease and you go backward in time, in real time, in patient time, okay, to see whether they were exposed or not exposed. Is that fair? And how now, once we get that data, what math will we use to look at the association? Yeah, yeah, exactly. So we go back to our two by two table and put in the letters and we'll look for the odds of exposure and do the ratio. This is an odds ratio, that's what we do with case control studies. So anyway, you identify patients based on the outcome of interest and you look backward in time to look at the exposures and then you compare the exposures in the cases, diseased and in the non-diseased. So do this for, set this up for our question, how would you do that? This is straightforward, no trick here, right? You'd look for a group of Parkinson's disease patients who have and a group that don't have alveolar heart disease and go back in time to look at their exposure, right? So you'd sort of ask a question like that, is that fair? It turns out that fortunately someone else has done this for us and if you go to their table too and you look at the actual data, we can just do this and short circuit all the mess of dealing with patients. And if you do that, I'll do it for you. But this is what we'll spend most of the time doing in the small group sessions. If you do that, if you take that data and you put it into the two-by-two table and you get that, it looks just like what we did with Bell's palsy and steroids. And if you press a few buttons on the statistical tools that you have, you get these results. Tell me what that means, interpret that for me. If they're exposed, they're almost nine times more exposed to a dopamine agonist. Good. So that's an important distinction. So remember with case control studies, you start with the disease. And so when you're interpreting the results, you start with the disease. So in patients with valvular heart disease, Parkinson's disease patients with valvular heart disease, they're almost nine times as likely to have received a dopamine agonist. That's not quite what we want to know as clinicians. What we want to know as clinicians is, if you receive a dopamine agonist, how likely are you to develop valvular heart disease? Case control studies, you have to put it the other way. Because you start with the disease, you look at prevalent disease, not incidence. How do you add in the p-value? How does that additional information provide to you? So nine times more likely, I'm sorry? Yeah. So this is very unlikely, this result, to be due to chance. And the confidence interval, how would that enter into your interpretation? So we know both because it doesn't cross one and because we see the p-value that it's a significant result. So that information is contained in the confidence interval. We could dispense with the p-value. But it does give us some more information. You would say with 95% certainty that the value would be non-significant or non-significant? Personally, I would accept that. A lot of statisticians would say, maybe don't put it that way. They would say, if you want to say that, then say, if we did this test 1,000 times, 950 of those times, our result would lie within that interval. Or as clinicians, we might say, our best guess is that the point estimate, our best guess is 8.8. But the study is equally consistent with a two times greater risk or a 40 times greater risk. So good. And that's basically what they got. But they didn't do as good a job as you did, right? This is all the data that you see in the New England Journal of Medicine article, which is a p-value. We know a lot more than the readers of that article. All they know is that some result didn't happen by chance. So p-values are sort of overvalued, right? We know how much, what's the size of the association. And it didn't take very much work to get that. It's a real failure of the journal, and a well-known journal, to not serve its readers by telling them what they need to know. So these studies, please. I have a question I wanted to ask. The value is too wide of a confidence interval. It doesn't sort of like p-value less than 0.25, right? Can't we do like all over the place, like 1.6 to like 0.69, where it's wide? Yeah, so what gives you a wide confidence interval? Or asked another way, how do you narrow the confidence interval? What do you do? Yeah, numbers, right? It's an issue of random error or precision, and we will talk more about that. So case control studies, because you start with the disease, these are good studies for rare diseases, right? Think how hard it would be if you were doing a study in, I don't know what, some pediatric crazy base of skull disease or something, where you had to wait for the disease to come up in the population. But if you start with the disease, it's a lot more efficient. And the studies, therefore, are quicker often and less time-consuming, less expensive. So they're good. And if you want to look at more than one exposure, you can do that and look at multiple ones at the same time. The problem is that they're very at risk for bias, okay? Because you're looking back in time to events, exposures that have already occurred. And they're also not good for rare exposures, because you're starting with the disease. Let's talk about the cohort study. How does that differ? We've talked about what defines a case control study. How does the temporal approach in a cohort study differ? Yeah, yeah, you start with the exposure, and then you go forward in time, forward in real time. Now, if the events you're interested in have already happened, which is a frequent type of study. So we go back, we look at all the cases of valvular heart disease in Parkinson's patients in the last 10 years. We look at all the Bell's palsy patients in the last 10 years. All of that stuff has already happened, right, that we've got 10 years' worth of data. But we're still, from the patient's perspective, going forward in time. We're still starting with the exposure, bromocriptine, dexamethasone, you know, the exposure, and going forward in time, even though all the data's there. And so we'd call that a retrospective cohort study. It sounds like an oxymoron, but that's what it means. Or you could say, for the next 10 years, I'm going to look at the exposures in these Parkinson's patients. And that's a prospective cohort study. But you're always going forward in time from the patient's perspective, okay? So that's what a cohort study is. So whatever the other bells and whistles that people dress them up as, if you start with the exposure and go forward in patient time to the outcome, then it's a cohort study. Okay? And it looks exactly the opposite if you do that picture with the red and blue balls. Start with the exposure and go forward in time to the outcome, whether or not the data is already a complete set or you're waiting to collect it. And then the measure of association for a cohort study. Now we're doing, right, incident disease. The disease has to come up as you're watching following the exposure. Now you can do a risk ratio or a relative risk. And if you put the letters in and you calculate the risk of disease in those two exposure groups, you get a risk ratio or a relative risk, okay? So that's the defining features of a cohort study. You use risk ratio and you go forward from the exposure to the outcome. And if you were going to set this up for our Parkinson's disease group, you would ask the question in exactly the same way. And as it turns out, in the very same issue of the New England Journal, someone else did that. So you do the calculations. They did a little unfortunate thing. They called their exposed and unexposed cases and controls. I don't know why they did that. This is a cohort study, but they've like mixed their metaphors severely. So here's the data. Okay. So you fill it in for me. Here's no use of dopamine agonist, this very top line. And cases are valvular heart disease and controls are no disease. The case and control, that's just a completely inappropriate use of the terms, I'm sorry. So what do you put in the different? Where does, what goes in these different boxes? Put some numbers in for me. Is that too small to see? I can't tell. 19 goes where? So 19 is no use, but yes, heart disease. Where does that go? Here? Yeah. So 19, where does that number go? Yeah. And how about, these are all dopamine agonists and these are all cases, what number goes here? Yeah. And I won't ask you to count up the other number. That's the number. And that's what you get. So interpret that for me real quick, because we've got four minutes to stay on time. Yeah. So now though, you're okay in saying it the way we want to know it, that like you said, if you're exposed to a dopamine agonist and you have Parkinson's disease, you're two and a half times more likely to develop valvular heart disease, with the same stipulations about the confidence interval. Okay. Does that seem reasonable? So these studies, because these cohort studies, because you're starting with the exposure, they're very efficient for rare exposures, things that don't happen very often. And they're much easier to control the risks of bias because you don't get to see the answer when you start. You start at the beginning and then look forward to the outcome. So the risk of bias reduced. In fact, a prospective cohort study is a really good study. It's almost, not quite, but almost as good as an intervention trial. But they have drawbacks as well. They're time consuming and over the time it takes to do it, your interests may change. The exposure may not be relevant anymore, for example. And so they're, in the end, don't provide as strong an amount of evidence as we'd like. So just to sum up, case control studies, you start with the disease, look backward in patient time to the exposure, and you compare diseased and non-diseased groups in terms of their exposure. Cohort studies, you start with the exposure, you look at exposed and non-exposed groups and go forward in time to look at whether disease develops or doesn't develop. And they have their places. If you're starting with, if you're a case control and you're starting with the disease, it's efficient to look at rare diseases. It's efficient to look at rare exposures with a cohort study. One uses the odds ratio, the other the relative risk. Case controls are usually smaller studies and quicker and easier to do. But they have a higher risk of bias that makes them more reliable. And that's what underlies this sort of hierarchy of evidence among the observational studies. So just as a final word, observational studies are often the best way in terms of efficiency to answer a question. Bias is always a big challenge. And we need other tools. They're not good enough for clinical decision making. And that's what Dr. Hain is going to tell us about.

observational studies

hierarchy

Bradford Hill criteria

randomized controlled trials

limitations

bias