I'm Tony Asher. I'm from Charlotte, North Carolina. And I've been asked to address designing clinical registries. And if I were to just restrict this to a purely operational methodological analysis, I'm sure I'd put everybody to sleep immediately after lunch in particular. So I'm going to mention a couple things at the very beginning. First of all, here are my disclosures. Really nothing that should affect this content in any significant way, aside from the fact that I am the director of our national registry, and so I'm going to have some biases with respect to the value of that particular effort. I do want to mention this particular reference. For any of you who are really interested in registry design, I'm talking about the mechanics of putting together a registry. And in particular, registries are used for a variety of different purposes. If you are particularly interested in the scientific uses of registries and the use of registries to do more innovative things, you're probably aware that the FDA is now using these for Phase IV analyses, in particular, of new devices. You should look at this. This is a major work. This is the third version of this. Organized neurosurgery got to contribute to this, and I was one of the contributors. But this is a two-volume, roughly 680-page comprehensive guide to putting together registries. It is extremely readable, actually. And so I would encourage anybody who is interested in the use of registries to look at this. Furthermore, if anybody is interested in looking again at the mechanics of how we put together our national registry, I encourage you to look at one of these references. If you don't get this down and you are interested in it, just e-mail me. My e-mail is simple. It's asher, A-S-H-E-R, at C-N-S-A dot com. And this information came out as part of a comprehensive series of articles on registries that we published in the Journal of Neurosurgery in January of last year. In any event, what I would like to do is first start by talking about my relative role in all of these registry-related efforts. I was a basic scientist many years ago. I got involved in clinical science, but then over the last six or seven years I've been involved in some national leadership issues. My particular interest, and now an academic interest, has been in policy strategy, collective approaches to the creation of a sustainable health care system. I've had some really interesting interactions with stakeholders on a variety of different levels and gotten their perspective on where things are going. And so what I would like to do, we've been focusing on the science here, I would like to talk a little bit about the broader societal issues, in particular the broader environment in which we exist, and I think should put our practice, science, and opportunities in a broader context, particularly a societal context. And in that regard, most of you are familiar with a graph like this. So with respect to our health care expenditures as a share of GDP, with respect to our expenditures in a per capita context, health care expenditures are going through the roof. The fact is that we're now up to almost $3 trillion with respect to what we pay in America for health care and the value of this care is being seriously questioned by virtually all stakeholders. As a result of that, this value-based purchasing concept is being adopted by most stakeholders to help achieve some level of sustainability in our current health care system, and the equation here is fairly simple. My perspective, and now this is just a bit of an ad, is that the present value metrics and the present value systems are basically based on performance and largely irrelevant health care processes. Now we're working with the National Quality Forum. We're starting to evolve away from process measures, but the fact is that most of what we measure right now does relate to health care processes that may or may not have anything to do with outcomes at the end of the day. The administrative data that is typically used to document performance is often incomplete or inaccurate, and this is a completely separate conversation, but I think most of us have had experience with looking at analyses of administrative databases and all the limitations that can be based in using billing data for trying to make quality determinations. And the emphasis is often on the cost side of the equation, which is there's nothing wrong with being sensitive to cost, but a number of the groups that are driving these things right now seem to be almost exclusively interested in resource utilization as opposed to the outcomes. And so a number of groups have concluded the present metrics do not reflect aspects of care that matter most to our patients, and more importantly, they cannot take into account the real-world variability and dynamism of medical practice. And it is a fact that most of the major stakeholders recognize these limitations, and so a lot of people just say, look, we can't influence this. The factors are too big. The forces are too powerful. Physicians have no chance to really make this meaningful, but that's not true. When you sit down with CMS, when you sit down with most of the major payers, when you sit down with advocacy groups, you find that they do want our input, and as the months go on and I'm involved in these conversations, I'm seeing more and more that they really want to know, what do we want to do? What do we suggest can be done to create a more sustainable health care system? And so we do have an opportunity. The problem is, as we've discussed here, we lack unambiguous definitions of both common clinical conditions. It's interesting. When one person says lumbar stenosis, they're thinking something different than somebody else. I can guarantee that, and I can't go through the information here, but based on some of the audits that we've done of some of our clinical centers, that is absolutely the case. And we lack unambiguous definitions of value related to those conditions. Value is defined in a lot of different contexts, not just physician's perspective, but also the perspectives of a variety of different stakeholders from purchasers and patients. You can go on and on. So value can be a variable thing. We lack, in general, robust evidence to support the value of services that we provide, and furthermore, as a group of physicians, we lack tools to develop and apply evidence to daily practice. It isn't just part of what most people do every day. This is a pretty selected crowd here that's very interested in it, but a lot of folks really don't even think about this to the level that we've been discussing today. So in any event, what I would like to suggest is that we need a specific path forward. I think it is time for big ideas in this area of practice, science specifically. I believe we need to abandon efforts focused on incremental change, all these little tiny things. I mean, the stern is up in the air, at least with respect to the financial aspects of the health care system. Picking at this in little pieces is really not going to solve things. And in particular, I think we need to focus on a deep understanding of priority health care conditions and real world practice, and by that I mean across all different types of practice groups, we need to identify conspicuous opportunities to promote value and high cost, high variability, high impact care, which is most of neurosurgery. So in any event, I think that what this is going to require is all of us, in all practice settings, ultimately becoming practice scientists. What we, in the collective sense, what we need to know and influence are the following. What are the expected and optimal outcomes of neurosurgical care? It seems like that should be obvious, right, but it is not obvious in all contexts. What are the outcomes of care with special consideration to all relevant stakeholders and specific comparable clinical disorders? When I say X, you understand what X means all the way down the line. That's not always the case. It's particularly not the case in spine care. How can we most effectively measure and analyze those outcomes as a group, a cooperative group of physicians, and how can we use that information to most efficiently deliver optimal health care outcomes? Bob mentioned some of this in a number of these slides. Based on conversations we've already had over the last day and a half are somewhat review, and so I'm going to go through them quickly. He mentioned this NeuroPoint Alliance. This has essentially been set up to facilitate collection analysis of clinical data for a variety of different purposes. The central theme of the NPA thus far has been these prospective patient registries. The N2QOD, the National Neurosurgery Quality Outcomes Database, is our large registry effort, which is continuing to expand over time, and most of the large organizations in neurosurgery are supporting this effort. This is a little disclaimer before I go on. I'm going to be talking about some of the advantages of registries, but we've already talked about the fact that we need to have a broad variety of tools in our toolbox to address our practice science needs, and so I don't want you to think that I'm talking about this exclusive of any of these other things that we've been talking about, particularly RCTs. In some instances, I think that we are going to have to incorporate, even within registries, multiple different mechanisms scientifically to achieve our ends. But in any event, I'm going to define a clinical registry the way the AHRQ does, as an organized system that uses observational study methods to collect uniform data, so we all know what we're talking about, to evaluate specified outcomes for a population defined by diseases, conditions, exposures, that serves one or more predetermined scientific, clinical, or policy purposes. I want to mention that this infrastructure can be adopted to a variety of other purposes beyond observational science, and we'll talk about that in a second. There's a wide variety of purposes that we can use registries for. They're often used to measure or monitor safety or harm, to measure or improve the quality of care, to assess natural history of disease, to determine clinical effectiveness or cost effectiveness of specific disorders, and perhaps in a certain context, conduct comparative evaluation of the safety and effectiveness of various treatments. But this is when this particular device becomes a little bit more problematic, and we would have to discuss when it's appropriate to look at using a registry vehicle, an observational vehicle, to look at things like CER, and we'll maybe have a chance to talk about that. I'm not going to dwell too much on this. Again, in review, a clinical trial is an interventional design. A registry is an observational design. I want to focus on the populations. I want to focus on heterogeneous populations because they have fairly broad inclusion-exclusion criteria. Research protocols specify those inclusion-exclusion criteria, and they're trying to create a homogenous population. That's what they do. So it is the case that if you look at the relative strength of this particular method of getting information, there is, when it's well conducted, fairly strong external validity, and that's mostly achieved by the fact that they include typical patients, the patients that we all take care of. There are broader inclusion criteria. There are fewer excluded patients compared to, for example, RCTs. And so, in theory, registries can provide good descriptions of disease course and impact of interventions, and they may, in some contexts, be more relevant in clinical trials for certain types of decision-making, particularly resource use. It is thought by some that well-designed observational analyses may approximate the results of RCTs on the same topic. I don't want to get into a long discussion of this. I think that there would have to be very, very specific design characteristics in order to achieve this, but there's no question that a variety of groups, FDA, CMS, that could go on and on, are becoming more interested in using registries to help power higher-level evidence opportunities. Other advantages of registry formats, outside of the generalizability, the observed outcomes may be more representative of what's achieved in real-world practice, and that's because we evaluate care as it actually is provided in clinical practice and not as it is assigned in research settings. These registries can offer the ability to evaluate patient outcomes when clinical trials are not practical or ethical. We talked about this. They tend to be relatively easily scaled and more cost-effective than RCTs. They're supported by multiple health-care stakeholders, and this ongoing format, Bob was talking about this yesterday, the idea that the registry isn't just starting today and ending in a year or ending in 18 months. These, for the most part, are meant to be ongoing efforts. They can serve a variety of different purposes, quality improvement, external reporting, and formal research. Here are some examples of ideal registry questions, natural history studies. I've touched on some of these things, and so I won't dwell on it. The disadvantages of the formats, the disadvantages that we talked about with respect to observational trials, and I'm not going to kill the particular point because we've already discussed this, but there are simply more opportunities to introduce bias in this format. There's a decreased confidence that the design, conduct, and analysis of registries can protect against bias, and, again, it's very similar to any observation analysis, although there are some methods that we can use to try and address that and make the information that we derive from these much more useful. And here's just a few different examples of how external validity, internal validity, analysis and reporting, how we can modify these things in the context of a registry to improve the relative value of the information. I'm not going to go through this again because we've already talked about this in the context of other studies. But we are, for example, looking at a couple of large centers to do some pilots with respect to some comparative effectiveness of patient groups, now that the registry has grown to a very significant size using propensity score matching. And this will be a preliminary experience, and we'll just have to see how it goes. But there is an idea that we can use some of these other statistical methods to do comparative effectiveness if we have the right data collection. So I'm not going to get into the methods very deeply. If you are really interested in knowing how you operationalize a registry, again, I would suggest that you go to this AHRQ handbook. But what I'm going to do is touch on these things very briefly, talk about how these relate to our national registry as an example, so hopefully there will be some context to this. So here are basically the major steps in planning a registry program, and it starts with articulating a purpose. And that can be a singular purpose, it can be multiple, and it just depends on what you're trying to achieve. If we look at our existing national registry and we were looking at how to define the purpose, we started with these various bits of information. Spinal disorders are the second most common reason for adult visits to medical providers, and low back pain, interestingly, is the most common cause of work-related disability in the United States. Spine surgery, if you go back to the AANS census, comprises 67% of everything that we do as neurosurgeons. This is remarkable. It's an incredibly common thing in neurosurgical practice. And the direct cost for spine care in the United States exceeds $90 billion annually. Actually, these are old numbers. These are seven years old. They may be twice this at the present time, but it's just an astounding amount of money that we're spending on this. In fact, there was an AHRQ study that suggested that spinal fusion alone presently may be costing us $180 billion, spine fusion. So, in any event, if you estimate, if you are willing to accept some estimates by the IOM and others that between 10% and 25% of spine care, and that's both diagnostic and therapeutic, are unnecessary and ineffective, this represents an astounding opportunity for us. If we can define the subpopulations of patients who are and who are not achieving benefit with our therapies. These conditions tend to be incredibly disabling compared to other things. We talked about utilization, just going through the roof, for a variety of different reasons. All you have to do is pick up the Wall Street Journal or New York newspapers or whatever it happens to be, and you'll see stuff like this. The public is beginning to question the value of what we're doing on a routine basis. And not just the public, various regulatory agencies, quasi-regulatory agencies, advisory groups, whatever, are beginning to ask some very, very hard questions about the value of spine surgery. So we think that all of these things really define a very significant opportunity. We have decided to, and we did, create a robust data collection system to observe outcomes in the real-world surgical spine practice related to five common lumbar spine disorders. And we want to use that information to develop risk-adjusted predictive models to facilitate targeted quality improvement, practice-based learning, shared decision-making, effective resource utilization, and reporting requirements. And hopefully I can show some examples of these going forward. But this was the information that was the rationale for going forward with this, and this is our defined purpose. Identification of key stakeholders is an incredibly important issue. I talked a little bit about value, but value is defined by the different stakeholders differently. And so we need to know what their perspective is. We cannot presume that we have a complete handle on value. The stakeholders have to be all the populations with an interest in our registry outputs or what we are doing every day. And this is really an opportunity to ask and answer the question, what outcomes really matter? At the end of the day, what matters? What are we trying to achieve when we're taking somebody in the operating room with a degenerative spine condition, for example? In some situations, it's pretty straightforward. Epidural hematoma, we don't have to ponder that too significantly. But in something like spine care, it can be variable. We can define these things and reach consensus through outreach with stakeholders, and we spent a lot of time with stakeholder engagement in the early phases of the registry development. This was an incredibly valuable process for us, and hopefully for the folks who will ultimately be consuming this information. Assessing the feasibility of these programs is incredibly important. These are very expensive endeavors. They require significant resource investment, both centrally and at the sites that are participating. I won't get into this in great detail. After we were looking at the scope of the project, we decided we had to have what we called a shared responsibility model. There are a number of groups that have developed registry platforms that are essentially industry-based. We felt that if we didn't have a self-sustaining program, there was no way we were going to survive past the first year or two of this, and we were determined to do something that did not rely on industry, or very happy to accept industry support for our efforts going forward, particularly in the context of data analysis. But this right now is a sustainable model, we believe, and the AANS has invested a tremendous amount of money into the initial development of this. Practice groups through a subscription also contribute to keeping the registry going. We've also received some nice research grants support through the NREF and other entities to start doing ad hoc analyses of the information. And so we believe this model is consistent with sustainability and planned growth going forward. And by the way, it is very similar to what other groups, such as STS and the cardiologists, have been doing for years. And so we did learn from their perspective. In planning a registry, you have to have a registry team and a governance structure. In particular, you need to have project management personnel, subject matter experts, registry scientists. You need to have health scientists, epidemiologists, statisticians, all these individuals have to weigh in on the construction of the registry program. We need to have data collection and management individuals. And finally, we need to have legal support. And I may not have time at the end, but the regulatory issues alone related to the registry took over eight months of intensive negotiations with OHRP and OCR of Health and Human Services just to get to the point where we can get an opinion on whether or not the common rule and the privacy rule applied to the registry. Astounding, it's eight months worth of work. And so our legal experts were very helpful in that context. But it's an astounding group, an astounding number of different disciplines and talents that you have to bring to bear on this. This is our general government structure. I won't get into this in great detail. The workhorse committees here are the operations committee, which is a multidisciplinary group that is involved in the day-to-day operations of the program, the scientific committee, which is involved in everything from designing the questions that we need to be looking at retrospectively with relationship to the database, funding opportunities, and how we need to design the data elements so that we can ask important questions going forward. This is our registry team. We work with Vanderbilt, the VIMPH, Vanderbilt Institute of Medicine and Public Health. It's been a tremendous partner. These folks have not only taught us a lot about registry science, they've also helped ensure that the product that we developed has been very, very robust and solid and will allow us to grow going forward. We have health information scientists who have supplied us with the REDCap database, which has been a huge boon to us because REDCap, if any of you use it, is an amazingly versatile database and we're essentially using it for free. It would have cost us a million and a half dollars to build this from code, and so this was a big deal for us. Project management that we do in conjunction with the healthcare scientists is shared with the NeuroPoint Alliance, but the healthcare scientists and epidemiologists at Vanderbilt do all the data collection, cleaning, storage, transfer and reporting, project management, the statisticians there help us with the data analysis, and again, NPA is very much involved in this whole process. And coordinating this, as Bob mentioned last night, really has become a full-time job for many individuals, not just me. We have to define the scope and the rigor of any project and we have to define the data set. So the scope and rigor, when you're looking at this, you have to start talking about a variety of things that influence literally the size of the project. How long is this going to occur? There's a quality improvement effort. We would like this to go on indefinitely. I mean, it is not the case that we're going to achieve a certain level of quality and it's going to stay there. Individuals will need to continuously show the quality and value of what they do and that's part of what this registry is supposed to accomplish. We wanted this available for everybody in the country who is interested in using it, orthopedic, nursery, neurosurgery. Ultimately, groups that are interested in non-surgical aspects of spine care where we can use this type of vehicle. And then we had to look at our resources along with the richness of the clinical data needed and we talked about this last night. The information that we need for basic quality improvement is probably not what we're going to need for comparative effectiveness research. And so, depending on what questions we're asking, we will define a certain subset of individual groups that might be involved in certain aspects of the project. Other aspects of the project, like quality improvement on the group level is probably something that every single site will be involved in. And we also wanted scalability. We wanted to make sure that we can grow this over time. So, very briefly, this is, we've very rapidly realized a national ambition. There's now 53 sites that are actively participating with 12 sites coming on board within the next couple of months. This has fairly rapidly become the largest North American cooperative spine registry. It's just the largest thing in existence. It's not the largest international registry. There's actually one in Sweden that has about 100,000 patients in it. But there are some limitations of that and that's been going on for about 10 years now. But that is the largest one in the world. This is, there's excellent representation across practice centers, academic and private practice, both participating in this. And most of the surgeon groups you can see are in this three to eight range. With respect to the data set, I won't get into this in too much detail because we had talked about different ways of putting data together. And I know that you folks are gonna be doing this in the context of looking at clinical trials in the second part of the session today. But with respect to registries, you start looking at the data set by defining the target population, the exposures of interest, and the outcomes of greatest importance. And in spine, we actually had some very interesting focus group data to suggest that the outcomes of interest to patients that were most significant were disability, getting back to what they would like to do, and also relief of pain. So it's fairly easy to look at some things from a patient's perspective based on those focus groups' analyses. You need to identify the data domains, which are the large, major categories of information that you want to be collecting going forward. That's what the domains are. And then, of course, we need to select the covariates relevant to the outcomes to be observed, the project scope, the rigor, and the resources that we have. And like all of these things, with any clinical research project, we have to balance the need to know with the like to know. There's only so much you can do. And in fact, collecting information that isn't relevant to the question can be harmful in a variety of contexts, and I'm sure Mike will be talking about later. And so, I touched on this a little while ago. In particular, we need to identify the possible confounding variables in a multidisciplinary environment. We, again, have learned a lot from Vanderbilt about this, and they were very helpful in defining for us some of the things that we should be thinking about that may not have been on our radar, but also disciplining the clinicians and getting us to really focus on those variables that are most likely going to be confounding in this context in lumbar spine surgery. And here's examples of these types of variables in various contexts, patients, providers, system. And then we also had to look at frequency and data, in duration of data collection. We determined, for the purposes of the initial project, that we would look at a 12-month horizon. We would look at very early morbidity, not much mortality going on in spine surgery, but morbidity data, and then we would be looking at three and 12-month outcomes. Some of the centers will probably be looking at 24-month outcomes for some of the lumbar effusions, but we also had to be very aware of the available resources. Some of the unique data aspects of this particular registry is that we are looking at patient-reported outcomes longitudinally. This, interestingly, is unique among surgical registries in the country. If you look at NSQIP, if you look at STS, if you look at a variety of different efforts that are ongoing, it's a very, very short time horizon, which is fine for procedures where there's a lot of upfront morbidity associated with it. So cardiac surgery probably makes sense to look at 30-day outcomes, and it's not that it's irrelevant for spine surgery, but when you're putting $20,000 of hardware in somebody's back and taking them out of work for a prolonged period of time, you really need to know what the outcomes are, at least at 12 months, and you need to have a patient perspective in that timeframe, and that's the reason why all the centers will be following patients for 12 months, and some of the centers out to 24. But it's very hard to do, and it has a lot of implications with respect to the difficulty in obtaining this information, particularly in a voluntary setting such as an observational registry. And I won't get into this too much. And then finally, with respect to data specifications, once we put all these basic things together, then we really have to sit down in teams and really, really talk about some specifics here. So with respect to the data elements, we have to get down to data definitions. What do we absolutely need? What are the explicit definitions for each variable? So the ranges, the acceptable values. This will help us ensure some internal validity. We need to look at the registry data map. Where are all, where's all this data coming from? Some is gonna be coming from the paper chart, the doctor, the patient, the EHR. We need to know where all that information is coming from, and we need to make sure there's uniform methods of collecting it at each and every site. And then finally, with respect to the database, this data element database, we need to define a data dictionary. We need to describe for the programmers the structure and attributes of the data to be used in the software application. And then we need to have validation rules. And these are basically just logical checks on entered data against predetermined rules. And so the information itself understands whether or not some of the data coming in is just bogus and needs to be flagged so that we can go back to the other centers. So this is an incredible amount of work that is a team effort between the clinicians and the quality scientists. So very briefly, when we looked at our data elements, here are the diagnoses that we included, fairly common things seen in the course of surgery. By the way, I mentioned one, two, three, four, five. I mentioned five. The adjacent segment disease was just added very recently. Here are variables. STS has like 300 variables that they collect. And we felt, we started with about 150. We basically weaned this down to 52. This is the first time that most centers in the country have been involved in this type of effort. We've always seen this as an evolutionary process. We wanted to make sure we were not overwhelming individuals and groups with the amount of data collection. But these are fairly typical fields. Patient variables, clinical variables, what are you actually doing? What are the patients like? Surgical variables, data surgery, what did you actually do in the operating room? What are the characteristics of the institution where this occurred? This is all very, very interesting and important information in adjusting the data down the road. And then with respect to longitudinal data, we're gonna look at 30-day quality, three-month quality, and 12-month quality. Again, some fairly standard things, fairly standard complications. We'll be looking at DVTP, UTI, all those things. But more importantly, we're gonna be looking at patient-reported outcomes and comparing those to baseline statistics going forward. And again, this is, I don't know that any large, specially-sponsored registry is doing this presently. Then finally, there's data management. And data management is simply, it looks at, it's an integrated system for collecting, cleaning, storing, monitoring, reviewing, and reporting registry data. And we're just gonna quickly go through this. So we have to have data management guidelines. We have to go out to the participating centers and the registry center itself needs to have some rules to make sure that we're doing this right. We need to tell participating centers what's acceptable in terms of collecting and abstracting the information, particularly from the EHR. We need to specify those data sources like we did back in the planning stages and tell them, look, this is how you need to do this. And we need to look at data entry systems and mechanisms, and this has become a huge issue right now. Is it gonna be manual? Is it gonna be automatic? We are transitioning to automatic, but we're doing this in a controlled way. We will not allow just any method of automatic data entry, and we are trialing different methods to make sure that they are safe and reliable, and safety with respect to patient confidentiality. And then the registry also has to have some rules for data cleaning, information when it's coming in, getting rid of bogus information, or looking at areas where we are missing information, and filling that in, or at least trying to get that from the centers. Sending out query reports. When we're looking at hundreds and thousands of these different data entries, we're gonna see patterns. And when we do that, particularly if one or two centers are doing things in a particular way that doesn't seem to be quite right, we send queries out to them, and we say, what's going on? We're observing this with respect to the data entry. Can you help us figure this out? And this is something that the registry center needs rules for. And then finally, the registry center needs to know how to store, secure, and give access rights to the information going forward. We developed what I think is a fairly unique method to allow the community that's collecting this information to interact, share information about challenges, and that's based in the practice-based learning network, which has an online presence, and also an interactive presence. Every week, the group gets together and discusses other ways to facilitate data collection. This has been extremely valuable for the effort so far. This is the basic flow of information. The data, obviously, is collected at the clinical sites. It's entered online into REDCap. It is securely transferred to Vanderbilt, and Vanderbilt does quality control, missing data checks, data validation, and auditing, and then ends up reporting back risk-adjusted information to the practice sites that can use that in quality improvement initiatives. The data coordinating center, specifically Vanderbilt, needs to look at data quality. The quality methods, as I mentioned, are going to vary with the intended purpose of the information, but they will be doing various things, and they will be using a risk-based approach to identify major areas of risk, threats to the quality of the data, balancing our quality needs with our available resources because we just don't have all the money in the world, but we're trying to intelligently apply the resources we do to maximize the data quality, and we've already gone through the types of barriers in great detail. So, I think what I'm going to do is not get into this in great detail, but I do want to mention that we've been focusing on this idea of audits. We talked last night about the evolution of these types of things. We are starting with what I would call a low-level audit. Our audits right now are 10% of the practice sites. That is on-site. We are looking at mostly inclusion, exclusion criteria and making sure that patients are appropriately enrolled in the registry. We are going to evolve to looking at the actual source documentation that relates to the outcome information from the registries going forward, and that's a second phase. The third phase will be having third-party auditors look at the information, but this is so external stakeholders can have some confidence in what we're reporting. We use a representative sampling technology, and so this methodology, rather, is a way for us to try and ensure that the information coming in is representative of all the patients that are being seen in that center that would meet inclusion, exclusion criteria, and it's a six-day rolling scheme where the first day of each six-day period varies from week to week, so we're not biasing it to a particular day of the week, and we've validated this particular model in a couple different centers. Site audits have basically been able to validate that their processes are intact, and we've been ensuring the data integrity. This is the result of our first audit, and basically, here's a summary of some recent information that we derived from the broader data set. Our data capture initially is excellent. Our follow-up at the specified time frames is also very good. Our random site audits have shown that our data accuracy and completeness has been very high, and the self-audits have shown that the data, overall, are accurate from a diagnostic standpoint, not as high as we would like it to be, but when we find errors, we go back to the sites and have them correct them, and we have a method for doing that over time. So, I'll just not get into this in great detail, but when we look at the analysis, interpretation, reporting of data, we have analyses related to quality, and we have analyses that are related to our registry outputs. The analyses related to quality, we kind of touched on a little bit. The analyses related to registry outputs are essentially two. There are descriptive analyses, say what you see, and then there are the analytic studies that we're performing on the data. In these analytical studies, we are looking at association between risk factors and outcomes, and we can use this for our risk-adjusted reports, and furthermore, ultimately, we can start comparing groups. We are not there yet, but this is where we want to go. So, we'll talk about some of these outputs in just a few minutes, but I want to mention that our outputs really relate to a variety of different needs. There are site needs, and the sites receive information that is related to their site-specific morbidity and quality data so that they can get more involved in meaningful quality improvement. We also are allowing information to start using this for public and private quality reporting. A number of centers are using this, for example, to participate in PQRS. It's a PQRS-qualified registry. There's also societal issues here, and we are using this information to describe real-world care and identify opportunities for care improvement in general across practice sites, and the aggregate outcome information will be ultimately shared with patients and other stakeholders, and we won't be alone in this regard. STS has been doing public reporting, voluntary public reporting of this for the last couple of years. I think something like 45% of the ST sites or STS sites are voluntarily, publicly reporting their information in a summary fashion. So, just a quick example of what our site reports look like, extremely detailed data related to the performance of these individual sites and how their performance compares to a national mean. And if you think that I just pulled the better slides from our particular site, you're absolutely right. I'm just showing you all our good data, but this is just an example of some of the information that the sites get, return to work after surgery, our patients, risk-adjusted national norm, for whatever it's worth. So, in any event, I'm just gonna go through some of the outputs very quickly, and then we'll try and wrap this up. When we look at our enrollment data to, actually, this is as of a couple months ago, 8,200 patients were screened, 7,500 patients enrolled. Our follow-up at three months was 80%, and we almost got to 80% at 12 months, but we were working on methods to be here. But if you compare this to the largest national registries, this is pretty darn good. Particularly for an initial effort, we now have well over 1.7 million independent variables collected. Here's our diagnostic distribution, really no surprise to anybody in the room. Spinal listhesis, disc herniations, stenosis, the major categories. We would have put the standard error bars on here, but they were just so small, it didn't make a lot of sense. They just made for messier slides. But these are our average outcomes over time. So the bottom line here is that patients are starting with fairly high levels of baseline disability. And regardless of the diagnosis, regardless of the procedure we look at, and let's just not look at adjacent segment disease here in too much detail, because we just don't have a lot of data here. But in general, patients experience improvements in quality of life, disability, a variety of other factors that is significant and sustained during this period of time that we're looking at on average. The other thing that we're noticing here is that there are opportunities. So, patient satisfaction. In general, 80% of patients say that they would have their surgery again, but the interesting thing is that a significant number of patients say that they would not have had the same procedure done for the same indication. And if we look at failure to improve over baseline, this is just failure to improve over baseline. This is with no margin whatsoever. This is just saying, folks, did you change at all from where you were, all right? I just find it fascinating that most of these people are having surgery for leg pain, almost one in five, are saying that at 12 months their pain hasn't improved. I don't know. I mean, to me, I think it's just an opportunity for improvement. We're doing a little better with respect to disability, but you still have 12 patients stating that they don't improve. I'm gonna look at that in a little more detail in a second. This is information I found very, very interesting. Again, this is just a pure descriptive analysis of the 12-month data as it exists right now. If you look at all the different diagnoses and you average that out, around 10% of patients, all the lumbar spine patients, are being readmitted at 90 days. 10%. So I asked some of my spine colleagues about this, and they said, well, you know, we saw this in sport and other things. And I thought about that, and I said, that's okay. It's consistent with other studies. But, you know, maybe not so much, Mia, because we're really looking now at a very, very large representative sample. These are practice sites, large, small, community, academic, it's really all over the board. This is real-world practice. In a real-world context, 10% of patients are coming back and they're being readmitted into the hospital within 90 days. I don't know why that should be an acceptable status quo. And I think that that's a remarkable area, an opportunity area for us. And then, finally, I just wanna show you this. These are individual patient responses to therapy. This is where I think our real opportunities are going to come from. So, basically, let's just look at ODI over here, disability, so anything below the line of equivalence represents improvement. Look at this variability. First of all, there's variability with respect to just indication. I mean, there's some people here who are doing better than I am right now. There's some folks here who are almost dead, and there's everybody in between. So there's tremendous variability with respect to who is being operated on. And then, if you look at this axis, if you look at the y-axis, you see the amount of improvement is very, very different. And, in fact, if we, and by the way, we see this in every center, with every diagnosis, with every procedure we perform. The same pattern on the scatter plots with respect to individual patient outcomes. If we apply something like MCID, which is to say, what would our estimate of a meaningful clinical difference, something that people could even perceive, these red dots represent individuals that probably are not experiencing a clinically significant amount of difference. It's an astounding number of patients. And I think that analyzing that and starting to understand who these people are is really going to create a lot of opportunities for us. I'm not gonna get into these predictors, because I'm gonna have to wrap up in a few minutes. We did start doing univariate analyses to start looking at individual variables that are predicting some of these more negative outcomes. And we'll be publishing some of this information soon. It should be no surprise that the more predictors that you have, the greater chance that you would have of a non-response the way we're defining it. Our hope is that we can convert some of this variation into value by understanding why there is so much variation in patient response. Mostly using multivariate analyses, we hope to generate these types of plots. In fact, we're already doing this the last two months. We've been sending risk-adjusted reports, ODEs, to various centers so that they could, in a risk-adjusted context, understand their relative performance with respect to a variety of different outcomes. Furthermore, this is where we really wanna go. And within the next two months, we hope to develop a patient calculator that will allow us to look at the relative risk of various outcomes based on specific patient characteristics. This is just an example. Patient one, patient two, both with fairly similar structural issues, both with similar baseline disability scores undergoing a two-level fusion. And you can see the likelihood, based on this particular prediction model, of a 30-day complication to readmit for those two different patients. And we're hopeful that this can be used to inform decision-making and, ultimately, to help with more efficient resource utilization. So if we just kinda cone this down to the N2QOD as an example of a registry, we believe that this is a robust, reliable clinical outcomes platform. We want to be using this risk-adjusted modeling to facilitate a variety of different things. Targeted quality improvement so you can understand your risk-adjusted expected outcomes and compare that to your observed and see where the opportunities for improvement are. Facilitate practice-based learning. Facilitate shared decision-making and effective resource utilization, perhaps through these risk calculators that we're developing. And then we are also using this to describe real care. What are clinically important differences? And we also want to identify large-scale improvement opportunities, such as the readmission rates that I just showed you. So I think this is gonna be helpful in a variety of different contexts. I look at this in this longitudinal way. Early on, our operations are gonna have a low bar for data collection, high cost, and we're trying to create a culture. Going forward, we're gonna have a high bar for data collection. We want to drive the cost down through a variety of mechanisms, more efficient data collection, economy of scale when we have more sites on board. And we're gonna create co-cultures. We're gonna have some centers that want to be involved in very high-level research, some centers that mostly want to use this as a quality improvement vehicle. With respect to science in the early phases, we're gonna describe care. We're gonna refine the diagnostic criteria. We're gonna identify major impact opportunities. Going forward, we want to not just describe care, we want to promote high-quality care. We want to share a common language, not only among nurse surgeons, but among other related specialty groups. We also want to develop a high level of evidence related to perhaps even comparative effectiveness. And finally, in the early stages, we're gonna define value with all these stakeholders that I mentioned. Going forward, we want to deliver value. So, I'm gonna wrap up by just saying a couple things about where we're going. We see that we can use this as a clinical research platform, particularly among these vanguard centers that we talked about a little bit last night that are achieving data collection on a very, very high level. We also want to be very sensitive to the fact that all of us are faced with a variety of different data collection requirements, and we're looking at ways of integrating these registry formats into a variety of things, from maintenance of certification to maintenance of licensing, even PQRS. I mentioned a little while ago that we especially has a variety of different needs that we have to respond to. We believe that embedding traditional research methods in these registry formats where we already have an existing infrastructure for collecting data can help us realize this goal of using registries as effective, disruptive technologies for advancing practice science. Their individual needs we're all familiar with, individual reporting requirements in that regard. The registry is now a approved method for satisfying PQRS, and ultimately we're going to go for this qualified clinical data registry status in the next round. But we do have challenges and opportunities, not the least of which is the heterogeneity and relative rarity of what we treat. We have to ultimately achieve the lowest energy state for data collection. There's just no question about that. Everybody knows this has to be embedded in the fabric of what we do. It's easy to say that. The technology is there to allow for it, but there are a variety of things. We could spend a whole lecture just talking about these challenges, but we're working through these. We're doing pilot programs with all the major vendors to try and facilitate automated data entry. And we also want to facilitate automated methods for longitudinal data collection, but you can give a patient a software program. You can put an app on their cell phone, but if you don't develop the incentives for them to give you the information when they're out at 12 and 24 months, it'll fail. It doesn't matter. Technology is going to help this, but it's not going to solve these issues. And then finally, we ultimately have to translate the data into meaningful quality improvement efforts. And so I'm not going to go through the regulatory issues. I will just tell you, if you're interested in these things at all, the regulatory paper that we wrote up describes our experience with the federal government over eight months. It ultimately culminated in a meeting at the White House where we resolved most of the regulatory issues related to this. There's only one site in the country right now that has described this registry as human subjects research out of, well, there's 53 participating right now. About 60 have had this go through their IV. So one in 60. But the reason why certain individuals see this as human research and some individuals don't is worth discussing. And so you can look at the Journal of Nursery article if you are interested in getting that into detail. So in conclusion, all of us in the context of this practice science paradigm need to think about the best ways that we can define, measure, and continuously, not just in short periods of time, not in a time-limited context, how we can continuously promote quality care and design and implement efficient care processes. Bob went through this last night. I love this concept. I love the definition. This is what I think our ambition needs to be. This is what I think that we all need to aspire to going forward, particularly the next generation of surgeons. We described the practice of science in this paper in JNS a little while ago, and you can read that if you choose. Numerous individuals have been involved in putting this particular effort together, and I just can't thank everybody here, particularly Bob, for their support. This is our website if you are interested in getting some information, particularly about our national registry. And that's it. So in any event, I really appreciate the opportunity to participate in this forum. I just think this has been an extraordinary exchange of ideas. I've learned a tremendous amount myself. I regret the fact that I have to go home this afternoon because I'd love to participate in your conversations about clinical trials, but if anybody has any questions about registries in general, about some of the stakeholder relationships that we've developed over time, and what I see as our major challenges, at least on the national scene, or the registry as it exists. We've had a number of young surgeons get involved with data collection, data analysis. We're now just doing the deep dives in the database. There's really very, very interesting things that we're beginning to pull out of it. Any interest along those lines, please feel free to contact me. And thanks for your attention. Thank you.

clinical registries

designing

stakeholder involvement

data collection process

patient-reported outcomes

quality improvement

patient outcomes

future directions

data integration