Thanks. A lot David and thank, you for taking the time I also learned, that this. Will be streamed to some outside, sites, so also welcome, for to, the outside people who will take the time I know everyone's. Very busy. That's why that I up there is in red because. It's something that we never. Really pay attention to the business of pathology, but also the business of doing that. First. Off I want to mention that I do not have conflicts, of interest and that has a specific, meaning to me because when discussing, things with payers that is always one, of the top items. I. Want to start telling, you something, how I got interested, in this and it's also part of a mentor. You. Know recommendation. So. After did my postdoc, with you know publishing, some papers that I'm still very proud of and. I thought you know I could totally change the world one, of the things that John told me about is, sort of so, what what, happens now what happens next the. Big promise, of translational. Medicine is that we want to change clinical, medicine, but, how to actually innovate, in, clinical. Practice is. Very tricky. In particular. When you think about your brand running out your, lab having published something you have to get funding, sustained, but. How to go through that door of making it really change, clinical, practice, I realize. That financial. Sustainability is. One of those key elements you may have a grant but the grant may run out and you, may have demonstrated, clinical, utility by, how to actually, get into clinical. Practice, to. Me was kind of like completely, uncharted, territory. So. Simply put what we do in the Center for integrated. Diagnostics, is to integrate new diagnostic. Approaches, into clinical. Practice now. That may, not be something, new and you all may, have done this in one way shape or form but how, to really make that systemic. And operationalize. Operationalize. Innovation, is something that we, tried to do so. From. An idea. Via. An analyte, to. Let's say a clinical, trial or publication. I think is familiar to all of you and then that, test may be validated. It could be an FDA approved tests and I get to that in a second or it may be a laboratory-developed tests, but. Then at some point it is probably. Live in clinical practice it. May have to require some funding. And to sustain it but, then at some point it, will be utilized and then you have to track whether that makes sense I don't. Know whether you thought about discontinuing. Some testing, that relatively, rarely occurs but, I think that actually is, part of you know outcome, tracking, and some, of the possibilities, so how does that actually work so with, any of these let's say innovative. Components. You, have to have some sort of key elements when you want to integrate things in here now. Having. A science background, I thought initially I have to refine the scope or I have to refine, the cost but.
The Extra thing is like you know you have to have a great team you have to have the time to do it and all, these elements have to be fully, aligned to, really make, sense and make something sustainable. So. The first element that I want to discuss is what are the major challenges, that. We need to achieve financial. Sustainability. So. Some of this may be basic to the people, in the room that know or the people online that know what is, necessary. Now. We know that for example and, I use the example of next-gen sequencing but, please replace this with your technology of choice that a. Perceived, value of a technology may. Be sufficient as emerging, evidence that. The AMA, who owns the CPT, codes may release these codes that. Means for you. Know many companies, and vendors they may say you know our new technology NGS. Matched. Up with the corresponding, CPT. Code that, is a necessary basis. To, claim. Certain. Things for example do. The billing with, payers that, however. Doesn't, mean that you get reimbursed, it is just a necessary but, not sufficient pre. Requirement. Now. The scientific, community, is driving, that meaning, you know representatives. In the cpt, panel. Will, drive certain. Patience or CPT codes. But. One. Element that is often, unrelated, to this is that the technology, may. Also, have to be safe. And effective so, a lot of in the. Discussion, of you know new technologies, and CPT codes one thing that is often related and, also. Important is the FDA. So. When you think about the. Implementation, of certain new technologies, in the market there's two options one is you. Undergo full FDA, review, and approval and then you have a kit that is FDA cleared or, approved, and you, can distribute, that in the market you know across state lines and you have a marketing. Authorization for, a certain thing for example advice with an intended use. There's. Also a pathway where, the FDA, exercises. Enforcement, discretion which, is called laboratory, developed tests pathway, there's. A lot to, know about LD TS but it basically means for one specific, setting, meaning your lab for your local setting and that. Is this. Thing. Called laboratory. Developed tests, both. Are currently coexisting. And I get into some of the more recent developments. Soon. Now. One thing when we talk about integration, of new diagnostic, technologies. And I, emphasize that some of those of you who want to claim Sam's and CME the. Bottom line here shows one of the, potential, answers, is that. You know for some of these very, new innovative technologies. Just. Because they are emerging, there, may not be the option of having, an FDA cleared tool. Or device or technology, so. The default or de facto rule. If you're developing something in-house, is, that. You know the most common reason for choosing an LD T is because there is just no FDA cleared, component. To this so. There's, a lot of controversy about, what what is related, to the FDA but. When it comes to the business. The. FDA Product, Approval itself. Outlined. Here, it. Is related, to the technology, but not necessarily, to reimbursement, it. Is encouraging, to know that, something, is safe, and effective and that may play a role. To the people who decide, whether they want to reimburse for that for example recent. National coverage determinations. By the CMS, encourage. FDA approval, for example, for next-gen sequencing unless. Otherwise you know locally regulated, so. When you see for example when you get an email from let's, say a, oncologist. You know that, takes care of bladder cancer patients, this, was last year the first-ever. Approval. For a targeted therapy and, bladder cancer they. Say well this is now FDA approved can we get the test and is it reimbursed you know will my patient get a bill those. Are probably three gigantic. Documents. That you have to dissect out it's a very simple question but it's completely different, milestones, that have to be accomplished. For. Example is this test now covered, has. Two interesting, meanings in the context, of next-gen, sequencing first, is well is it covered meaning, is there coverage, on the essay the, other part is will, medical, insurances. Cover this test in their policies, this. Is contrasted, to illustrate, that there's, two completely different things one is you have to have the essay ready to go that. You would identify a subset, of patients that may benefit, from this new you.
Know Safe and effective at least approved therapy, on the other hand to. Get to this reimbursement. Stage you have to have the payers buy, into, that meaning, they, have to decide that in their you. Know group, of patients that this, may apply to they. Consider, it medically, meaningful, enough to, cover for that test. So. Effectively, we have on the side to, the right. The. Payers. Determining. Whether they want to cover the test via. A medical, policy so. Only if you have a technology or test or device plus. A CPT, code plus, a medical, policy and a setting meaning. The medical policy would decide. Whether that setting is appropriate, only then you. May get reimbursement, the rate meaning, the dollar amount is then contingent, upon whether. You have a silver or gold or a platinum plan, for example with certain insurers, so. The reason behind this is and if you put on a pair hat is there's. A big population. That, has to come this cost while. For, some settings, only a small subset, of people, for. Example the scientific, community may actually want to drive this, so. That is I think in, a nutshell the. Basic principle. Of the. Reimbursement. Structure, at. Least when it comes to diagnostic, testing. Now. I want. To dive into two or three elements, of that a little deeper to give you an idea how, diverse. Or how this, diversification, of, billing, and reimbursement, has, become. For. Example in molecular testing. The. AMA, has established. Several. CPT. Groups, so, these are code sets so. In surgical, pathology there, are certain tiers etc, but over time from. A molecular, test non-specified. We're. Currently dealing with I think it's over a hundred now, different. Subsets, so there for example mo path codes, molecular, pathology, codes those. Follow the classic, tier 1 and tier 2 rules where. Tier 1 is for specific genes, and here 2 is by, complexity. Different. You know single gene tests, and it's graded. Into nine different layers, of complexity, in terms of technology and or. Professional, interpretation. Complexity. But. There are two additional codes that I want to bring to your attention probably. You've heard if, you do next-gen sequencing that, there are next-gen sequencing codes, these, cover panels, as well as exome or genome, including. Things like. Reinterpretation. Of existing, files, meaning. Where you would take a genetic. Sequencing. File and reinterpret, the findings for example, due to new, discoveries, or you know better classification, of variants, however. One thing that is not, widely known but it's really important, is this group, of maaa. Codes, which stands for multi, analyte, essays, with. Algorithmic. Analyses, I know. That's a mouthful but it means effectively. You have a bunch, of data and you have a lot of complexity. And you build an algorithm, that is able to interpret that so. A lot of people consider, some, of the machine, learning tools. Likely. When, related, to molecular data to potentially, fall into this code set. Unless, probably, otherwise classified. So. That is the overview of the CPT codes and all the relevant let's, say, elements. That have to be in place, so. What I thought of sharing is in. The next slide a relatively, complex diagram, of how. That would play into, the entire field, of what we're talking about so, to, go, through the logic of this scheme so. You've all been patients. Meaning. You saw a physician and the physician may, have ordered a test the. Test may be accession. Then. Comes the medical procedure potentially. A report, that. The physician receives and then discusses, with you so this layer here, is, what I think everyone, knows the. Components. Here are this. Blue box which could be effectively, any medical, procedure so. You can replace this with any medical, procedure and if, it has a CPT code it is officially, registered, and available. Underneath. That layer is a gigantic, IT layer, meaning information, technologies, and computer, systems I'm. Sure. You have googled, something today, you know most physicians in, clinical practice use, Google every single day but. They also interact with numerous, other, medical. Information, technologies, for example your laboratory, and or, your Hospital information system. And possibly some data marts. Interfaces. Or other resources. Obviously. That has to be maintained, and it glues everything, together now. Underneath that layer comes, what, is what can be summarized, as the revenue cycle management layer, this. Is all the other stuff that has to happen to. Get a test financially. Sustainable and, there's. Several components I want to illustrate the patient may you may want to know how, much is my out-of-pocket, cost estimate, I would. Venture to say that while, probably, many people in this room do diagnostic.
Testing Every single day but. If I asked, what is the out-of-pocket, patient. Contribution. To that test for a certain payer for a certain CPT, code I don't. Think anyone knows it's, very very complicated, but. For an out-of-pocket cost estimate, service that has to actually take that all into account. I think it is absolutely, fair for a patient to ask that right because your ultimate, ly responsible, for some of the cost the second. Step which, we will discuss in a little bit is prior authorization. A process. That was initially designed as a utilization management, tool, that will contain some testing, to make it only applicable, to meaningful, tests and you, have to get pre authorization, before you. Actually get the test reimbursed. Then, comes the entire billing and reimbursement with. All the Magnificent, processes, billing. And Finance integrity, and ultimately. At the end when a patient gets a bill there has to be an accounting, service and a financial clearance, mechanism, to make sure that. Patients can pay and all of this has to be tracked. Briefly. For a test this, is a next-gen, sequencing platform. These, little figures shown here are the, people and that minimum, postgraduate. Years, of experience, to maintain this pipeline if. You quickly sum this up for, a next-gen sequencing test, to. Run in clinical, practice you, need about 55. Postgraduate. Years of experience, to maintain that just. To illustrate what, a high complexity, test, requires, in clinical, practice. Now. Having said that I, thought. A calming, read slide would be just the best thing afterwards. So. How are we how. Are we maintaining this, and what are we doing sort of you know in this lab and. With. This I want to obviously thank the team but I want to illustrate how. We try, to set this up and it's in evolving, but, I think with, the next two or three slides I want to show what that means, so. The lab itself, with, all the different people consists. Of a research lab in. This case portrayed, as Jonah, Freddie's lab and the, people are trained, to. Effectively. Provide certain, new developments. Clearly. With the roadmap the, TrueNorth, pointing, towards clinical integration, so. Recently for example one, of the postdocs, developed, a cell-free diagnostic. Test that, we could easily roll into the clinical development pipeline to launch it and i'm talking about the technology, the, non-technical, aspects, of will cover in a second, we. Have a core lab for research support. Trial. And by OMA designs, and then, we have an actual. Clinical, lab, organized. You know as you can see you with several senior technologists, a technical, director etc, we. Have several diagnostic pathologists. And we aim to balance. The number of pathologists. With a similar number of biotin format assists, the team currently. Directed. By. Marcus Herrmann is. A computational. Pathology, division and it's. Not only by informaticists, I think, to most people, in the room you know that computer, scientists, fall.
Into Numerous categories you have to have web developers, database, managers, coders. Infrastructure. Maintenance, people people, managing the lymph system etc. So. What we're trying to do and this mirrors from the left the ideas to, the right actual, implementation. To, try to maintain this team and keep it as innovative, as possible, meaning. One element, that no one wants to do is all the boring stuff but. To innovate things in clinical practice there's, a lot of boring stuff that needs to be done to. Illustrate that concept, let's just say we take a new test and we, treat it as a black box the, black box functions, and we have to assess that separately, but. What else is needed if you want to implement that black box into clinical practice. So. We organized that in a spider map or spider-man, meeting. Where. We get together and, gather all the different information that's, needed and we. Streamlined, this almost, in a blueprint that you have to have very little work. To do so. The the, idea is for example certain, tests have certain requirements, to. Clarify those that can be done before the meeting for, example what's the anticipated, volume what's, the turnaround time needs who. Would order it etc so, the in, FDA terms intended, use and specimen. Requirements. Have, to be clarified because otherwise this may not work, example. Is if you have a great Alzheimer, protein, test that requires 22, grams of human brain that, may just not work. Or. You. Know the platform, selection, obviously, is familiar. To all of us but the platform has to be available etc. For. The validation one, of the things I want to point out is, it's very hard to do a validation without, the, samples, being available so. If we have a test that's really great but, we don't have any ground truth samples, that we can rely on it's very hard to validate that and then. Moving forward the SOPs, and the write up basically. Have a, standard. Operating procedure, on how to write standard, operating procedures, it's, a matter procedure but. You know it's it's streamlining. Things we can implement a new test I basically copy and pasting it in and following. That blueprint, so that it's just available in. Addition setting, up the billing we have to get charge master code cpt, codes if we don't have that it won't work, personal. Training proficiency, testing the requisition, has to be built out in epic just. Having a standing epic II care team order. Meaning appending, you, know ticket is helpful, and then, someone ideally, early on has, to find a clinical, champion, who, can educate, and tell the people this is when or why you should order the test I, don't. Know if you find all of these aspects, boring, but, it actually has nothing to do with, the performance or, the quality of the test but at least for some tests all of this has to be in place, ideally. From, the moment that you want to have this spider map meeting, to, actually be available and someone has to do that for. Us it worked really well because we have some people that really like billing, and. Then they take that on and just go with that others, like the SOP, write-up we, have a great QA QC team who, will streamline the process, and workflow and what, to do when is kind of the magic sauce. Now. This meeting, is, outlined, in red here, and this is one of our steps, in a multi-step, process, and. I wanted to show you this and then illustrate, it in an example to show why that's so critical. So. If you take an idea and you really want to check whether that is possible to innovate in clinical practice we screen. Immediately in, our healthcare system, with, the appropriate, people in administration. To. See whether there, is a pathway to financial sustainability. That. Means we want to check with them early, on is there, a, contour. Can be modified that is their CPT code can, we go into the current negotiations, with the payers and can, we start that early on even, before we commit to that project, on a new essay so. This happens during, the selection very. Often we find that there is no appropriate CPT, code that means we wouldn't even take the project on for clinical integration we. May still do it but then on a research basis, as, soon. As we select an initiative, we, budget it out and we only commit currently, with. The team size to three projects, going on at the same time while. One always has to be at the very final stages, you've. Seen the project. Mind map the. QA QC procedures, and these steps should make sense and then we deploy it in clinical practice and this.
Year Is probably in fda terms real world evidence, meaning. We launch it and we see whether it works we. Track whether the payers and how they react we. Collect that data and then make a rational decision whether we have to put more emphasis on. Efficiency. Quality or, other things or whether we want to renegotiate with payers basically. What happens, in clinical practice. What. Element that's really tricky we're currently trying to figure that out is we. Want to then hand, this over back to other people meaning. Can we give this to someone else to free up the team to do more innovation, that's. Kind of the output. To, make roughly sense I don't know it. To. Give you an example of how this, works in clinical practice, there's. The. Setting is lung cancer you know next-gen, sequencing and, the, value of that has, been established you seen a top road of rough, work flow, now. Given, that EGFR. Mutations, are clinically, actionable one. Request was what. Can we get possibly. Does EGFR, mutations, earlier. Now. Keep. In mind this is next-gen. Sequencing which, is from, a business perspective challenging. To begin with but. The clinicians, and I think rightfully so, asked can we get a rapid, EGFR, essay beforehand, now. Two, things were, clear that we needed to learn here is one this, essay could be rather to be simple and if mutant, we could make you, know for an actionable mutation, the, treatment decision, clearly earlier. Now. The, second part is that this, perceived, turnaround. Time of NGS, testing, started. Way earlier, basically. When. The clinician says, I'm thinking about a biopsy. Specifically. To get the genotype, to make a treatment decision, so. From an oncologist, perspective, that time of idea, meaning. Or the time, when if when you know I have that treatment, decision, in mind. Starts, to turn around ten o'clock and you. Can tell on the pre analytic side as well. As on the post analytic site there are some delays that add to, this perceived turnaround, time so. What we did is we, wanted. To kind. Of get to this to. This workflow much faster, after. The rapid EGFR, testing. Was established, that got us basically, this Delta, we. Implemented, the workflow that collaboratively. Involves. I think two different divisions five or, two different departments five, different divisions.
When. Oncologist can immediately schedule, an interventional, radiology, table, once. The diagnosis, is confirmed by a side of pathologist, an extra. Core. Biopsy, goes into sailing goes into frozen section, the. Cancer is confirmed, just whether it's present, we. Cut extra sections go straight into extraction. And get. A molecular result, out only, with the confirmed, diagnosis, of cancer but, not a final, pathology report, and end. The decision here can be made even easier obviously this is a lot of operational. Efficiency. But I'll show you what happened so the. Non rapid testing group because not every single patient was available for this we. Made a so, this is the time from, you. Know ordering, - getting the patient on a tyrosine kinase, inhibitor, was, in, the control room about five to six weeks the. Rapid reporting, took roughly one week with the rapid test and then patients went on drug for about three or after, about three weeks, the. Ultra rapid workflow, worked, that we got basically the rapid testing done in one day so. The treatment decision could be made right the next day and and this yellow delayed. Because, of you know payers having to agree to pay, for the tyrosine kinase inhibitor, so. Ultimately we beat our own system, the rapid testing with ultra rapid testing where we get patients on drug about a week my. Dream would be to get patients on a drug the same or the next day but that obviously requires. Additional, modification. Now. I mentioned that we do this while monitoring. Reimbursement. So. What we checked here is what, happens, to these claims, that we submit, while. We're improving this let's, say as a clinical, initiative so. What you see here are twelve private, payers in Massachusetts. And the. Claim that we're. Reimbursed. So. You see some payers really, do not believe that this is meaningful, or, just. The indications, weren't right or the medical policies didn't cover that or the patients plan etc, but. Other payers adopted, that some at a very high somewhat medium and somewhat slower rate overall. About 63%, of, the, claims that we're submitted, we're, actually reimbursed, now, you can say well that's not enough, or that's not good, but at least for sustaining. A clinical operation, this is sufficient and it comes down to contracting, how that is possible. So. Given you know some of these clinical, benefits, we. Now can approach some of these payers and say look this is what we accomplished, obviously. Some payers will say well does that equal. A survival, benefit, and others. May say well this sounds really interesting yes we're supporting it, it. You know it's kind of variable, on how payers react but we at least have very, good real-world, evidence on how to negotiate with, the payers so. What. We try to derive, from this is that this is a transition. From. Let's say a local. Siloed. Localized, endeavor. To. Spreading, this innovation, into practice, because once the policies, are out there they're, available for all the patients not just for our lab so, this is actually innovating, outside.
Of Our realm as well now, what we try to derive and, this, is a paper in the oncologist, 2019. Is. A. Paradigm, for this financial sustainability. Now the paradigm, works as follows. On. The left you. May have your new program, or initiative, meaning. It may be a completely, basic, science project, or it may be something for clinical, practice and then. On the right side you have a potential, funding source meaning. It could be the government it could be a philanthropic organization. Etc, now. While. You want to deliver certain things, the. Funding source may have certain aims or goals that you should fulfill basically. Outlined, in an RFA, for example, for a certain grant, now. The shared element between that is the value proposition it, could be that you just want to understand, how certain, brocco mutations, work or, it could be that you want to improve the health of patients with bracha mutations, by giving, a certain treatment, the. Idea is if you draw a line from, your let's say want, to. The, funding source and it deviates largely, in the y-axis it's, going to be very hard to align that the. Close-set is horizontal, or the closer you line your value proposition with, the deliverables, to the aims and goals of the funding source the easier it is now. What, happens with any of these projects, at some point this. Shifts, from understanding. To. You know changing, it to outcomes research, to actually making it a sustainable, you, know thing where payers want to pay with, a heavy. Interest by patients, that that would be the case so. What we aim for is to create, real-world. Evidence by whatever means so, that we can go to payers and say look at this data it. Is really important that you do this and ideally. The data is so good that then the payers can say no and thereby, you shift the, financial, you know burden from whatever initiative, you have to. A more sustainable one I hope. That makes, sense I think, that values another rad slide. So. These are I think some of the considerations, that we took. Into account to, move. Towards, financial sustainability I, want. To approach one topic, now that is somewhat, distinct. And that, is machine. Learning and some of these innovative, technologies, that we talk about now because. They are very, very different, and distinct, and it's very hard, to show some of the value at it but, I will try it so.
Simply. Put and I think you see this in the background kind of an old matrix reference, you know are, you choosing the red or the blue pill this. Is referring, to are, we reporting, something or are we not reporting. It something in the medical record that, is effectively. A I, don't know a computer, scientists, approach to determining, what a pathologists, job is I don't, know very oversimplified of, course now. There is a decision tree underlying, that, illustrated. Here as a random forest tree and when. You simplify a pathologist. Job like that you. Can possibly. Emulate. That with a computer, tool meaning we. Simulate, the pathologist, ability, to put something in a report yes or no using, a machine learning classifier, so that. Backs the question well how do you do that what. We did and this, was a work by maksim near who was you know one of our summer. Interns was how. To implement, machine, learning in genomic medicine now. This is a diagram. Of the six. Elements that you have to have in place so. First you, have to apply it as somehow whatever, the application, is in terms, of reason and in terms of infrastructure. You. Have to download a couple of libraries you. Have to have data the, data actually have to follow a certain data model and I get into that in a second and the. Model then has, to be tested, probably, trained, optimized. And then you have to look at the results, so, simply put this is similar, to any of the other diagnostic, tools that we use and we assess them in the same way it's. Just a computational. Tool now. Having. Said that we, considered. Our next-gen sequencing pipeline. Which. Is you know a certain amounts. Of variants, and. Trying. To portray this in a slightly, different way is what we do if. This. Is the raw, output of our, genes over sites, for, about 19,000. Variants, what, pathologists, do is we screen these for, what. We do not want to put in the report so, that what's left is what we actually do report, so. This is like effectively, manual, filtering, of reviewing. Variants and not reporting, those, so. When you think about this from an AI or, machine, learning perspective, we. Cannot possibly take into account all the different, outputs from a computer, nonetheless. All of these outputs are discrete in nature meaning, from a computer science perspective these, are all already in certain, bins and fields and cells in you know VCF, files etc. So. Given. That we had with, our team the, necessary. Computational. Resources, we. Thought that's an ideal setting to, kind. Of optimize, and test whether that would work so. Simply. Put. The. Human, decision, of yes putting, something in the report and no we, tried to model using a machine learning tool, so. To to, get to this component. We have to obviously understand, what is a problem definition, for machine learning, so. Largely put and this is from Jason Brownlee. Simply. Put machine, learning tools or machine. Learning itself, is a technique where a computer program, learns with experience, that. Experience, should improve the power of a task that you predefined, and this. Task or use, case if you wish you. Have to obviously be really, careful about how to define that I want. To give you a simple example so, why do you want to solve this it, may have a benefit, or a use case if. You think like a computer scientists. That may be slightly, different I want to illustrate this, with this clock example, a benefit. May be that you know the time for example that you don't talk too long when you give a speech and then you check your clock or, a. Use case may be that you use the time to wake you up in the morning those. Two things are fundamentally different, and it's very important to predefine that when you talk about you.
Know Certain things especially. When building a machine learning algorithm in, addition, while. Building this out even just manually, you may actually reveal, trap domain, knowledge for example in, the, case of you, know what, we built here we found out that certain elements, are not outputs. Of the pipeline, but actually previously, done by a pathology, for example, tumor. Type or, tumor. Percentage, etc. So. I thought it may be helpful, to say that these individual. Components. Of a machine learning project, would. Then be unfolded. And done so, we took, our dataset, which, actually takes a long time to build but I want to show you what that looks like and I. Don't know whether you're ready for some code but I thought you know Wednesday, afternoon why not look at some code why not. So. The key. Elements here and this is from mix Omni R is you, install your libraries which is effectively, you know launching, these different, functions and then. You prep the data to, actually look at what that means so, you basically, take a look what your data looks like and that, is basically, already, 80%. Of the work. In. A very good Forbes, article from 2016. There's. A great breakdown, of what, data science, spend, most of their time with and given. That you all have probably published, something and you probably all used Excel, this. May resonate with you because you probably spend a lot of time mining, and cleansing, data looking. Up ages or looking up certain clinical features so most of the time before, you can actually analyze or build anything is spent on cleaning, cleansing, and collecting, and finalizing, data. What. Is a trap. Domain knowledge in there that, what you're effectively doing is you built a data model while. You're doing it because very rarely we know about the data model before we do this but it's, evolving, over time and then the actual fancy, stuff here is taking. Up much less time than probably, initially, anticipated. This. Has some practical and things which I just want to briefly mention here. If. You're hiring someone who really wants to do machine learning and it's an expert, in AI or, you know has highly you know diversified. Skills in that and ultimately. The person is only doing you know 80% data, cleaning that may not lead to high job satisfaction, just. A small hint right, make sure that you hire, right and know what the person will be doing. What. Happens then is you encode, the, data meaning. The data itself, may be discrete, but. Have to be turned into zeros and once one, hot encoding, as a password and, then, you initiate, a grid search where, you define what, your outcome is and how many iterations you, want to do and tell. The, computer program when to stop so. What we modeled, here is the f1 score and we. Mentioned that you know you should do 35. Iterations, of learning learning, from experience, and if. You see that there's no improvement after three additional, iterations stop. And. You see that after 16, rounds, of training it converts at 93%. And, then this didn't. Improve any further you. Can use different predictors. And classifiers. And I won't go into any of these details but. You see that depending on which model you choose to, model our reporting, decision, that. May differ now. What you end up with is a certain, performance, as an area under the curve for. This model to emulate the human decision. Then. You still have to decide on, a cut-off meaning. You have to say am I using this as a screening or as a confirmatory, tool and the. You know performance. Metrics are shown here. We. Decided to implement it as a screening tool meaning. We were totally, okay with having a lot of false positives because, we wanted to find things that we may miss rather. Than confirm, what we wanted to see and, why. We were doing that the, output, was, obviously. The aggregate, model and you can see here is you know not reported, versus reported, but. The interesting thing is in our practice, we had at the time six pathologists. So. The, computer scientist says well I can also derive. Pathologist. Centric, models, and that, was sort of an oh wait. That. Means you can give us the performance of different models based on different reporting, decisions, by different people which. Is what we do when we go for example in a consensus, conference right, so. What we did is we did an aggregate model as well as individual. Models and some, people sign up more some people sign out less and you can see the model performance, here and what, we then decided, at the time to do is for, any variant, we. Implement, this as a in a graphic user interface did, you get a basic, on-the-fly.
Consensus, Call based. On six different models, derived, from the pathologist's. And then, you can go and say hey. John why didn't, you want to report this and, John says well I haven't even seen the case well I mean your model like can you explain this right. So. We realized that it's not just a decision that we're interested in but, that we want to go one level, deeper right we want to understand why did the model say. 99 percent yes, and then we wanted to explore these decisions, so stiff. Slightly. Portrayed. Differently, now. We're suddenly not building, a tool. Replacing, the decision, but we want that we can explore, the tool and the, underlying, features leading. To that reporting, decision so, basically, going backwards, and, this. Is now implemented. So, whenever we get a variant, we. Get the, percentage. Or ratio, of should, put it in the report as close to one not putting it in a report is zero and you can click on this as link and it opens the random forest decision, tree and you, can actually explore, the individual, features and why they. Were chosen, some. Of these make sense and, I intuitively clear, for. Example yes you should report b-raf, because, NRS wasn't, present but, others are so complicated, that you know you would have to know I don't, know so meta LR, rank score some that is a composite, of five different values, that is in this case something. Here for example a primary. Call or low freq was. One which is larger than 0.5, that may not be intuitively, clear to everyone but that is a contributing, feature for a reporting, decision. Now. Once we had that that was obviously, groundbreaking. Because then we could see why a machine makes a decision on top of the decision that you may render which. I don't know you have to experience to fully understand, that it's phenomenal. But, it also, prevented. One thing from happening that. Was we were hoping to demonstrate, it will become more efficient. But. That was the case because we spend our time looking and understanding, the machine which was you know a completely, different thing but you know we should have thought about that but you know failing. Early right any. Case then we came across something which was also great which is that, we switched, ese now. Initially, our si contained, 39, genes. Excluding. Braca 1 an ATM which are you, know Brecker 1 & 2 in ATM a gigantic genes and then. We expanded, the panel to 816. Gene panel, it. Comes with the spider map meeting with, all the revalidation, cetera. But, then we thought well what would happen if we just release the model as it. Is on a new panel so, this is equivalent to a molecular. Fellow, switching, from one institution to another right, learning, with experience, so we were anticipating that the model will not perform well the, computer, scientist said when he began have, to retrain, right. And we said no we don't want to retrain we want to test what.
Happens If, you would expose the model to something else and here's. What happened so this, was v1 trained ultimately. Over, I think. 19,500. Something variants and then. It was not retrained, applied to the new version which is a transfer. Test if you wish and obviously. The performance dropped, and then, we just checked what happens if we, retrain, the model on t2 at 568. Variants which is far. Too low. In terms of number two really retrain, something, but. That was you know the result so. Obviously there are some, limitations to this but I think what's important. Here and that's probably the you. Know take-home. Message that is you, need good data models to really tap. Into this you know benefit. Of some of these AI tools, now. One data model that may be of interest more, on the AP side but. I heard there are some developments, that may be. Relevant with respect to anatomic, pathology I thought, I mentioned briefly that for. Image data and that is also relevant not only for ap slides but, many other gross. Images, clinical, images is. The DICOM standard which, is you know digital communication. In medicine for imaging data and. Really. Capturing, the metadata is one of the very, important, components and, this, is effectively. A very, complex, data model, and an evolving standard with. High relevance so. I hope that this. Part here was clear, that some of the tools applied, be. It cold or Beadon you know anatomic si are really important to really unlock some of these potentials. Given. That that was really easy I thought a blue slide would match. In. The, last couple of minutes I want to come, back to how to merge. These two areas, so. In the one hand we have cutting-edge, technology, machine learning highly complex stuff computer, science that, could be that black box in the center while, on the other hand we have a very you, know streamlined, development. And implementation platform. Meaning spider. Map and all of that well. Clearly there's a surrounding, environment, that could be considered, the you. Know framework. Or landscape, of that. So. To kind of outline, this, I thought. There are different, levels and I don't know whether this will resonate I couldn't come up with a better term but so. In the one hand we have gigantic. Super, high level meaning. Global. Or international level. I think it's fair to say that machine, learning and AI is, an, international. Search a lot of interest, in it but.
You Know this may apply to other things as well value proposition, safety, quality, etc, so. The top part, is, very. System, focused, and it's a long term strategy if you, want to change a field in medicine at the international, scale, good, luck you, have a lifetime ahead of you on. The other hand you know you can trickle down regional. Local system, institutional. Departmental. Divisional, I don't know your private life and that goes really small and like order tests, or even out-of-pocket estimate, individual, fragments of something, this, is bottom up at hawk or you know you could say patient centric and, here's a quick thing, you. Could talk to insurances. Which takes a long time but. In absence, of a, good policy you still have to figure out how, to navigate individual. Patients, with their concerns etc, so, I put here that you know a robust, appeals program to. Overturn some of the insurances, decisions, can. Be an effective strategy so, meaning can do two things now. Here's one element why I'm, really fascinated about this business compose, of this entire innovative, pipeline, and that is the following. In. The following two or three slides I want to try, to convince you that, personalized. Medicine, and a. Big promise of that actually. Requires. Personalized. Billing, or financial, clearings processes, because without that you can't really realize it as in the sustainable, fashion. So. Instead of explaining. What, I thought. Is underlying, this I want. To show you what you need to do to actually get there and then it may hopefully, make sense so. First you have to develop a payer matrix. Because. You, will see a lot of patients with a lot of tests and requests, that may or may not make sense, so. Here's an outline that we did and this is a few years old just, imagine, this map, fluctuates. Like traffic, lights, these. Are the 12 payers these are the male molecular, CPT codes some. Payers this, follows a traffic, light logic, so rep means not covered, yellow, means prior off green. Means it's you, know okay according, to policy these. Policies, change constantly, and not all, at once and not by payer and not according to any rules or regulations. It's just fluctuating. So. You have to have individual, you. Know prior autumns that know the policies, and deal with that what that looks like is, if. You don't have prior off you order you, do the procedure and then, you get the claims management with prior off you. Have these different, you, know rules and, appeals. And you know prior off may or may not work and this. Is relatively. Complex, the. Prior, off itself, sounds like a very, simple thing but when, you look at what that looks like in clinical, reality it looks something like this, so. You first from the order have to understand, what what payer does the patient have what, policy, is the patient on you, know and then you look up all these things to ultimately, at some point arrive at well, here's the date of service now, we can actually submit the claim this. Is what, the reality looks like this is the concept, what. Emerge from that is that a lot of people including, insurers. And a lot of professional, societies came together. Said why don't we streamline, this guess what there are some great technologies, for example machine, learning that we can use to do this so. At some point some of these technologies, merge with some of these requirements. Now. At the same time there's a lot of pressure and there's, one thing that will happen I don't. Know whether it will happen this year or not but I wanted to briefly mention it, one. Element that is especially, happening. For high complexity, tests, is a. Proposed. Legislation, that's currently existing, in draft form you, can google it it's valid, Act if you google valid act you get the current discussion, draft, that. Is simply. Put proposing. A pre-certification, for, labs to. Basically demonstrate, to the FDA that there are, good. In doing. Certain. Components. Meaning that they are able to do assess. Technologies. In the right way by, submitting it to the FDA the FDA would, review it and hand. Out something, akin, to a driver's, license, where then you can move forward in an effective, way this, is by the way absolutely I, don't, know I'm telling, you like a vast, oversimplification of. Multi, hundred page document. But. I think the actual, components, here are not necessarily, good or bad but it's just a force that is to. Be recognized whether the FDA and, how they would regulate is really complicated, I think, why I wanted, to show this when, it comes to integration, of new technologies, into clinical practice is one.
Thing That we all kind, of ignored and I'm guilty. As charged. That is we're. Not really providing, regulatory, science, input. Into, regulatory decision-making. Meaning. We believe that we can do it and a lot, of I don't know I'm all I think ofda, so complicated, but, really thinking about how to improve the field would also mean. Challenging. Some of the ideas, and also accepting, some of the ideas so. What we did is we tried to start a collaboration, this is including, the, medical device innovations, consortium, also people from the FDA to really. Clarify and improve some of these pathways and, while. It may not resonate why, regulatory, science is important, I think, I want to briefly mention that we have participation. On a lot. Of fronts and, maybe. This is kind of a vision. But. I think you all know that the, diagnosis, is what's driving our, practice. And a. New technology, for example next-gen, sequencing has, changed to feel quite a bit and given. That this technology derives, a lot of data and that. Has been already integrated, into how we classify diseases. We are in a data science, realm. Right now however. To make all these data science, initiative, safe, and effective I think one of the things that the field should, move to is to. Really check whether this, makes sense from regulatory perspective the. Regulatory sciences will have to improve and adapt and they are, but. Without appropriate scientific input it may go in two directions that, we may or may not want and if you look at some of these aspects, here it's really important, that we provide this input, do. A little bit of advertisement, here tomorrow, and on, Friday we. Host one of these alliance meetings there's one upcoming, in the summer as well where. Representatives. From a lot of different people you see some of the speaker's here and share. Their experience, and their viewpoints on regulatory, sciences. And we, hope that with that we can try, to at, least improve, the scientific, input for clinical, innovation. To. Come to them I have, to thank a lot of people first. And foremost leadership. And my mentors. John. And David, incredibly. Wonderful. People in my team just to, mention a few here and then obviously my funding sources and with that I think you for your attention and I'm, happy, to take some questions and, answer thank, you very much. Are. There some questions yes, see, when you. Mentioned that the. Final. Projects going through the FDA, but isn't CMS, involved, in the reauthorization. Issue. Mm-hmm. Um so. The question was whether the. Pre-authorization. Is, only affecting, FDA, or, whether, CMS, is also involved briefly. I think they call it pre certification. And CMS, and FDA, are. Both. Related, and involved, but the issue is that according, to how I understand, it and I know some of the experts in the room weigh disagree.
But How I understand, the valid act as proposed currently, it. Wouldn't, touch clear. Meaning. It, wouldn't touch the issue, here. Is. That answering a question, well. Not. Exactly because I think the payment. Process, the FDA has always been, touching. Has. Pretty, much exclude, self from the payment process correct, and so. The, given, that free authorization, is primarily a payment, issue, you. Know it'd be interesting to see how that good feedback. From. The FDA into the CMS I I, know. That you know some of these breakthrough. Designations. They work both together and for next-gen sequencing the. National coverage determination. Has that but maybe, I'm not quite, understanding, but um. Don't. Know really how to answer, that I apologize. Other. Questions. Yes. I really. Appreciated the. Showing. All that sort of hidden infrastructure. Underneath it. You, know institution, that wants to innovate, and, I wonder. If you go even a level deeper what. You think has been either. Particularly. Important, in your institution, or particular, challenge, when it comes to even, things. Like higher, getting. POS, out the door or getting compliance. To sign off on putting so putting something up on a serve like I I feel, like those are even. Or, subterranean. Forces. That would prevent innovation. That are really difficult to call attention to what in your experience those. Sorts of things, so. To. Summarize two questions so the question was whether there. Are other elements that go even below, what, I've shown for example ordering, I don't, know hiring etc. I can. Just, say that it is time. Consuming you know and I think there's you know problems, left and right I think, one issue is you know a. Competency. Based compensation. Profile. Meaning. There's a lot of rules about, you, know Human Resources and equity which I think are absolutely appropriate but. On the other hand you know really encouraging. Talent, and retain, manof highly, qualified people, is probably one of our biggest challenges, and. You mentioned some of these aspects. Regarding you know I would call that like in a big realm of administration. Of the lab ordering, purchasing, etc what you outline I think, that is highly. Related, to individuals, and you, know their abilities, to really maneuver things forward and not, giving up and that is something largely, you know it's very hard to incentivize, that and, have. People who are really diligent, about that we have spent. A lot of time tracking our inventory. Lot and you know streamlining. Again. Mundane, tasks, to make that more, easy. To to do the ordering but I think it is very, very. Important, to keep a great team spirit and. We've. Done a lot of you know person. Alice's, and examinations. On how to put teams together you, know some people are very effective, you know they are they just want to get things done other, people are much more paying attention to details and then you can have both in the wrong position and it may never work other.
People, Are just great team players but if you put three team players together they, may have a great conversation but, not get anything done so. It's you know it's like it's a lot of personnel. Management and, especially, when reaching out to other disciplines, it's. Just like really being humble, like being aware. That they don't know what we're doing and it's, like you know talking to you know other outside, institutions. It's like you, take things for granted that, may be really great locally. And then. You know you always focus on the negative things but you know taking. All of that into account I think it's all like, gigantic, team, business. And like. You know having, great. Team players I'm extremely, happy about my team they're like amazing, I don't know I couldn't function without them but, then retaining, them if a company, you know across town. Is offering so much more money is really challenging because. You know there's like life outside of work right that you have to also value, and respect but. That is I think that's probably some of the most challenging aspects. Of you know running something like this but the. Other hand I don't know the the, impact that the team feels when changing, some, patients, lives is, also, tremendously. Valuable we, have a lot of people. For. Example some, of the machine learning people that I mentioned, in the talk who, volunteered, who, made it a requirement in, their you, know much better you, know jobs to really continue working with us a lot of people come to us to say hey you, know I've done sort of you know ABC. At certain companies but I really want to change and add value with my skills I think that's the one big pitch that we have but, it's. Definitely a challenge but you know why not. Other. Questions. So. My, question is about the machine. Learning that you, develop so maybe. You said and I missed it but did, you look at outcomes, associated, with that I mean so you. Know is there a way for you to see you know what reporting, is optimal, in terms of the actual outcome, in terms of you, know disease-free, survival patient. Or overall, survival that kind of thing and, so, the question is about the machine, learning whether, that was you know track to any you, know objective, outcomes of survival, or progression-free, survival and, no we didn't do this so currently. We are I think in the early stages we. Have deployed. This to, just help us with it's a decision support tool we. Haven't really checked we. Left out about 10 to 15% of, all variants without. The machine learning tool to prevent snowballing, from you know kind of the machine learning to call something and then it's kind of, self-perpetuating. So. We left some of it out but um I don't. Think it's, easy for us to track whether that is really. Changing outcomes. The machine learning part, so, we're trying to figure out what is the actual benefit, in. Terms of you know is it more efficient, or is it providing, certain things so one. Element, that I didn't, put in the talk but when in alch rearranged, lung cancer, when. Under, tki certain. Out mutations. Developed as resistance, mutations, it. Was very interesting because in. 2018. Roughly. And early 2019 when, this sort, of emerged as one of those you, know elements, and we you, know the lung cancer center published some of this we, saw that the machine learning picked, this up and, we didn't really know what to do with it like we we started reporting it and initial paper came out but. Then over time as we, kept calling it and you know the model effectively. Learned again under. You know several relearning, transitions. So. That that improved, but obviously, we can't really say I'll link that to outcomes, but. So currently. I think the biggest value added, is that we, use it to not miss anything so that's why we kept the, cutoff basically, as a screening tool so. Some tert promoter mutations. Occurred, and we're then called and then you know kept, being called and it's, really fascinating because most of our variants. Are not in promoter regions, and we covered the third promoter but you know most of it wasn't so I don't, know not directly answering a question so no outcome, data yet but it was definitely helpful to. See that there's a certain diagnostic, value to it at least for now thank. You, thank. You since you began to talk about. The. Challenge, to fiscal sustainability, I'm. Curious how the Center for integrated. Diagnostics. Achieves, fiscal. Sustainability is. It off clinical. Revenues, diagnostic. Revenues, does it require a massive, subsidy.
From Some other entity, you, have a lot you have a big operation. Yeah. So very. Very good question so the question was is how, is the Center for integrated, diagnostics, finance. Simply put and how are we sustained in this so. Our main, focuses. I mean obviously we have a lot of support. Number one it is mainly. A clinical, operation, on a hospital budget, on a regular no, budget. Cycle we, get however, substantial. Support, a from the Cancer Center and B from the Department, of Pathology certain. Initiatives certain. Strategic plans so. To come back to the one figure that I showed with the financial, paradigm we. Try I think as many people, to use as many financial, resources, and sources, that we can so. There is you know an age funding, you know for certain projects, etc but, it, is, simply. Put a gigantic. Hospital, operation, that I'm really thankful for that, has emerged over the years but. I think it's beyond, the time that we can say we're just eating up money, some. Of the processes. That we have and some of the payer initiatives. Are successful I. Wouldn't. Go there for saying that all of this entire team is financially, sustainable but, we're getting there, does. That answer the question. Other. Questions thanks. A lot. You.
2020-03-17