MDR certification for Artificial Intelligence Software with Leon Doorn
welcome to the medical device made easy podcast here is Monir el azzouzi from easymedicaldevice.com and today we will go a bit in the science fiction because we'll talk about artificial intelligence with our guests today so we'll talk about software that are helping some doctor or some physicist to detect some disease and here with me we have Leon doorn from aidence Leon is the regulatory compliance manager in aidence and he's helping us to understand how idents got CE certified under eumdr so Leon welcome to the medical device made easy podcast thank you nice to be here and very interesting to be in your podcast thank you so Leon as i said so mainly we'll try to understand more about how an artificial intelligence software can get eumdr certified but before that can you make an introduction of yourself so who you are and what you are doing and also maybe a bit of aidence software what it is exactly yeah of course so i started a long time ago to study foreigners i finished my nursing studies then i quickly decided i did not want to work as a nurse because to be honest in practice i wasn't able to respond quick enough to large emergencies that came into the hospital and i decided that actually research and science was much more of my interest so i decided to continue my studies i studied health sciences at the university in norway in amsterdam and then after i finalized my studies i also had no idea at that stage what i wanted to do i got approached by a company whether i was interested to come and work for them as their office manager and i decided to go for it which was for qsurf at that time which is a small quality and regulatory consulting firm i worked with them for seven years doing just consulting and i then i decided i actually wanted to move over into the industry decided to go and work for a larger medical device manufacturer worked there for a total of four years and then decided that such a large company was also not the kind of company i wanted to work for and i really wanted to switch to a much smaller company with innovative products and joined aidance at that time aidens was still with a team of 10 people so relatively small and now we're we've grown to about 40 people so we're growing yeah so mainly so you are located in amsterdam aidence also located in Amsterdam that's correct yeah okay we're actually we're located right across from the amstel hotel here in amsterdam many people will know it it's a beautiful location and you're going to the office when it's possible unfortunately right now it's a little bit limited but i hope that will be possible soon again so in terms of uh aidence so mainly as we've said it's a software company that is mainly doing a software for medical devices so we said also it's using artificial intelligence so I just want to ask maybe first what is I mean for people that don't really understand that so what is exactly artificial intelligence for this kind of software and is this something that like where kids maybe we thought about those robots that will replace humans so is it exactly the same thing that is happening now with those type of software it's a good question so maybe i should start a little bit on what artificial intelligence actually is because there's i think in the market there's a lot of misperceptions on how ai is and can be used first we need to go to the concept of artificial intelligence if you look at traditional software it's all programmed by people so that means that people are considering they have a specific task they want to solve it software and they start coding the software to resolve that specific task that is not always as effective as it can be so there for some applications that's perfectly fine and it works really well for other applications it's a bit more difficult especially when there's when the answer is not going to be black and white but it's going to be somewhere in the middle and there needs to be a decision made so what you do is you take a lot of data that you gain for example images from the medical field and you feed that all those images all the data into the algorithm and algorithm will decide on the basis of or will actually train on the basis of that data and will start understanding where the border between certain issues is so if you look at the lungs for example if you see an abnormality within lung it's some tissue might be an abnormality some tissue might not be an abnormality but whereas exactly that border that's very difficult it's even difficult for us as humans to clearly see that so what you do is you take a lot of data where humans radiologists experts in the field have decided this is an abnormality and this is non-abdominality you feed the data into the algorithm and the algorithm actually starts to learn and understand what is an abnormality and what is not an abnormality so if you compare that to traditional data it's not hard coded it's trained on the basis of large sets of data so that's mainly the big difference so it's like having a child and and teaching him what is how looks a car how looks a play you know how it looks things so you have to show him the image and him many times and then to say and then he has to say this is a car this is not a car or whatever so is this something that if i can say stops learning at one time or it's always learning even after it's released yeah it's a great question so at some point this is a decision that you can make as a manufacturer so some systems will continue to learn in practice if you will set it up that way however if we look at the regulatory environment that we have nowaday that is actually a bit difficult because the regulations do not stipulate how such continuous system would look like so and it can introduce difficulties if you look at the market right now where we're at with artificial intelligence specifically in the healthcare sector it's not the case so how these algorithms work is you train them then you validate them against a completely independent data set and you see what the performance of your algorithm is then you sort of freeze the algorithm and that's actually what you bring to market so in practice what you can do is you can gain additional information you can gain additional data you can retrain the algorithm in the background it and then you test it again against your validation data and you see whether the performance improves or it does not improve and if it does improve you can freeze it again and then release that in a similar fashion as you release the first advice as a second updated device onto the market so in practice that means that an algorithm can continue to develop and it can become better but it's not that it's learning from the data that's being fed into it from the clinical practice it can work that way but today it does not okay so i think it's interesting because maybe there is some people that think that even a software an artificial intelligence software that was released on the market if i can say it's continuing to learn so to change its algorithm and to change its decision but it's not the case actually so in reality the artificial intelligence part is just on the development of the software but when it's released there is no more artificial intelligence working is correct so the algorithm that's included within the device is still based on artificial intelligence so if you would compare a algorithm that's released that's developed by ai or whether it's developed by standard coding is that with standard coding you know exactly what the algorithm is doing and why it's making certain decisions because you have checks and balances that are all computed by humans whereas in artificial intelligence algorithms you do not know so you've trained it to make certain decisions but we do not know why it's exactly making those decisions the only thing you know is that it is basic institutions on the input that you've provided into it so if you have all these images analyzed by intelligent and id expert medical physicians in the field you know that the algorithm will learn from the best and then you know that the algorithm will probably mimic the performance of those best radiologists so this is really a black box so it's something that you put something in at the input and you get something at the output and here what you are measuring is really that this output is always correct and performant and safe for the patient then exactly so but the black box even though the inside of the algorithm might be a black box you can test it right you can see against an independent data set how well it performs you can see how it performs if you compare it to human performance so you are able to test the performance of that algorithm which is super important because it will show whether it's performing just as good as radiologists in the field whether it's not as good as radiologists in the field or might even be better than the radiologists that on our day in the field or other kind of physicians because i'm here talking and referring to image recognition but there might be other types of artificial intelligence as well of course yeah and this technology is starting to grow we have a lot of companies that are starting to use that but in the previous ages if i can say when the regulation was in place this was not really mature enough so what is now the difference in terms of for example of classification for this type of software when it was mdd and when it is with the new mdr actually so maybe we should start by saying that under the mdd software in general can be classified from class I to class III and this is how depending on how you interpret the rules and the type of software that you are producing if you look at that at a high solution specifically under the mdd there is a number of companies there are a number of companies that have classified these kind of products as two to a products there's also companies that have classified it as class one products the even though i think we can all understand there's quite a bit of risk involved with devices that will support clinical decision making if you look at why this is done and if you go back into the regulation is that when the first classification rule was set up under the mdd it didn't take software specifically into consideration that's i think one aspect and the other aspect is that the rule as set out in the mdd leaves a little bit room of interpretation and it goes into specific need out of the top of my head for making a diagnosis which if not the case then it the rule wouldn't apply and you would just fall under other and you would go back to a class I product so one example that's given by the by the european commission i think in the borderline manual is that a device to as an example to analyze lesions of the skin to determine whether that could be actually a melanoma yes or no yeah would be classified as a class I product which is quite tricky because this can be used by a lay person in their own environment that we use in application that would check whether skin abnormality would be cancerous or not and if they receive the feedback that is not cancerous they probably will not go to a doctor because they assume that it's good but then what if the algorithm makes a mistake and it turns out that isn't actually a proper decision the consequence could be quite severe that someone goes thinks that is not an issue to go for the doctor with and actually the cancer would develop and will become more malignant and maybe less treatable as well so that's something where i think and i personally believe that the risk of these kind of devices warned that such devices should not be a class I device and we can say that the the idea if to just to remind to people so class I devices are self-certified so there is no notified bodies looking at that nobody is really looking at that only when there is maybe a major issue or a big complaint on it so there is a the member state that can go and look at the company that made this device but it's self-satisfied so it means that there is no real proof that the all the tests were done correctly to check that this software is really detecting a melanoma or not and this is the issue here in this kind of the device that we call low-risk device because they are class I yeah exactly and it might not necessarily be a problem right because for a class one device it's not necessarily true that you have to do less they do not have to validate your device performance the only differences is the way that it's being controlled so that means that if you have a class one device you can self-certify which means there's no external body that's going to look at your complete technical documentation unless there's an explicit need for it whereas with a class or a device or higher you will have a notified property that checks all the documentation so if you look at actually at the mdr and what's changing is that i think the european commission has recognized that there can actually be a potential risk with these kind of devices and that the software classification wasn't properly covered under mdd and that's why they introduced new rule 11 which is specific to medical device software and that rule has changed the layout of the classification and in my belief it says that all diagnostic and therapeutic software that provides information that can be used by a physician to come to a medical diagnosis or which could lead to any either too serious well either to minor injury or to serious injury or even death should always be classified as class 2a or higher which means that this whole class one group of devices that now can exist under the mdd will stop existing and will all need to go through a notification so I think it's something that is important just to recognize that the mdr is taking into account all those new technologies and then really look at all the possibilities that they can do now that they couldn't really do before when the legislation was in place and that they recognized that yeah there is higher risk on this kind of devices and there is as you mentioned specific rule now under mdr just for software just to review the software devices just a precision so as we said about your device at aidence so it's using artificial intelligence but do you need still the help of a physicist of or it's something that you just put in a kind of a scanner you get the information and you just have the letter sent to the client to say you have this disease so is there kind of another automatic system or that or you still need so coming back to your question because I think your question was first can ai replace the doctors in the fields right yeah because that's the that's often the perception that people have that ai will start will do in the longer term at this moment it's absolutely not the case that ai will replace a doctor for especially if you look at our product so what we do is the product is being trained the product has a certain performance level to detect abnormalities within the lung so we help a physician a radiologist to find those abnormalities so if you look at a radiologist maybe i should go at it from that perspective they make very long shifts they have to review for one single patient at least 300 images within the lungs that are the result of ct scan and they have to go through all these 300 patients to find the abnormalities which is quite quite a bit of work and you can imagine that it's easy to miss a abnormality especially when they're smaller right an abnormality is already something that can be of five millimeters in size which is really small and hard to see for a radiologist so i think what the software can really do and can help with is to improve the performance of doctors in the field and to help physicians become better at what they're doing nowadays so ai can be seen as something that's helping healthcare getting better where there's a clear benefit to the patient but also to the radiologists himself you would probably if you would ask anyone do you feel comfortable with just one radiologist looking at your data or would you feel more comfortable if the radiologist is looking at your data together with the support of an additional program that also will help to identify abnormalities then i guess that all everyone all patients will say yes please have that second barrel of ice which are being offered by the software so it and this is also the experience i think of the radiologists themselves they really appreciate the help of a second pair of eyes such as that offered by our program so on the short term i do not believe that ai is going to replace a doctor but i do believe that ai will help the doctor to become better at their job and it will also help the patient eventually get better outcomes so in the case for example somebody has the id to say my device is 100% accurate so then I don't need a physicist at all so will this increase the classification of this product so it's a good question because right there's typically a trade-off between sensitivity and specificity which means that if your product is 100% sensitive that means that you will find all abnormalities but it can also indicate a normal tissue as being an abnormality or the other way around which can be problematic so it's still good to have second pair of eyes to look at the data if the algorithm would surpass human performance so it wouldn't generate false positives for example and it would its sensitivity would be better than human performance it still doesn't mean that there is no need for an additional radiologist in place for example our device detects abnormalities on the lungs which are called pulmonary nodules but on the lungs themselves there's also other types of abnormalities that our device does not detect because it's not trained to find those so we can support in taking over certain small parts of their entire process but the radiologists will still view the entire lungs to find if there's other abnormalities that are so far does not detect because it's not trained to find those abnormalities so in terms of in terms of the product itself as we've said so the class of the product is increased now due to mdr in comparison to mdd it's not something that is automatic it means that you still need some people to use that so the physicist can still decide to ignore the advice if i can say from this software so it's not like there it's mandatory they have to go through that root they are still there to make decisions and say yes it's it is a nodule or it is not a module and then we can move to the next patient directly so in terms of eu mdr as you said the classification is higher are also the expectation from the notified body higher in terms of artificial intelligence software so is it more difficult for you because you wear mdd certified under ce marked mdd so now you are certified under mdr so was it more difficult for you to be mdr certified i actually think if you ask me honestly i think that the classification of our device and the risk profile associated with the use of our device is already quite under scrutiny with the note 5 body nowadays under the mdd because they feel it's a new kind of device they really want to see the evidence they clean clinical evidence how do you validate this and i think if if i compare personally the expectations from the notified body between class 2a and class 2b on that mdr so previously i'm a 2a under the mdd there's not that big of a difference in how they review the documentation what i do believe is that there's a much higher emphasis on post-market data so getting your post-market clinical data how are you going to follow up how are you going to make sure that when the device is launched onto the market that it will actually remain safe so how do you where do you gather that data and this is something that we at aidence have placed also quite a bit of focus on to make sure that we get this post-marked clinical follow-up data to make sure that we can demonstrate that our device trained on a large set of data and validated on another set of data also actually performs in clinical practice in a way that we promise that it will because a validation data set you try to create one that's as representative as possible for the clinical practice but you only know once you put your device out in clinical practice whether you've achieved that to do that job properly so i think for ai devices specifically but probably also for other software devices the emphasis on voice marketing follow-up is really important and also post-market data so you need if I can say some data to prove that your device is doing what you are claiming that it is doing so for that are you where are you using some kind of clinical investigation process or where are you more getting some data from centers and they provide that to you so how what was the methodology here to prove that your claims were correct i think what's fun with that with our type of device right is that it's quite new enough the people in radiology are quite happy to work with these innovations and they're quite happy to investigate how it actually helps them in their work so some of the hospitals that we work with they're really happy to investigate what the benefits are of our device in their practice and they are happy to publish about it as well so we have a number of hospitals that offered to start doing some studies with use of our device and we greatly welcome that because it fits right in with our post market clinical follow-up study so we have a which is i think very cool for us is that one of the hospitals actually done an study right now and they shown that the performance that we claim for our device is exactly matching the performance that they found in their clinical practice for our device which is just confirming that our device does what we say in our documentation so the so you have all those data you are claiming all those things so you have all those reports i think at one point as you mentioned the notified bodies are reviewing that notified bodies were not as with mdd there were more class one devices under other software so there were I think not a lot of experience for all notified bodies about a software study so was there any kind of need to have a specialist of algorithm of ai under the notified body or was it a normal person a normal auditor that was doing also orthopedics or whatever no so at our notifier body they actually assigned the person well our main reviewer was responsible for the review of all the regulatory documentation but he wasn't responsible for the review he is ultimately responsible for the review of the clinical documentation but our notified body does use external experts in the field to review our clinical documentation so they and they actually use i think a radiologist from the field to review all our clinical documentation which makes sense because a radiologist is also someone who's very interested in the kind of studies that are published around these devices so they will know what is required to get these devices out on the market okay so yes it's really so you have here like a team of specialists that are reviewing everything for clinical software all the data here so your notified body was your notified body for mdd and it is your notified body for mdr yes so how long did it take for you to make this transition from mdd to mdr what was really the struggle if i can say the most important struggle on doing that process very good question so when we started i think it's about more than a year ago where we started the investigations during a gap assessment and finding out that actually the classification rule had changed so when we looked at the classification rule our first interpretation was that our device will be indeed a class it could be varying between class 2a class 2b and class 3. we had long discussions internally with the team on which definitions we should follow if you look strictly at the definition from the mdr you could end up in clustering if you look at the definition from the guidance document that was issued by the ndcg you could end up with class IIa or IIb and then there's additionally a reference made to the software as a medical device document from imdrf which we also went through and then we ended up coming to a classification of 2b the very first thing that we did when we had that discussion when we found out that there is interpretation to be made to the classification rules that we plan the meeting with are not fair body and we went there and we presented our case to them and we asked them for their thoughts as well so what we did is we even long time before we started submitting we already send in our classification rationale and came to agreement with another body that that advice was going to be class 2b and that's also the moment where we started working on changing our quality system we had a stage one audit in january this year and then we submitted the whole product documentation in the running of april then obviously as you know corona also started to become a real problem here in europe and that's where our device was submitted to the external clinician and that cost us quite a bit of delay in the review process because the radiologists were extremely busy with daily practice and getting corona on the control and helping out with the corona situation so that caused some delay in the review process and that made that the review at the notify probably took another six months for our product file but eventually end of september we now actually in october we received our mdr certificate for the product so the one-year delay uh due to corona was also helping a bit here to have it everything on time then exactly so it for us it took from i think from the moment from our first mdr audit which was in January when we updated the quality system to date was about nine months but if you also calculate in all the time that we spend on updating the documentation and getting the quality system in place and starting this discussion on the classification rationale it's easily a year or more just for one product which is quite a long timeline so if you had now to remember all your story about this certification so what was the most challenging or the struggle the biggest struggle that you had on this journey you know the biggest struggle was undoubtedly the review period where we expected that we would have response much faster than what we did but we understand why it took longer because of corona but the struggle is that when we decided to move over to the mdr we submitted our product to the notified body for review and during the entire review we found out that a product that we had on the market also needed to be updated but which was the product we submitted and then we got quite stuck because there's a product out on the market that we can't update because we don't have the new certificate yet and then is where we decided that we're going to continue making updates to the product it was out on the market under the old regulation but obviously you can't introduce major design changes anymore because they would not be reviewed by the notified body because you you're also working on your mdr application so yeah we had to postpone actually making some changes to the product which was a bit frustrating but finally and now that we do have our mdr certificate we still can't go to the market because of this discrepancy between the device that we have on the market which is more up to date and the device that we submitted back then to to our notified buddy yeah so it's it's a bit of silly because the device that we submitted is no longer state of the art that's a product that we have on the market still on our mdd so now we need to make changes to the mdr product to become state of the art again so i can imagine that a lot of software companies will go through that route that will submit the software but in the meantime, they will have to go through a new version because of maybe some changes that are happening and that when they would be mdr they would be stuck also and they will have to make an update of all the their things so i think it's a good thing it's a good message if i can say just to inform those companies that this can happen and this is something that most likely will happen just because of the time the duration for the review of all the technical documentation from yeah and it will happen for sure no it's a great story so really thank you for that so aidence as we said so is providing this type of software for hospitals so who i mean who here is will be really interested to use this type of software so our software right it detects pulmonary nodules on long ct scans so the main person who is reviewing lung ct scans is the radiologist so the benefit is directly to the radiologist and eventually also obviously the patient but our users are really the radiologists in the field so when we are saying radiologists so mainly it's all the people that i mean it's really used on a real patient so is it a real-life i can say a live experience that you will have for the radiology so it's something like they have first to take the scan then they will have to go to another computer and to check or it's directly coming from on real-life or for them so that's very good question if you look at our software it's a direct integration into their existing workflow and it offers as little interaction as they well say differently so if you're a radiologist you really don't want your whole workflow to get disturbed you want the actions you need to do as a radiologist to get the outcomes to be as minimal as possible so it doesn't so that it actually provides time efficiency right you don't want to go through another long program where you have to complete a lot of things so what we do is we integrate directly into the workflow we obtain the scan is automatically being processed the results are immediately being sent back to the back system of the hospital so that means that when the radiologist goes into the system you will see the data that he usually reviews for the patient but then additionally immediately the results that we've generated so that all and it just goes to his normal results as he always does but then can decide to also look at our results at the same time within the same system without having to go through other separate systems okay so i think it's really something that is important also for as you said to not disturb the workflow and that exactly are getting really the information directly and can move forward as you as you mentioned they are looking at a lot of patients so if they are reducing their productivity just because of that it can it can be maybe damaging all the workflow so great idea here okay so leon really thank you for all the information i think i will i hope it will help some other medical device software company to understand the process get eu mdr certified and again congratulations for being that because i don't think there is a lot of companies that are eu mdr certified now so it's really great to have people that are following this process and that are completing that okay so now mainly where people can follow up with you so is there some kind of are you on social media or it's more like website or how is it working well everyone can is obviously free to send me a message to linkedin or just go to our website we display or our full team of data scientists our software engineers and our medical and regulatory persons as well in the company on our website with images they everyone is free to click on my image and send me an email if they want no great looking forward to get any questions yeah i will anyway yeah i will put your details on the show notes so people can go directly on instruments and contact you yeah if they have some questions maybe for artificial intelligence or maybe how to pass a certain audit so it could be great so yeah i hope this will be really helping a lot of companies on doing that okay so leon so really thank you for your help thank you for all the information for all the people that are listening to this podcast so please don't hesitate to provide any message if you have any questions i will forward that to Leon so if there isn't anything that you can help you to answer and if you are listening to this podcast on your car or during your workout so don't hesitate to provide a review on your podcast provider and if you are looking at that on youtube so don't use it to put a like and also to provide some comments i will be really happy to to answer to them okay Leon so really happy thank you for your help thank you for all the information and I wish you a nice day okay thanks a lot, i wish you a nice day too thank you bye bye you
2021-01-25 22:37
