NIH Proof of Concept Network Common Metrics and Outcomes Tracking
with that I will introduce the next speaker uh uh for the NIH Proof of Concept Network common metrics and outcomes tracking that will be given by Alan O'Connor so Alan thank you very much for that Krishan and I hope everyone's doing well um this is usually one of my you know favorite things to do during um our annual get togethers to just have that opportunity to connect with everyone and report out on the success of our network one of the exciting things about this particular year is of course that we have a full and open public meeting and that means that it's a great opportunity to report out on what we're doing across the network in terms of what our outcomes are the processes that we use and to share some of the ideas and insights that we have accumulated over the past seven years that we've been running the NCAI and REACH programs um so for those of you who are not familiar with me my name is Alan O'Connor I am an economist and a program evaluator based at RTI International with a particular focus on evaluating science and innovation programs these are something that has been kind of a hallmark of my career something that I've really focused in on and I joined this network in 2015 um to help with developing strategies for tracking the overall outcomes and performance um for our particular um for this particular network um so what I thought I would do today would be to share some thoughts on um why do we go about doing um you know monitoring evaluation um have a few highlights about our evaluation program overall touch on some of the metrics that we use um for tracking technology progression to market um you know thinking a little bit and sharing some insights about the infrastructure and the overall approach that we use and then summarize with some of the the current status of the portfolio with respect to progress to market overall including such things as follow-on funding startups small business support via STTRs SBIRs and of course market approvals market approvals is being probably one of the most important thing for NIH because the overall mission of the NIH in general is to turn discovery into health um and these programs are great mechanisms for working with academic innovators on how to go about the process of taking their great discovery and their ideas and moving them towards the bedside to assist patients um I will note that today I'm primarily going to be focusing on funded projects and also technologies as outcomes so there's a portfolio of around 386 projects that have been collectively supported by the 11 NCAI and REACH sites we're going to be talking about the progression to market and the outcomes for those funded projects I want to emphasize that a big portion of the um the overall goal of these programs is around culture change it's around stimulating and engaging academics in translational science it's about building capacity it's about connecting institutions more strongly with their regional innovation ecosystems so there really is a broader focus to these programs that's very important to keep in mind I'm only going to be talking today about a small portion of what we're doing to kind of report out and track outcomes and that's mainly those that are associated with uh the funded projects okay um those primarily stem from two of the programs so NCAI and REACH both 2015 and 2019 um also the NIGMS Tech Transfer Hubs are important part of this network but they only have a very small number of projects and they're still being incorporated into our metrics and so I'm not going to present on those today um but will at a later date so just as a quick refresher for those of you who may not have been able to attend yesterday but the NCAI and REACH sites are distributed across the country um going from Maine to California the state the states here in which there is a REACH or NCAI institution are highlighted in blue there are 11 some of the sites are consortia namely MBArC uh the University of Kentucky and University of Louisville KYNETIC program the Long Island Bioscience Hub UC-CAI in California NCAI-CC in Michigan Illinois and Ohio and B-BIC which is largely concentrated in the Boston area but also includes institutions in Maine and Rhode Island other programs are single site so that would be WE-REACH which is uh based out at the University of Washington SPARK-REACH at the University of Colorado and Rutgers HealthAdvance um which is based at uh Rutgers University collectively there are uh 70 more than 70 institutions included in the network um the largest numbers are from MBArC and KYENTIC largely because um uh MBArC has a broad uh collection of institutions across Missouri and Nebraska um but then all uh the KYNETIC program also includes the technical and community college system for Kentucky so that's a quick overview of the network just to kind of remind everyone of some of the details that Matt reviewed yesterday morning so here I just want to kind of touch on and just remind everybody about why it is that we do this and um I'm on the faculty of the International School for Research Impact Assessment and we always tell our um you know our funders and our research performing entities that you know there it's really the four A's okay so um there's analysis which is really where we're thinking about um learning so that we can make evidence-informed strategy program design and policy decisions going forward um there's also an accountability function um and that's particularly important in this program because although the NIH and congress have provided a substantial amount of resources for NCAI and REACH um there's also matching support coming from the institutions that are part of these networks those institutions committed their own resources either from their state governments or from institutional accelerator programs to kind of support and encourage the the overall progress within their own communities so it's really important that we're not only accountable to the uh the federal stakeholders into the NIH but it's also important that we're accountable to the institutions that have allocated very scarce resources to support these projects and to further these programs because they're that invested in the mission so accountability is really something that's very important to us and of course there's also allocation decisions how do we allocate a marginal dollar so if I have um a certain pot of money that I can put into a translational science program what's my programmatic mix what are the types of levers that I want to play with where where do I put that marginal dollar and sorry I'm an economist so I'm going to use terms like that but that's really important for helping to shape strategy and to make effective decisions and lastly um it's really about advocacy which tends to be more about providing data information lessons insights so we can talk about the importance of these programs um so that we can share best practices and lessons learned and then also to um say hey this is what we're learning from this experiment this is why this was valuable and this is what you should be taking away from this and how can we implement that into either our formal culture or organizational uh structures um or um other aspects that kind of characterize our our research ecosystems so what's RTI's approach um so we designed this in close collaboration um I would say it was probably 2015 2016 working with NHLBI in particular as well as representatives from other parts of the NIH to come up with a holistic evaluation plan that would be meaningful useful and that would provide value for all the reasons that I just finished describing so in a nutshell um you know it really involves um kind of engagement with the site teams who are developing and implementing new strategies and approaches for managing translational science and encouraging academic entrepreneurship that also means that they're developing strategies for cultivating an application pipeline likewise we're really interested in the profiles of applicants who are attracted to these programs one of the things that's always been amazing to me is both REACH and NCAI have been able to reach those innovators that really don't have a whole lot of experience in commercialization um and yet they have an idea they're attracted they want to try it especially those earlier career investigators and their postdocs and their associated teams that really get excited about the program so it's great to learn about what their experience is um and it's interesting how you get this diversity of perspectives from those who um are are more the um kind of the usual cast of characters those tenured professors who really know how to to kind of move technology they have a lot of experience in commercialization then you have the other half who have very little experience um and what what's interesting is that you know the outcomes are largely the same um so it's been great over the years to see these innovators grow um and mature as entrepreneurs and some of them are now CEOs we also spent a lot of time working with the with the funded innovators um just kind of engage understanding their program experience to get suggestions um insights how how things can improve what works well what doesn't work well kind of is there a secret sauce with a particular site from their perspective because they are the target customer in many ways um and then what I'm going to focus most my talk on on today which is um tracking projects over time and this really relates to what's the technologies progression what are the commercialization milestones and outcomes um I don't want to move on without saying that a big part of what we're doing is also thinking about skills development and engagement and of course throughout this whole process there has been sustained collaboration between RTI NIH and all the sites so there are papers being published um data is shared freely um across the network um and so that's really been one of the things that has kind of characterized this particular network it's the fact that it's very much been a place where people can collaborate they can share their insights they can share best practices troubleshoot challenges and and have a sustained dialogue where it's focused on particular issues and academic innovation particularly as starts to connect to entrepreneurship so what are some of the the summary metrics categories that we care about right so that's that's the title of my talk common metrics so um here's a quick overview um you know as as Matt was saying yesterday uh at the beginning there were about 400 ideas for different vectors metrics to capture and I remember getting a call one day from um from one of the Deloitte team and they were just like Alan we have this list uh you know it's about 400 things long and you know we're trying to sort out what's meaningful you know what what's the best approach to do here can you can you help and I'm like yeah I want to do this this is exciting stuff so what we did is we started like streamlining everything down to what would be meaningful um so first we start thinking about um you know the innovator profile so what is their commercialization experience do they have they licensed something before have they been a startup have they engaged with industry before um so that's important to know because you have your usual cast of characters like I mentioned before but then you also have also have those newbies right and those are in many ways they're kind of like the targets when it comes to culture change then there's also demographics you need to be thinking about um what is our representation and how does that compare to what you know about the broader innovator community overall um so we think about race we think about ethnicity uh we think about um academic rank um gender and other issues um we also track technology profile um so what's the technology type such as diagnostic assay, say um therapeutic you also track disease areas cancers and cardiovascular so so those are some of the high-level things that are really important to us now when it comes to the funded projects um and we're tracking progression our ultimate goal is market availability right so the mission here is how can we take these discoveries that we've invested so much basic science dollars in and push them out to solutions that will help patients so market availability at the bottom here is my most is the number one thing that we're paying attention to now there are other metrics that we can use as signals along that path right because we know that it can take if you're in the private sector 10 years most translational science programs 17 years so it takes a while to really move a lot of these technologies along to the market and so some of the things that we track on an ongoing basis would be startup formation and growth um it's the intellectual property status it's the licensing status are there SBIRs or STTR applications or awards how much follow-on funding and who's who's providing that follow-on funding is it something that's coming from NIH is it a strategic partner um is there venture round involved uh for particular concern those are things that we really pay attention to um and likewise we pay attention to technology maturity so DOD and BARTA have done everyone a favor they've come up with a basic rubric for biomedical technology around technology readiness levels or TRLs and we can we have adapted them back to the clinical and translational science setting that we can use to essentially um track the maturity of the technology over time um so um one of the other things that I'm a big fan of is AUTM so I'm stoked that they're presenting later on today um because AUTM has worked really hard for the past couple of decades coming up with some common definitions so why reinvent the wheel they have great definitions for everyone to leverage they're well understood they're well characterized um so we use those um and then of course we also track the regulatory pipeline um because that's what allows us to say well where how close are we getting to potentially having a product on the market so how do we collect the data um so we have a tool that we've created in collaboration with the sites over the past few years it's called the UpdateTracker which is really just technology update tracker we were not feeling particularly creative and it just stuck so that's how that came about but it's a web-based infrastructure and we jokingly say it's like turbo tax for technology development projects you there are wizards and recording features there's a business logic all of that is programmed in by um a stellar team that essentially sees the world as an endless series of databases and they know how to think about how to characterize and pose questions in such a way that it's easy for the sites who are our collaborators in this to provide information and of course we can also ingest data from salesforce or other CRM tools and there's a lot of role-based access control features for security um and the way that we go about doing this is is the project managers at each of the sites who are overseeing the technology development uh projects themselves they'll meet with their staff during um update meetings um you know ask them a few questions by using an I-pad or maybe you'll have the screen open or something like that um and they will ask them the questions they will just populate um the data as they move through their milestone meetings sometimes it takes a minute sometimes it leads to a broader conversation about commercialization strategy so it's also an educational tool um and then we go through a process of validating um the data largely using a a whole host of third-party data sources um and generally speaking this has been an approach that's been well received by the team um so we're really happy um that over the past several years that this has made it relatively easy uh for us to um collect um technology outcomes data now if you were to talk to um an individual person on the sites they're saying well we still have to you know collect information from innovators we still need to plan this into our workflow so it's not a lift but it's not painful and we're not necessarily asking for a whole bunch of stuff that no one will ever look at or use which is part of the strategy so basically just to kind of give people like a little show and tell um you know we have you know log a website where it's called the UpdateTracker where everyone can create an account or log in and then once you're there um you know essentially you'll be able to enter details about a project um so there will be a project abstract a lot of metadata information and then a running timeline about what's going on with the project so that the technology managers the Tech Transfer Offices whoever can go into the site later and review what the progress is and have an informed conversation the other thing that's kind of nice about all of this is that once it's in the database there are ways that we can visualize the data so that people can essentially run queries download graphics um prepare for presentations um so that's really important but what's great about it is that it saves the NIH and the sites from a constant stream of one-off emails and conversations asking for updates and it really starts to drive um people a little crazy so so here's the project update wizard that I mentioned and then and then here's a kind of an example of the application pipeline where we have um an application that's um you know what the what the total total demand so to speak for the programs are so that gets visualized we have follow-on funding interactive reports so there's just a variety of tools that we've made available using this infrastructure that that has really kind of made it um easy for us to track so many projects over time um involving both the sites so that they see all the data that we have any data that we find out um using third-party sources we populate in here so everybody can stay on the same page to how the projects are are doing um I want to just kind of offer a quick um you know additional editorial remarks about metrics tracking um you know again I mentioned earlier that um it's really important that um everyone be always mindful about um about whether each metric that's provided contributes to meaningful situational awareness um if you're not going to do anything with it um if a metric is being proposed but you're just like well how am I going to make a decision with that and what is it really going to tell me then it's probably not something that you need to collect I mentioned earlier the the critical importance of harmonizing common clear definitions um something that Kathleen Rousche shared with me um who's the program officer for NCAI she's just like oh Alan make sure you you tell them that it's really important to start early plan and prepare because the last thing that you want to do is to have to go back and collect information when you're five years into a program one of the other things that we've done here is that we've also provided resources and tools um you know we always strive to kind of create and sustain value for the user community um we have a monthly development cycle to make sure that we're meeting everyone's needs because we want to keep our customer base which is the sites happy um so we will you know make tweaks and adjustments to suit their purposes um to help kind of sustain engagement and you know and honestly we've grown and adapted over time this is not like a a one-time engineered solution we've just you know kept growing and morphing to make sure that our infrastructure and our evaluation program are as dynamic as the technologies and the programs that we are tasked with providing monitoring and evaluation for the last thing before I start launching into some kind of overview of our high level metrics is that um I just want to remind everyone that these metrics are just a snapshot in time and they do not reflect important intangibles okay so these would include things such as culture change learning and demonstration effects professional development um whether there are insights from these programs that inform one's tech transfer and innovation management strategies the data also like tend not to really show a good picture about what is the underlying institutional research strengths mission or portfolio composition right so if you're only looking at summary metrics then you're kind of looking at like the the outputs and outcomes and you're forgetting about well what am I working with and what why is what why is it like that um there can also be some regional context but I will tell you regional context is is not as important as people um think um you know the Midwest um the upper Midwest in particular punches way above its weight class when it comes to innovation performance um and it's not necessarily someone would expect when people tend to think about the kind of concentrated ecosystems of Boston or the San Francisco Bay Area or San Diego so um as I touch there's just a couple of things that I want to remind everybody from Matt's talk the other day um so the first thing is that remember when we talk about the NCAI program it's focused on heart lung blood sleep okay REACH 2015 and 2019 pan-NIH mission something else that I didn't put on the slide is that there's also a resource difference between NCAI and REACH so the REACH sites generally have about two million dollars of capital that they're working with per year and they were funded for um about uh the first cohort for about three years with some extensions it ended up going a little bit longer and then for the current cohort of sites I believe it's four or five years um so you have to keep that in mind the NCAI program was funded for a full seven years so there are some programmatic differences and that of course will show up in the results when we start talking about the numbers so um you know I don't want everyone to really focus on this slide too much the main takeaway is that the overall composition of the funded projects between NCAI and REACH when it comes to technology type is more or less comparable what I've noticed is the big difference is around NCAI with its larger resource base was more likely to have additional therapeutic devices so across therapeutic areas again NCAI has a real focus on heart lung blood sleep and then for REACH um you have a strong weighting towards cancer but there's also cardiovascular technologies in that portfolio as well there's also a large number of research tools apps health IT that really don't have a specific disease or organ indication that's something that's been really interesting about the research portfolio in particular because some of our innovators have put out um apps they have put out new health um new APIs and other health technologies that for example first responders can use um so it's a it's a real difference in kind of like the technology composition and that relates also to the underlying research strengths of the institutions where these projects from which these projects emerged so in terms of follow-on funding um right now um across the whole network we are approaching uh 1.9 billion dollars in follow-on funding um the vast majority of this is from the private sector um in 2021 alone um there was 980 million dollars worth of follow-on funding invested in this technology portfolio and it's been increasing over time there were a couple of early investments that were quite substantial um some of the earliest projects um you know basically picked off low hanging fruit and really accelerated technologies forward and there was some kind of large amounts of follow-on funding secured but there's also a lot of technologies that are attracting additional investment so um in across 386 projects in 2021 alone there were 43 investments of less than one million dollars okay um there were um you know 19 investments of more than one million dollars um and there was also new strategic partner funding totaling more than 700 million so it's quite substantial the amount of interest that this portfolio has attracted keep in mind many of our innovators had very little commercialization experience prior to participating in this program so um as would be expected um some of the the earliest projects um funded back in 2014 are those that kind of attracted have had more time to germinate so to speak and a couple of those have really led to some quite significant investments from partners but as you look across the years um there's actually a kind of a nice distribution of follow-on funding events so um for example in 2019 um you know out of that particular funded project year you know 15 of those 31 projects are already getting follow-on investment 2020 projects 17 of them so we're really seeing a lot of interest um from investors in the projects that have been selected for this portfolio here's just a high level summary of our funding by source um most notably some Amgen and some of the large um uh kind of biotech companies have really invested in technologies that were rooted in um this particular portfolio so if I were to kind of censor out some of the like that big money um for a couple of projects um this is what the balance looks like um so you still have something like 300 million dollars in venture capital has been attracted you have a lot of um other follow-on NIH funding for either uh clinical work translational further uh translational work uh contracts etc. and then others parts of the federal government are investing heavily there's about 36 million coming from the SBIR program and about 4 million from the STTR in terms of success rates when it comes to um uh the uh application for SBIR programs and um STTR programs we're seeing a pretty healthy SBIR success rate of around um you know somewhere between 40 and 50 50 percent success rate which is um really high it's the typical would be about half that if that so the SBIR success rate is really great and that's important because we're trying to to kind of connect these technologies and the startups that are formed to the SBIR portfolio um and then uh there's some kind of other uh data here that people can look at later but I'm gonna keep moving for time um in terms of uh cumulative startups and jobs created um we're now at 101 startups overall um 48 projects have either a startup um and SBIR and STTR app so there's really gonna look kind of like that there's a lot of connections between these different programs which is great to see um something else that we observed when we were looking at the latest data is that the the median time from project start to startup formation it's about seven months um so most teams are learning pretty early on um what is the best commercialization mechanism for their particular technology should they try licensing or should they startup startup formation for um you know startup or excuse me for um licenses and options um you know we have some some small businesses are actually coming in and licensing technologies um which is kind of which is really great to see um but we're noticing there's takes a couple years between um when a project has started where um someone is kind of picking up on the um on that and kind of optioning or licensing the technology I think right now we're at 16 licenses and then there are 11 options so I mentioned earlier around um how do we track technology progression and high level um you know BARTA has a structure um where basically it goes all the way from kind of a review of the science scientific base and then it ends with FDA approval um so along this trajectory um in collaboration with the sites and using some tools we have developed a plan uh whereby a lot of the sites are now trained to kind of identify and track and demarcate what the status of the particular technology is and then there are resources and tools available to help them with that process but what I find to be particularly interesting um is that um if you focus kind of on the the right hand side um is that typically the the time progression to market um for a translational science program is is roughly 17 years and this is based on meta-analysis um that was kind of done several years ago um but that's probably that's like a really um like well-known kind of reference point for how long it takes to get some projects from an academic lab onto the marketplace and what we've seen with this program is that in fewer than eight years there are products on the market and so what this particular graph in front of you is showing is that um in general depending on how well how long ago your project was funded so that's the project age along the x axis um you know that we're seeing broad progression so you know if it's blue that means at least you know one TRL advancement for a project from that particular cohort or that particular year of support um and so you see most of the technologies they're going at least you know once and the interesting thing is that the technologies that are you know between four and seven years old the outcomes in terms of progression um are similar kind of implying that like once you get going that you know that really um there's that momentum um can be sustained um and that depending upon what the broader character kind of characteristics or market dynamics are um there can be uh uh you know a substantial or a strong pathway to um the next two stages of technology maturity I think we had one technology that went from TRL 2 so kind of very early proof of concept all the way to market approval so speaking of market approval um here is kind of some updates about where we are so I mentioned at the beginning of my talk um that we really care about market scouts so on the market today um there are 18 tech or excuse me there are uh four technologies that have regulatory clearance for clinical use there are three additional technologies in the market that are not requiring regulatory approval and then there are an additional 18 technologies that are either in clinical trials or research settings right now so that's a phenomenal um success after only eight years I have not seen such a strong rate of progress in other programs a couple of highlights would be some pulmonary stents there's also this great first responder toolkit that received a lot of attention because of its use and application for healthcare workers who are struggling with the the trauma and the challenge of the pandemic um so really there are some of these tools have really had quite an impact on um on the community so um a couple of parting remarks and then I'm going to open things up for questions there are some some things that these teams have done very very well um so one is on-site team composition um you know they've chosen leaders that have a strong track record of academic entrepreneurship they have good visibility and they're supported by really strong teams around them they also have empathetic project managers with industry uh product development expertise and empathetic is really important because remember a lot of our the innovators in this particular community they've gone kindergarten to postdoc to a faculty position they're very good at the science they're very good at science um but they have never many of them have never kind of moved a technology or a concept into the translational pipeline before and so being paired with an industry trained product manager who has really good uh uh communication skills can coach that innovator can can really help kind of add a lot of value and ensure that that kind of technology development project doesn't drift into hypothesis driven research but stays on track towards what we need to do to add value and get your great idea into a product or service to help somebody um something else that um kind of characterize the sites was um close integration with Technology Transfer Offices in some cases the Tech Transfer Offices are part of the leadership teams or in a couple are the leadership teams themselves they also leverage resources from their CTSAs and local innovation ecosystems something else that was particularly effective was um kind of leveraging external advisory boards and proposal selection boards um we saw a couple we had one instance where a board for choosing applications for funding um the composition was just basically a lot of like uh university administrators and senior faculty and deans and people just like oh these are the wrong people to be evaluating technologies for uh for commercial merit and scientific promise and relevance for the market um and so what we saw is that a lot of proposals the selection boards were retooled using life science executives entrepreneurs and people from the broader community um something else that um that kind of characterized the sites was active outreach to the research community um um so the more that the site teams were engaged in interacting with faculty the more likely that they would ask questions about the program and increase the propensity to apply so actual active engagement not just passive engagement or newsletters but actually going to meetings being there championing program and program and sustaining that proved to be particularly significant something else that's just a signal because it's an n of 1 but I do like to just to share and I believe that um Paula Bates from the University of Louisville shared this in a talk a couple of years ago but essentially the the full innovative innovator-facing team from the University of Louisville ExCITE program were women and it just so happened that in the end sixty-three percent of funded PIs or co-PIs were also women and in our interviews with those innovators they said that representation and seeing someone like themselves in a leadership position as an entrepreneur inspired them to take the next step forward with their idea so I think that's probably themes that Monique and team will pick up on when they talk about diversity and inclusion here shortly um something else that was kind of a great practice was you know application processes as an opportunity for learning and feedback so watching innovators struggle filling out um a you know kind of a market plan a business development plan um you know it was also an opportunity to kind of coach and provide feedback um something kind of related to this is that for educational programming um many of the sites were starting to time all their their seminars their boot camps with the application cycles um and that pursuit proved to be particularly useful to help those innovators as they muddled through their application for technology development funding of course once the projects were funded a lot of the innovators said well the most important training was my one-on-one coaching with the site team and my project manager because you know they basically counseled me they were my sounding board and they helped me solve some of these challenges about how to add value to technology when this is the first time I've ever done it um so that was a real common anecdote that we've heard across the board they also were effective at um you know implementing milestone-based project management tools so tranchd funding fast fail strategies the use of target product profiles and other tools um you know things that are tend to be um you know uh fairly new to an academic investigator but it proved to be particularly um helpful in um building out their knowledge about how to move these projects forward so high level takeaways for me um so this has been I think this is the sixth time I've given a talk like this um and I really love it um but the thing that always strikes me is um that really there's a lot of evidence about how these types of programs can transition those basic science discoveries to help patients and as a science policy nerd um you know I really think about the return on basic science investment NIH is a 40 billion dollar a year enterprise a large proportion of that is going to basic science research to help patients and people and discover new knowledge and technologies and it's great to have programs that kind of de-risk the probability that you know a great idea just stays a great idea and doesn't make it into a patient um it's also really exciting to see how this kind of fuels our small business innovation programs um so it provides um some great technology um both to existing small startups uh but then also uh kind of forms new ones um such as what we've seen with the 101 that have started in this these two uh programs in particular I'll mention it again products to market in less than eight years when typical 17 um from a translational setting um and then you know some some of you um are likely um you know uh run a small fund either at a university or um maybe a a state-based innovation fund you know if you're you know Pittsburgh Life Sciences Greenhouse or you know one of the other Ben Franklin's uh for example you're probably saying well you know I really want at least a 10x on our return well this program right now for you know if you were to you apply those typical rates of return um measures that are used by the the the broader venture community either public or private and um you know you're basically looking at 35x for NCAI and REACH um combined um that's not bad um and then I guess the other kind of parting note that I just want to share is that there's a strong base of program management insights emerging from these teams within um within the past year alone I think uh three or four new publications have emerged um from you know Colorado-SPARK from B-BIC in Boston and there's one from uh from the NCAI-CC team we put together an overview so there are resources available particularly in the journal of clinical and translational science so if you're really interested in this stuff about how you can kind of improve program delivery what are the types of metrics you should use what are some of the key insights and perspectives from the leaders and these these really great site teams you know these journals are the place to go to kind of check out what the takeaways are and with that I will take any questions thank you Alan for a great presentation it's always uh you know good to hear you give sort of an overview of the network but you know also talking about why these you know outcomes and metrics and collection and the standardization is so important uh so they were we we have been answering a bunch of questions but uh there's one that you know it might be helpful for you to weigh in on uh someone is asking uh if we have looked at or if you have looked at uh innovative demographics in terms of that follow-on funding gender race and all of those factors like is there do we have any break break break out of that uh if you could comment on that yeah actually that's a that's a really great question um we we have started to investigate that data in general I can tell you that once innovators reach so for the SBIR portfolio okay just as a reference point once you have your phase one SBIR outcomes are the same for phase two transition and thereafter there's no difference by gender or race so that that means that really what we need to be focusing on when it comes to broader incluse inclusion when it comes to kind of these types of commercialization outcomes is getting getting the innovator to that first milestone and in this program uh largely we're seeing strong results um the the hard part is is that we have noticed that there are differences in the technology profiles by gender so um you know we have a lot of the therapeutics um are largely from men but and when we have a strong base of research tools um diagnostics and assays from women that's the only thing that I've observed but we are starting to look across the portfolio it's just hard to draw inferences because it's only 386 projects so just kind of take all this with a grain of salt of salt it's just a signal not nothing you can take to the bank thanks Alan and uh you know I think we are we are trying to look at some of the data on the uh non-NIH side as well but that's a little tricky and uh it's always challenging to sort of figure out uh the the exact attribution and all but we are interested in figuring that out and looking into it uh someone was asking about I think like you were talking about that milestone driven work and fail fast philosophy versus hypothesis driven research um and so just one or two you know there was a question about what does that mean and can you clarify why hypothesis driven research should be avoided so just before I pass it to you I wanted to make a clarification that we are not saying that we should avoid hypothesis driven research and that's what most of what NIH does uh our pure basic research but for the kind of work that we are supporting with the NIH Proof of Concept Network for those product development work uh you know the milestone driven approach is what we are uh advocating for but um that and I'll let you comment on that well I think I think you're I think you're touching on it really well Ashim there's a time and a place for everything and um this particular program is focused on that the additional validation and maturation of a known technology it's filling a gap because a lot of the basic science funding is really looking at hypothesis driven research generation of knowledge right it's not at validating and further developing something that you've already discovered and the focus of this particular program um is to kind of do that validation and the additional technology development work so that the idea moves forward into something that's a practical usable tool therapeutic device etc. so that's the picture here hypothesis driven research that is bread and
butter for NIH this program is distinct from that because it has a very particular lens on trying to essentially move that technology forward that's the that's a that's the difference here thank you uh I think one let's see uh I wanted to acknowledge uh you know Paula and Claire they have put the link for some of the articles so please take a look at those are definitely a great resource uh someone was asking about some of the metrics that we are using and collecting so is it possible for you know people to access those uh do we have I’m trying to think like I mean I think in some of those articles we might have those but we can also think of ways to put it out so because it's you know nothing is proprietary we want everyone to use the same data elements that's right so um in response to this um so I would say that all the papers that I mentioned are you can go to the journal of clinical and translational science um or Paula Bates has dropped in that if you're interested in that news article she can get that to you um there's also paper in nature review drug discovery um so um when it comes to this evaluation um program I’m in the process right now of crafting a full paper that describes our entire evaluation program as a reference tool um and so if you drop me an email I’m more than happy to share with you the evaluation plan that I've already produced and other materials I can email it to you but there's also a paper that's that's currently in develop development thanks Alan uh let's see uh one question about that you know 35 x return on investment uh you know how do we calculate that could you sort of talk about that a little bit and uh how we are measuring it yeah absolutely um so so basically it's a ballpark it's a moving picture at any point in time but it's essentially it is the um kind of the the um non-federal follow-on funding as a numerator with the federal support for these programs as the denominator so the denominator will be the full cost so all like you know I don't, I forget that number off the top of my head but I think that um REACH um is some the current cohort it's like 4 million per site per year or 1 million per site per year for four years um so basically we have that um and then for um for NCAI is a bit larger so what we do is we just kind of lump all that together and that becomes the denominator and the numerator is the non-federal follow-on funding and that is about 35x it changes but that's the ballpark and in terms of uh you know how do we know like if the REACH product was relevant to it again there is a lot of validation we do right so we check with the program project managers and program managers said each of the Hubs and sites we validate with the innovator who was funded as well we want to make sure that we are getting getting the right attribution so we're very careful uh in terms of making sure that you know whatever numbers we're quoting has attribution to the original REACH or NCAI project that's right and we also we have a lot we have a lot of kind of data science team on our side that are essentially running crawls and they're getting all the validation data and really you know the sites um in collaboration with um RTI and NIH I mean we've just done a phenomenal job over the years getting all this stuff down and you know every now and then we'll be something that we'll miss um that they'll remind us or vice versa but largely we feel pretty good about our broader situational awareness great this has been great I think we are a few minutes over but this we're supposed to have a break so uh that's totally fine uh thank you again Alan uh and you know we will be we'll be sharing this recording at some point in the future we're figuring out where and when uh but you'll all have access to it and then yeah you know we'll share all of the the relevant links for all of the papers that were mentioned earlier so you all have access to that as well but yeah as Alan said reach out to us if you have any questions about some of the processes that we are using and like the the metrics we're collecting and um and yeah i would love to uh and Lena just put out a link for uh for one of the uh the RTI's evaluation uh articles so thank you all uh we have a break for the next 10 minutes and we'll have a very exciting discussion on uh equity diversity and inclusion uh with members of the uh NIH Proof of Concept Network uh EDI Action Committee uh and Eric Padmore who is co-leading the um NIH working group on biomedical entrepreneurship workforce development um and Almesha Campbell from Jackson State and Jonathan Fay from University of Michigan so it's going to be a great panel so we'll see you back here in 10 minutes uh you know go around and stretch your legs and we'll see you back here in a little bit
2023-02-12 23:45
