Good afternoon and welcome to the IARPA AGILE virtual Proposers' Day my name is Bill Harrod I am the Program Manager for the AGILE program we are here to present a very exciting program it is called the Advanced Graphical Intelligence Logical Computing Environment program which we refer to as AGILE yes the word computing is not represented in the acronym for the program title that's because the program is focused on developing new computer architectures for data intensive applications not compute intensive applications uh besides the 'C' would mess up my acronym thank you for your interest in learning more about the IARPA and the IARPA AGILE program and doing business with IARPA a few notes before we start the presentations during the day please submit your questions and comments you'll notice a Q&A button on the bottom right hand corner of the WebEx screen use your this button to enter questions and comments other participants will not view your questions or comments we want to receive your questions and comments the topics could include IARPA the technical presentation doing business with IARPA or the technical section from the BAA later in the day we will select some questions and provide answers the WebEx audio and video for all participants will be disabled a recording of the entire day will be posted on the AGILE BAA website this presentation provides information concerning IARPA and the AGILE program the presentation will assist you with your plans for a proposal submission to the eventual AGILE BAA the Proposers' Day does not constitute a solicitation for proposals white papers or abstracts the BAA will be improved based upon the feedback that we receive during and after the proposal state so I encourage you to provide us with your questions and comments your questions and comments will influence the development of the BAA The goals of this meeting are to provide information concerning IARPA and the AGILE program hopefully I will also hopefully it will also facilitate teaming the successful execution of the AGILE program will require teams that have a diverse set of RD experience and talent the teaming form posted on sam.gov will enable proposers to find other groups that have capabilities that are required to establish a strong team during the meeting there will be two breaks the first will last 15 minutes and the second will last 30 minutes during these breaks I encourage the participants to submit their questions and comments we must receive your input before 2:15pm we will provide answers to selected questions starting at 2:30. We will read all questions and comments they will have an impact on the eventual BAA this is the agenda for the Proposers' Day meeting our next presentation will be by Pedro who is the Director of the Office of Collections at IARPA he will provide an overview of IARPA after Pedro's talk I will present an overview of the adjust program this will be followed by a 15-minute break during this break you should formulate and submit your questions and comments at three at 1:35 Pedro who is going to pretend he's a contract officer will talk about doing business with IARPA after Pedro's talk we'll have a 30 minute break during this time you should submit your final questions and comments please finish before 2:15 pm at 2:30 I will provide answers to selected questions thank you for listening and now we will start with Pedro's talk so Bill thank you so much for that good afternoon to everybody I am Pedro Espina I am the Director of the Office of Collections at IARPA Bill will you please be so kind to continue to move the slides forward for me absolutely thank you okay so let's go to the next slide So as many of you might be aware IARPA is part of the United States Intelligence Community the United States Intelligence Community is formed by 16 agencies and at the head of it is the Office of the Director of National intelligence IARPA is part of the Office of the Director of National Intelligence our function is that to serve as technical uh know-how to the entire Intelligence Community till the next slide please So IARPA operates this way we envision and lead high-risk high-payoff research that delivers innovative technology for future overwhelming intelligence advantage our problems typically are very complex and their nature is typically multi-disciplinary we emphasize technical access and technical truth okay so the truth guides us in everything that we do at IARPA next slide please We try to use a full open competition in our to solve our problems to the greatest possible extent why do I say that because sometimes some of our problems that we tackle are classified and when they are classified we can only use full open competition among people that have their appropriate clearances to help us with those problems in addition to that we have a category of world-class program managers who typically rotate in and out of IARPA into academia industry or government every three to five years nobody from the Director down in IARPA is a permanent employee When we execute a research program at IARPA we have very clear goals that can be measured uh and we have those goals are very ambitious but they are credible we don't try we're not trying to do we're not dealing in science fiction we're trying to deal with things that are within the realm of the possible we employ independent and readers testing and evaluation we form government teams that help us come up with the testing and evaluation of the performance of those people participating in any one of our programs we involve the intelligence community throughout the process our partners from IC agencies come in at the beginning to tell us what their most challenging problems are they observe how we progress throughout the entire process of developing the program and then they accept our research as input into their operations okay our programs like I said before last typically between three and five years we have some little programs that we call seedlings that typically last between nine and twelve months but the main programs that ones like AGILE today will last between three and five years we encourage our performers to publish in peer review papers why because that meets the criteria of good scientific work okay in some instances we cannot do that because some instances some of the work that is being produced is classified and in those cases we still try to publish it in peer review papers that are classified and we try to transition every capability that we develop into intelligence community partners in other instances sometimes we transition our capabilities to people that are in other government agencies still click the next slide so the IARPA research is very very broad okay it includes programs in mathematics programs in physics chemistry biology neuroscience linguistics political science cognitive philosophy and much more okay about since the since IARPA was stood up about 70 percent of all the work that we have done has actually transitioned into other government agencies the that work has produced in excess of 3,000 peer-reviewed journal articles and some of the people participating in IARPA programs have actually been awarded with very high accolades among them the Nobel prize in physics for quantum computing the McArthur fellowship Nobel prize and others we are connected inside government all the way from the White House to the bench scientists in a national laboratory at the level of the White House we participate in the national science and technology council activities for example we have been participating in the White House Brain Initiative in the national strategic computer initiative in the select committee artificial intelligence on the subcommittee on quantum information science and the subcommittee on economic and security implications for quantum science so we're plugged everywhere where science is needed by the government of the United States next slide please So how do you work with us so this afternoon our contracting officer had an unfortunate situation today so I will have to be the person talking to you about how to work with us from the contracting officer perspective but for now let me tell you this the principal vehicle for you to contact us is our website IARPA.com you have a phone number in this slide that you can call us and your route your phone will be your phone call will be routed to the appropriate person within the structure we also have a mail dropbox called info@iarpa.gov where you can send your questions
and somebody will typically answer to you between 24 and 48 hours later um we have a number of ways to engage with us um RFI uh actually if you go into uh beta.gov you will see that presently we have some RFIs out and those RFI typically lead to workshops like I said before we have these little programs that we call seedlings that last between 9 and 12 months and typically we use those to turn uh doubt into disbelief as we are actually handling a new concept we sometimes put prices prize challenges out on the street to see what comes back and we built very large research programs like the one we're talking today agile okay and with that I think that next bill will be actually talking about the bones of AGILE as a technical program okay and I will be back with all of you around 1:35 so you can actually see how ignorant I am about contracting law okay thank you so much for your time if you have any problems please type into the Q&A section telling us that you're having problems that you're gonna hear or see us or whatever and we will try to address your technical needs okay Bill back to you Hi thank you Pedro again this is uh Bill Harrod I am the Program Manager for the AGILE program I'm here today to present the details concerning the program we anticipate the BAA will be released in early January The Program's purpose is to develop new computer architectures that are efficient and scalable when converting massive and diverse data sets into actionable information in a reasonable time frame computers that are available today and within the next few years are truly awe-inspiring however today's computer systems are not designed to provide optimal performance for future data analytics problems these computers will limit the size and complexity of the data analytics problems that can be tackled and make poor use of the resources we need to have a significant departure from the previous 60 plus years of computer development to resolve these problems we need a system level design approach driven by real applications we cannot use a band-aid approach to improve the components independently we need to start at the system level this is my agenda for today uh topics that I will be covering uh we'll start with what is AGILE and why are we doing this program we'll discuss technical challenges and there are probably many more than what I have presented or will present we'll also discuss a few research strategies to resolve the technical approaches we'll talk about the design process and the expectations we'll talk about how success is measured and what is the schedule But first let's take a look at what is AGILE really about analyzing massive data analytics problems is an important but extremely challenging effort the digitization of society science and security has resulted in massive data sets that are complex and rapidly changing there are extremely large sources of data that include social networks public data sources sensor data and many more sources of data the data sets can include billions of elements the data could be streaming to a computer or found in data lakes or warehouses the various relationships and events found within the datasets could be constantly changing streaming data will require new algorithms or implementations of existing algorithms the items outlined in red in this diagram show where the AGILE program will focus the object is to generate actionable knowledge in on 10x more data than today's capabilities at a rate of 10 to 100 times faster than today's computers IARPA is seeking new revolutionary computer architectures that are solving the emerging data analytics problem AGILE is envisioned to be a three-year program beginning of approximately June 2021 and extending through June 2024 performers will be provided with four workflows and data sets there are target metrics for each workflow and benchmark codes performers will develop RTL designs for their proposed architecture including a runtime system designs will be evaluated using an AGILE enhanced modeling simulation tool set called assist that is based on the Sandia National Laboratory software the structural simulation toolkit or SST SST is an open source toolkit there are numerous tutorials and documentation available on the internet The data of interest can't be transformed into actionable knowledge in a reasonable time frame unfortunately often events and relationships are discovered after they have occurred the data sets are increasingly sparse random and heterogeneous the analytics are characterized by having minimal data locality poor data reuse fine grain data movement and data driven parallelism the fundamental problem is that today's computers solve yesterday's problems they are not efficient or scalable for emerging data intensive problems we have a data movement and storage problem not a compute problem we need a new system level intelligent mechanisms for moving accessing and storing large random time-varying data streams and structures designs need to be driven by end-to-end workflows not simple kernels Let's talk about how data is transformed in knowledge often data is represented by a graph a vertex in the graph represents an entity and the edges represent relationships between entities these graphs tend to be very large and have very few relationships between the entities or vertices often the graph is represented by a matrix thus the resulting matrix tends to be very large and sparse algorithms come from a variety of classes ranging from graph analytics to machine learning to linear algebra for streaming problems the data is constantly changing and randomly distributed across the system we are drowning in data and not able to analyze it before its full values can be exceeded data types vary from measurements to documents many algorithms incorporate reasoning processes where there are issues concerning the heterogeneous integration of graphs and databases thus the vertices and edges have a contextual locality property as opposed to the ordinary spatial or temporal locality the surrounding data provides meaning or context will be found within a diverse set of data sets to fully understand the context requires accessing from a variety of sources this is the fundamental problem of the AGILE data that causes the greatest computational challenges The AGILE challenge problem is dealing with streaming data efficient and timely execution of the analytics will allow us to transition from a forensic analysis environment to a predictive environment having the capability to analyze why an event occurred is extremely important but what is what we really want is the ability to predict that event could occur streaming data analytics provides a continuous flow of information from updates on the data to um uh queries and analysis of streaming graphs helps us understand extreme scale and dynamic real-life interactions and helps predict future events this has motivated the development of streaming algorithms to incrementally update the graph analysis moving away from performing nightly updates of graph properties to providing these updates in minutes or seconds the bottom line is that systems need to be designed for rapidly changing sparse data and processing flow Now we'll examine some of the challenges associated with the AGILE computational problems today the results are required in near real time up to hours identifying event after it occurred doesn't have the same impact as knowing that an event might occur predicting the future streaming data causes unpredictable changes to store data often the computations involve extremely fine-grained data movement where the data is distributed across the computer data computation tasks are typically determined by the data and the streaming queries but these tasks have extremely poor locality and data reviews many graph algorithms can be recast as sparse linear algebra operations recently researchers have been furthering this approach to a graph analytics for further background information I encouraged you to examine the book graph analytics in the language of linear algebra written by Jeremy Kepner and John Gilbert Next we'll examine a few examples that demonstrate the low efficiency and scalability properties of several graph analytics computations the sparse matrix times a vector called SpMV is a standard algorithm that can be used in many graph analytics implementations Brian Page from University of Notre Dame optimized a distributed version of SpMV and used the software to compute the operation for an extensive collection of sparse matrices this graph shown on this slide shows the performance of the results for the densest matrix and the sparse matrix as you can see from this chart the distributed version of SpMV doesn't scale in fact for the Sparse matrix the computation probably should have been simply computed on a single CPU for the AGILE data of interest where the data is randomly distributed across the system memory space the performance results will be much worse this chart shows the performance of the industry standard 8 HPCG benchmark these results were reported recently in November of 2020. You'll find much more information about this benchmark on the HPCG benchmark suite the benchmark uses an internet algorithm for solving large sparse linear systems the main components of the algorithm are SpMV that's the first matrix times a vector global sum and synchronization the fastest system as determined by the benchmark is also the world's newest and fastest computer computer however the HPCG benchmark for this system is only 3.6 percent efficient yet the it is 82 efficient for the HPL benchmark which solves a dense linear system and provides you with the results for the top 500 benchmark Finally this chart shows the performance of the graph 500 benchmark for a specific multi-CPU computer these results were generated by Peter Kogi at Notre Dame the graph 500 is a benchmark standard benchmark for data intensive applications there are two versions of the benchmark the results shown in this graph are for the breadth first search version as you can see from the chart for the grass 500 results it takes approximately 100 CPUs to achieve the same performance as a single CPU the chart also shows the performance for HPCG on the same system it takes over a thousand CPUs to achieve the same performance for one or several CPUs the traditional HPCG system is not capable of providing scalable performance results for this class of problems or the AGILE data problems So as previously said today the HPCG systems that are available and emerging in the next couple years are truly impressive computers so what is the problem with today's computers fundamentally the problem is that the original designs were driven by 3D multi-physics simulations the fundamental computation infrastructure has remained the same over several generations of designs vendors are interested in improving the performance of a computer by taking advantage of faster CPUs or GPUs utilizing stack dram or nvram and 3D packaging technology so this works extremely well when you're focusing on a node but the overall efficiency of the computer is not optimized for the emerging data analytics problems Today's computers have three primary subsystems these are communication memory and computation the communication subsystem is the network that interconnects memory and computational elements for most computers there are actually two separate networks often they are designed to move blocks of data not optimal for individual data items a memory system provides mechanisms for accessing and storing data the memory architecture typically provides optimal performance for cache line loads there is no local processing and the CPU is actually what controls all of the data movement finally the computation subsystem provides mechanisms for executing and controlling the floral path most are based on a von Neumann architecture which appears to be poorly suited for data intensive problems today's computers are mostly defined by commodity parts and interfaces for example a computer could be based on nvidia GPUs and CPU GDR or GDR memory and infiniband vendors are constrained by the standardization of the interfaces that determine the properties of data movement operations and are resistant to making major changes in the system architecture industry has been focusing on improving CPU GPU components and local memory accesses but not on global data movement or processing random data addresses or changing the computational model What we need to do is tear down the traditional barriers or boundaries between communication memory and compute elements in the design we need to change the overall system model yes break 60 plus years of tradition it's important that the AGILE architectures provide a highly productive environment clearly this change in the computing paradigm will impact the software productivity on our computer the AGILE program is not seeking to develop a special purpose computer that will result in only one computer being built if the computer is extremely challenging to program or execute programs then its widespread usage will be very limited the blurring of the component boundaries will increase the complexity associated with the development of software and its execution on a computer however the runtime system will have a significant and positive impact on this problem thus a runtime system becomes the fourth major subsystem in a computer it is what will glue everything together it is how high productivity will be enabled on the system Let's examine six technical challenges associated with the data analytics problems and as I said previously there are probably several that I have not included in here first is designing the processing elements to execute efficiently a broad spectrum a data driven task perhaps not known until execution time data analytics workloads can have an execution patterns with poor spatial or temporal locality and instruction streams with numerous branched conditions that cannot be predicted ahead of time next area is supporting fine grain irregular data movement ranging from the processing elements to the system level data irregularity will arise in both data layout and compulsory data movement to keep a data movement pathway fully occupied which is required to get the best back bandwidth out of a communication system uh irregular Sorry skip online next we support fine grain data fine grain irregular data movement ranging from processing elements to system level to keep the data pathways fully occupied there must be enough data movement requests in progress concurrently that will keep the channel full and thereby hiding the latency algorithms will exhibit data-driven parallelism at all levels fine-grained to core screen graph processing for irregular time varying structures has parallelism that can only be discovered in real time we need a new memory storage architectures for random dynamic to regular data structures AGILE workflows will have non-regular structures whose number shape and size are not known until runtime and may change dynamically during runtime since the data properties may be highly varied and unknown until runtime the system and even processing elements may need to have dynamic introspection The I/O system needs to be capable of ingesting high velocity streams of data from multiple external sources file semantics make it challenging to ingest data from multiple data sources into a file interface that has the sequential semantics while still maintaining global consistency across all clients system security data integrity compliance services need to support multiple applications cooperating with in a same memory space for modern high-performance computing systems multi-tendency usually involves many different users and applications from different access credentials to share the common database So in summary we've touched upon several of the challenges um associated with the emerging graph analytics problems these range from a fine-grained unpredictable date of movement to various levels of extreme parallelism to the need for computational elements that are designed for data intensive computations and security and compliance As I previously said what we have to do here is break the 60 plus years of traditional architectural design and these all started back in 1945 with the von Neumann architecture The primary characteristics of a von Neumann architecture are basically that they are optimized for AOU utilization there's a separation of compute and logic and memory and there is a some sort of mechanism for providing sequential instruction issue let's examine some of these requirements and why they necessitate a non-von Neumann architectural model for systems designed with analytic problems of the FPU or ALU isn't the most valuable resource it's operations associated with data movement in storage that are actually the most important future systems could take advantage of computing in memory or pre-processing the data before data movement and thus the need to move the data could be limited latency hiding schemes for data analytics requires asynchronous operations that result in unpredictable instruction issue thus breaking the sequential instruction capability global namespace will enable many features including moving compute to data There are more issues associated with data intensive applications and many examples strategies for addressing them I am only presenting several of them and the ones I presented aren't the only ways to address these issues As you can see from the slide this slide there are numerous research strategies for overcoming the AGILE technical challenges this slide provides a few of them this gives me confidence that the AGILE program can succeed we just have to be willing to tear down the barriers first involves the development of the integrated system level design again you have to start from the top not with a box full of components however I am neither advocating or discouraging any of the strategies you find on this slide I'm simply pointing out that there are possible strategies other strategies could impact an AGILE architecture the critical point is that your AGILE proposal needs to propose a coherent research strategy and why it resolves the technical challenges it is certainly possible that this presentation and the resulting BAA don't state all technical challenges and research strategies proposers should specify what they believe are the technical challenges associated with emerging data analytics problems and the justification for those statements and what research needs to be performed to resolve these challenges Okay the AGILE program what is it all about well the program objectives are first to process 10x more data second is to achieve data to acknowledge in 10 to 100 times faster than today's capabilities and three achieve the AGILE target metrics research teams will be funded to develop new validated designs that can achieve the program objectives and target metrics there will be independent testing evaluation team that will evaluate and validate the design the program will have two phases each lasting 18 months to improve and mature the designs during the first phase research teams will develop a high level behavioral model with a performance model for the proposed architecture during the second phase research teams will develop an RTL model for their architecture the metrics and and the testing evaluation process will be discussed later in this presentation The AGILE program will develop cost-effective and open designs this means it will not involve exotic expensive technologies that cannot be realized in silicon before 2020 before 2030. During the
program performers will openly discuss their designs the AGILE PM will include this information in future public and private presentations the designs will be developed using a co-design process that is driven by the four workflows benchmarks and of course the target metrics This is an important slide proposers need to understand what should not be included in their proposed effort for example the following will not be funded the design of the power distribution system a project where the design methodology is not based on a co-design process the implementation of IP that isn't open or licensable by other parties a system design that doesn't scale from a single node to a larger multi-node system also the program does not include the development of the programming environment however the runtime system and memory model will clearly have an impact on programming proposers should provide in their proposal a vision for the anticipated programming environment address the issues concerning how programmers will develop applications for the new architecture describe the methodology for converting a program written in a high-level language into an optimal Executable okay as I said at the beginning there are four workflows that will be provided there will also be kernels associated with the workflows and benchmark codes that are available today performers will have access to the source code for all of these applications you'll also be provided with um the target metrics and the BAA I list them here in the presentation um I encourage everybody to pay close attention to target metrics within IARPA uh a performer's ability to achieve target metrics is a very important measurement of success Okay so as I said there are four workflows the first one is knowledge graphs this involves groups relationships and interests detection this includes system and event patterns sequence data this means the identification and clustering of data sequences and network cyber which is the security of network systems for each workflow kernels will be generated to exercise one or more of the algorithms associated with the metrics and finally there are three industries standard benchmarks that will be provided the first is breadth first search which is graph 500 the same as graph 500 I should say triangle counting and Jaccard coefficients The workflows shall drive the design efforts this is accomplished by using a co-design process that allows the evolution of the design and implementation of workflows so that the resulting architecture achieves the best performance for all workflows the performance results will be evaluated using the agile enhanced modeling simulation toolkit called assist it is based on the Sandia National Laboratory software structural simulation toolkit SST thus clearly based on my diagram the co-design process involves three major components the workflows your design and the validation process the chasm between the demands of today's escalating data intensive problems and the capabilities of yesterday's computing systems is unbridgeable evolutionary improvements are now providing diminishing returns we need a fundamental architectural approach a fundamentally new architectural approach for the design of computers for data intensive problems I encourage proposers to take advantage of this program and develop architectures that are truly revolutionary we are not interested in a simple refresh of today's computers resulting architecture needs to not just achieve the target metric it needs to lead towards a new class of highly productive computers tear down the barriers by starting with a clean sheet of paper head towards productivity your proposal should discuss the impact of the design on productivity The funded research teams are expected to be comprised of researchers and developers with expertise in communication memory computation runtime subsystems data analytics applications and our focus on providing architectural innovations for data-driven computations the primary tasks are listed on this chart at the most basic level though research teams are developing system designs that can be demonstrated to achieve the target metrics using assist This this chart shows the different responsibilities for the research team and the test and evaluation team the research teams are responsible for developing the design the modules for a assist and the models that are used with an assist that are necessary to estimate the performance using assist and tailoring the implementations of the workflows kernels and benchmarks for their designs the testing evaluation team will validate the design and verify the performance using assist they will evaluate the tailored versions of the workflows kernels and benchmarks while it is certainly true that the research teams are responsible for utilizing the co-design process the testing evaluation team will monitor its usage and correctness in the design process so let's take a look at a runtime system because this is a very important part of this program there is more to this than just doing silicon design the design of a runtime is an abstraction of the computing system software structure and operation for a specific system model it provides a conceptual framework for the co-design of the computer including the architecture programming interfaces and system software the attributes of the main attributes of the data analytic challenges are extreme parallelism asynchrony self-discovered parallelism adaptive management and global name space An important aspect of the AGILE program is that performers are developing a holistic system model it is not as I said several times a collection of parts performers will need to start by establishing a system level performance model this requires consideration of the performance factors for the model in this example four factors are presented on this slide these are starvation latency overhead and weighting these factors are all related to the utilization of system resources and the execution of parallel work across the system proposers should describe their own system level performance model including the performance factors which may not be the same as listed on the slide describe how the performance model will drive the design decision Many of the design drivers that impact AGILE's performance models performance factors are not uniquely caused by the data analytics problems so one should keep that in mind let's talk about some examples of how data analytics affects slow performance factors starvation of a computing resource will occur when there isn't sufficient parallelism due to the limited number of tasks or threads in a data analytics algorithm increased latency will delay an operation when a data atlantic operation is gathering small data packets from several memory locations and there are insufficient paths to the data overhead which is wasted time is the time spent setting up data analytics transfers for data involving small packets waiting for a remote memory occurs when the memory remote memory has a large number of outstanding small memory requests proposers need to specify how data analytics problems impact the performance of the performance factors in the model As I said before there are four workflows these are all representatives of problems of interest in the intelligence community again to repeat myself there are four graph knowledge graphs detection sequence data and network but now we're going to talk about these in a little bit more detail and I'll also discuss the target metric but the BAA will provide more details than what I'm able to fit into this presentation knowledge graph workflows discover new entities and relationships traceable evidence paths complex events anomalous patterns and complex system models the goals for this workflow are to increase the data ingestion rate by a factor of 100 reduce the time required to both entity classification relationship discovery by a factor of 50 reduce transformation of data to knowledge from days to minutes Detection workflow identifies events event requirements and temporal evolution precursors and key event drivers and influences the AGILE goals for this workflow are to increase the size of the graph that can be processed by a factor of a hundred I'm sorry a thousand increase the number of data streams that can be processed simultaneously by a factor of three increase the capability from single event detection to multi-detect invention an enable incremental detection analysis incorporating new data without the need to re-compute both of which cannot be done today because of computational limitations Sequence data workflow seeks to extract common structures inheritance and functionality from massive temporal sequences of data the AGILE goals are to increase the number of sequence analyzed per unit time by a factor of a thousand reduce time to results from months to hours or days and estimate uncertainties or priority events something that is not done today Finally there is the network cyber workflow it models complex interconnected networks and their defense examples include power grid comms and control social networks etc. the AGILE goals are to reduce the time to identify key influential energies by a factor of 60 identify and prioritize anomalous activities and react to attacks in near real time so finally there are the graph benchmarks um this chart shows you these the three benchmarks that we're using there's brett's first search triangle counting and Jaccard coefficients I've also listed the target metrics for two different problem sizes and what must be reported So a lot of information um has been provided we discussed the program objectives benchmarks and workflows I want to give everybody insight into how this all fits into the overall effort from now and throughout the life of the program The objectives benchmarks and workflows play a critical role before and during the program proposers must analyze the benchmarks and workflows in the proposal and describe how their design will impact these codes and provide an estimated performance for the metric during the program research teams will utilize the benchmarks and workflows in a design co-design process that will result in optimal performance for these codes throughout the program research teams will report on the estimated performance of the benchmarks and workflows for their designs that are under development So the deliverables there are basically um a few this chart provides the primary deliverables for each phase if you look in the technical section for the BAA you will I'm sure have seen a much more extensive list of what the deliverables are um but these are the primary ones of interest the big ones of course and basically um it comes down to the designs with supporting documentation test plans and the modification of the application software Also the performers need to provide an application framework that will enable the the construction of the executables for the workflows benchmarks and kernels so that they could either be evaluated on an existing computer or the assist environment So let's talk about the testing evaluation process this is a very important aspect to this program that has been touched upon a couple times already The government test and evaluation team will reproduce and thus validate significant performer results validate all projections experimental results designs and test suites performers will submit their models and designs for validation and performance using assist achieving target metrics is one factor in considering which teams move forward to the later phase in the program proposals should be written with the testing evaluation process in mind testing protocols as written are preliminary and will be finalized by the program kickoff meeting This chart provides information concerning the task that was performed by the testing evaluation teams they range from validating the designs assessing the cops IP modules developing the assist environment which as I said is based on SST providing guidance on how to incorporate the design into assist basically the performers are developing a model within the assist environment and evaluating the modifications to the workflows kernels and benchmarks I would not of course be surprised that a some of the application software changes the implementation of the algorithms based on the change in the architecture so that seems perfectly valid we just need to validate that it's performing what is it supposed to be performing Okay so this is the program schedule for for phase one as previously stated the first phase will last 18 months nine months after the start of the program there will be a review of the performers preliminary design at the 17th month start of the 17th month performers will submit their designs for test and evaluation during this phase there will be several site visits PI meetings and the obligatory monthly reports um I imagine the way this program will be executed though is there'll be constant interaction between um the the design teams and myself and the testing evaluation team so we really won't be waiting until the 17th month to see what the performance is I think we'll be marching towards that as we go through this program Then for the phase two it's very similar to the first phase it's still 18 months long four months after the start of this phase there will be a review of the updated performers preliminary design followed by a critical design review at 24 months at the start of the 35th month performers will submit their designs for tests and evaluation and again numerous site visits and such I do want to point out something I skipped over on the phase one it's really important that these designs come in at the beginning of the 17th month I will not be allowed to advance to this next phase until the results have been determined about the performance of the workflows on the target with the workflows and how they achieve the target metrics this is an important consideration for the advancement of the program so therefore delaying that will simply delay the start of phase two for a particular performer which is another reason why throughout the program we want to have constant interaction between the performers and the AGILE team So I'm almost done I talk too fast sorry about that so you'll have some extra time to assemble questions anyway so the chasm between the demands of today's escalating data intensive problems and the capabilities of of yesterday's computer system is unbridgeable evolutionary improvements are now providing diminishing routines AGILE is the first program in decades to offer a clean sheet approach for completely rethinking system level computing architectures AGILE systems will enable a new era of data analytics applications turning chaos into order AGILE offers you an opportunity to pioneer a new class of high-performance systems that will change the face of computing to be the next pioneer in computing who changes the direction of computing I think this is a very unique challenge and I am hoping to receive some very exciting proposals Okay so thank you for listening to me describe the AGILE program I encourage you to propose a new exciting architecture that can genuinely have an impact on computing carefully read the baa write a proposal that fully responds to the BAA when describing the proposed architecture include a description how the design impacts workflows and benchmarks provide an estimate of the target metrics and finally be realistic when considering the cost proposal I have never seen a program at IARPA or DARPA where the program manager was able to get more funds for the program because proposers didn't include enough enough budgeted time into their program so please do not do this if something isn't understood when reading the BAA then please submit a question to the AGILE BAA email address I believe that this program provides a unique opportunity seize the moment thank you for listening to me okay sorry for the technical difficulties this is not what my job discussion is about okay good afternoon welcome back to the AGILE Proposers' Day um our contracting officer Chris Fox had uh had a personal emergency today and she regrets not being able to be here to actually go over this lives with you I am not a contracting officer I do not play one on tv so I am going to pretty much read this line if you have questions you are welcome to ask them in the questions section of this WebEx and we will try to answer them and if we cannot answer them then we will actually forward your questions to a contracting officer who will be giving you the official answer on behalf of the agency okay with that I'm going to go to the next slide maybe okay there we go so this is this is going to be a Broad Agency Announcement um I'm going to try to right now to to do questions and answers uh regarding to um regarding to this possible BAA who's eligible for applying how to prepare the proposal how to submit the proposal how the proposals will be evaluated and how the awards process will take place okay we'll talk a little bit about organization conflict of interest intellectual property pre-publication reviews academic institution acknowledgements multiple proposal submissions and contract types and there will be of course a disclaimer at the end So IARPA uses the BAA-type solicitations conducted under the far part 35 research and development contracting okay um you know where to go and look and look up the far and on on the internet and you can see what part 35 is all about the BAA will be posted on beta.sam.gov which is the place where all our baas are posted okay we typically allow we typically allow between 45 and 60 days for proposals after the BAA has been issued we are planning on this particular case to do I believe 45 working days which turns out to be about two months calendar uh by the time we are all done with the period for proposal writing all the information needed to submit a proposal for this opportunity will be included in the BAA nothing that I have said today or Bill has said today supersedes the material in the BAA so when in doubt go to the BAA the BAA will have a question answered period during which prospective offers can submit questions um that will be very clearly stated when we announce the BAA and you can submit your questions the emails for the question the email to where to submit the questions will be provided in the BAA the questions will be provided in beta.sam.gov
so be sure to check regularly for the answers to your questions okay no answers will go directly to the offer nor shall questions be sent other than than the email designator for the BAA and know that your questions will be posted at beta.sam.gov so if you think that you that there might be some proprietary information in your question please be very careful because we will make all your questions public and all answers to all your questions will also be public okay so be careful about that collaboration and teaming are generally encouraged by IARPA okay so from all the people in this call if you guys want to get together and make a single proposal because you think that a team proposal will be more solid please we encourage you to do that okay however we do not facilitate for you to actually uh for you to actually team that is up to you to actually come up and team with other people um foreign organizations and individuals let's talk a little bit about foreign people um this is the program dependent uh clauses and the BAA the BAA will specify the limitation okay there are other programs in path where we allow participation from foreign nationals and foreign entities okay regardless of eligibility any foreign organization or individuals must comply with any contract security clauses or requirements and any united states export controls okay and implementing contract contract clauses if you have doubts about that please send your questions about foreign foreign participation to us and we will try to answer them once we open the BAA the following are generally not eligible to submit proposals for IARPA research programs or participate as team members on the proposals submitted by eligible entities other government agencies we do not allow government scientists to respond to our BAAs okay FFRDC's we do not allow FFRDC entity employees staff to actually be performers in our BAAs we do not allow UARCs to be performers in our BAA okay uh there's a clause there for you are an entity of which only a portion has been designated at you or may be eligible to submit subjects to an um OCI review is started it is stated in the BAA okay um so let's take a university that only a portion of that university has been designated at New York elements of that university outside of the New York might actually be able to propose against the BAA and we will do a review a conflict of interest to actually ascertain whether or not you can participate other organizations that have special relationships with the government that will be given the access to privilege or proprietary information access to government equipment or real property so you see this is not really for government people to participate this is either for industry or for academia to participate so I'm sorry but that's those are the rules and that's how that's how we conduct business the BAA contains proposed proposal preparation instructions please pay attention to the instructions okay if we say five pages and you decide to send 10 pages the last five pages will be redacted and it's import if the important information that you wanted us to know about what's in those lives last five pages it will end up in the trash can okay so please be careful following the preparation instructions on the BAA the BAA will include the due date and the time okay page limitations and format information to be addressed in the proposal for example technical cost and administrative templates were required for post proposal attachments for example the cover sheet the conflict of interest forms academic institution acknowledgement id and data rights assertions cause breakdowns the BAA also contains the evaluation factors for a war including the technical evaluation criteria for example technical approach relevance to the IARPA work plan experience of the key personnel resource resource realism etc. okay so you will know by which metrics your proposal will be judged the PH describes a method for evaluation and selection the BAA may only request the technical volume initially with the detailed cost volume requested after selection so let me try to explain that for those of you that are new we we might the BAA might actually only be asking you for the technical portion of your proposal and if we like your technical idea and the way in which you are trying to address it we might then ask you to tell us how much is it gonna cost okay um but read the BAA the BAA will be very careful about that so technical volume means the technical portion of the proposal the cost volume means the resource portion of your proposal proposals must be submitted through IARPA idea system the IARPA idea system if you see the website there it's iarpa-ideas.gov um you guys have to register electronically into ideas before you can actually be able to offer your proposal against a BAA okay um right now if you're going to try to raise her you will not be able to do it because the BAA is not yet on the street so wait until the BAA hits the street through beta.gov and then go into ideas
register and then you will be able to submit your proposal insurance that offers are strongly encouraged to register with ideas at least one week prior to the proposal due date why in the past you know bad things happen to computer systems in the past we have had events in which ideas has gone down for a couple of hours it just happens that it was the last day of a BAA and you were desperate to try to get in your proposal and now you're going to register so take a week out of would we make maybe registration at least a week before so if there's some technical delicious we can actually help you overcome them offers more most ensure that that the version submitted to idea is the final version um be careful double check your document okay don't send us track changes and stuff like that census sent us a clean final version of your proposal if you decide that your proposal is going to be classified the BAA will explain to you how to send a classified proposal to us okay um the BAA will have instructions of how to respond in their system pros with ideas I already said that before just follow the instructions that will take care of you okay the cost volume is not requested until after selection it will be directly submitted to the contracting office officer not two ideas so let me explain this again once again we are looking for the technical portion of your proposal okay that's the one that you send through ideas a selection will be made based on the technical merits of the proposals okay at that point you are not at the other side of the rainbow yet at that point you will enter negotiations with the contracting officer and you will have to submit the cost volume to the contracting officer directly okay based on that negotiation with the contracting officer you might end up having your proposal rewarded with an award from IARPA okay each BAA will detail the method for evaluation and selection but IARPA generally follows a two-step process first step is evaluation and selection for negotiations this is conducted through a scientific peer review process after which offers are notified of selection I just said that like a minute ago the second step is a negotiation and contract award conducted by the contracting officer I said that a minute ago okay so we will deal with your technical idea first and then we will deal with the contracting part okay proposals will be reviewed individually against the BAA requirements in accordance to 535 and not against each other this is not a beauty contest we take each one of your proposals on its own merits if your proposal addresses the requirements of the BAA we will start a conversation with you to follow up okay IARPA follows far part nine regarding organizational conflicts of interest the main principles being preventing conflict conflicting rules that might bias a contractor's judgment and preventing an unfair competitive advantage okay the BAA will describe how offers are to identify and disclose all disclose all facts related to potential organizational conflicts of interest for the offer as well as any proposed team members it could be that in your team it's only one person that might have an OCI then you disclose the possible OCI of that person and we deal with it one-on-one okay OCI disclosures may require a mitigation plan describing the actions of the offer that the offer will take or intends to take to prevent the conflict in other words just because you have an OCI doesn't mean that we're going to kick you to the street we will try to work with you to develop a mitigation plan to try to take care of it so you can still contribute to the to the BAA IARPA generally prohibits contractors from currently providing systems engineering technical assistance theta and T&E support while being technical R&D performers due to OCI concerns each case will be determined in the beach individually so like you remember from my presentation earlier today we typically break the world between those people that are proposing and performing against our programs and those people that are doing uh technical evaluation okay and we actually separate the two so if you're playing on one side of the fence you cannot play on the other side of the fence and vice versa there might be an incredibly weird case in which we might allow you to do that but we will determine that on an individual basis the government needs to be able to effectively manage the program and evaluate the output and deliverables communicate the information across government organizations and support further use and development of program results authors will address their intellectual property rights assertions in their proposal the government may request additional information as may be necessary to evaluate those the government will evaluate the IP rights being offered and whether they are in the government's best interest in other words protect your proper your intellectual property okay we will help you do that but your intellectual property is yours it's your responsibility to help us help you protect it okay IARPA encourages the publication of of unclassified IARPA performer research in peer-reviewed journals presentations at conference and publication in conference prior to the publication release of any work submitted for publication the performer will communicate the results to be publicly released with the IARPA Program Manager Bill Harrod in the scales to discuss any sensitivity for example security speculation on icus etc. and provide advanced currency copies to the IARA program manager and contracting officer of the publication there's a process inside IARPA for the review of three pops and I can tell you that until I sign off on the pre-publications you will not be able to publish that's the responsibility of office directors like myself so we take this very seriously we address your request on a timely manner and you are able to public to to publish your your work okay according to executive order 12333 contracts or agreements with academic institutions may be undertaken only with the consent of appropriate officials of the institution an academic institution acknowledgement letter is required for offers that are academic institutions and for any proposed teammate that is an academic institution a template of this letter will be included in the BAA each letter must be signed by a senior official of the institution for example the president the chancellor the provost or other appropriate designated individual IARPA requires this letter before entering into negotiations and or awarding contract it is highly advised that it be submitted with the proposal ask permission to submit the proposal from an official at the university before you send it to us without that we cannot consider it proposal submissions to other entities typically the behavior the BAA has offers to name in their proposal other federal state or local agencies and other parties receiving the proposal or substantially the same proposal for funding of the proposed effort if the offer has submitted the same or substantially the same proposal to other entities it may impact decision to select and fund the effort I promise you we talk to each other if you send your proposal already to DARPA and you're sending it to us again we will know that okay so please avoid doing that multiple proposal submissions to IARPA the BAA usually allow an entity to participate in multiple submissions as a prime or a subcontractor if allowed by the BAA multiple submissions which include a common team member shall not receive duplicate funding for the same work I no no one entry can be paid twice for the same work so if you want to go and be the guy that develops gadget a for proposal one and you want to be the guy that developed gadget a for proposal two that is allowed however if proposal one and proposal two are selected you will not get paid twice for gadget a Cost or cost plus fixed fee type contracts are typically awarded due to the nature of the R&D work IARPA may in some instances consider other contact types such as firm fixed price as well as non-part-based agreements such as other transactions the types of contracts and agreements that will be considered and the conditions for such consideration for example small business startups commercial foreign entities etc. will be addressed in the BAA so please read the BAA careful about that the information conveying this briefing is for planning and general information purposes and is subject to change please carefully read the final BAA and adhere to its requirements which may differ from what has been presented in this briefing in other words my word or the word word of Bill today is not the final word the final word is the BAA and with that it's time for me to be quiet and allow you guys to continue to send your questions in there's an army of of IARPA uh staff members right now reading your questions answering them and then Bill will
2021-01-09