hello and welcome to the mit sdm systems thinking webinar series my name is naomi gutierrez communications administrator for sdm and i am today's host thank you for joining us today's speaker is dr tyson browning professor of operations management in the neely school of business at texas christian university in this role he has conducted research on managing complex projects in greeting integrating managerial and engineering perspectives and taught mba courses on project management operations management risk management and process improvement he is currently co-editor-in-chief of the journal of operations management in addition to his academic research and presentations dr browning has trained and advised organizations such as bnsf railway general motors lockheed martin and the us navy he holds two master's degrees and a phd from mit and we're pleased to welcome him back virtually to mit today his talk today is titled planning tracking and reducing a complex project's value at risk if you have questions for tyson please enter them into the chat window at the side of the video they will be addressed during the q a portion of this session and a recording of this presentation will be available online after today's session a link will be sent to all registrants and with that tyson please go ahead thank you naomi it's a pleasure to be here with you to come back to mit virtually and present on a topic that goes back to my time as a graduate student at mit as you can see from the list of sources at the bottom of this title slide this goes all the way back to my dissertation as a topic that's been on my mind and thinking and part of my research for about 25 years it began when i was a researcher with the lean aircraft initiative as it was called at the time at mit and we were working with companies such as boeing and lockheed martin and others and one aspect of that project dealt with product development and questions of what is lean how does lean apply in that context and really sparked a lot of the questions that led to this topic and this study some of these motivating questions are as follows and certainly apply very broadly not just aerospace not just to product development but all kinds of projects and they include how to measure progress in complex projects what does it mean to make progress how do we know if we're making progress how do we know if that progress is enough how do we know if we're ahead or behind schedule or plan or is that even right how do we plan and track not just cost and schedule but things such as technical performance quality scope and then for that and for cost and schedule how do we account for uncertainty how do we account for risk opportunity how do we trade off among these areas technical performance versus project cost or budget and duration or schedule and how do the uncertainties and risks and opportunities figure into all of this so these types of questions are important and as we go through them i'd encourage you to think about the projects you've been involved with your experience how would you answer these questions for those projects and what challenges did you run into accordingly regarding product development projects in particular i'd like to highlight a few observations first these projects are typically aiming for some kind of result that is trying to achieve chosen goals whether we call these objectives or targets or requirements maybe different words for different contexts i'll use the word goals generically here as what the project's aiming for often assuming that by achieving those goals the project's going to add value or be successful which is not always true but we can kind of take as a given that achieving these goals would probably be a positive thing for most projects product development is a creative discovery process it's iterative not linear it's not so easy to layout the activities the tasks between where we begin such a project and that final destination we're aiming for often we'll learn a lot as we go and the task to be done simply checking the boxes is not sufficient to guarantee that we reach that chosen destination and those selected goals therefore in product development projects but i think more broadly in a much larger variety of complex projects uncertainties are costly and some more so than others and as we gain knowledge as we learn about what will will not work on the project and the path towards that destination then that knowledge replaces our uncertainty hopefully reduces our uncertainty and this is valuable this is useful one of the most common ways to track progress in projects is called earned value management and i figure some of you may be familiar with it others maybe not but this is a method that's coming to the mainstream it's part of the project management institute's body of knowledge in their guidebook for example and it's been around for decades and attempts to track things like actual cost and earn value against a planned value but it has some problems and shortcomings it's better than not tracking these things at all but it can be misleading first it takes a completely deterministic view of cost and schedule it doesn't account for uncertainty or risk it assumes we know exactly how long activities should take for example how much they should cost it ignores completely the third dimension of what's sometimes called the iron triangle in projects there is nothing there about quality or performance or scope of the project except what's implicit in terms of the activities that have been specified evm tracks what's really called perceived progress in terms of a famous paper by cooper in the project management journal about rework meaning that there's a difference between actual progress and perceived progress due to undiscovered rework and other things we don't know we don't know yet about what's lacking in our results as we go through a project so despite the occurrence of this word actual throughout the earned value management methodology it's actually perceived rather than actual progress it's getting tracked and therefore evm can sometimes create some perverse incentives for example we get credit we earn value in evm by doing activities so we're often choosing to do easy activities first because it gives us credit or we may like to start some easy activities to try to get back on track if it looks like we're behind this can cause us to start some activities prematurely which in projects is very problematic often there's nothing physically preventing us from beginning certain tasks because the inputs the precedent precedence relationships or dependencies for those activities are often purely information and in those cases we can simply make assumptions about that information and go ahead and begin the activity but this increases the risk of these activities going forward that they will have to do rework later so not only does evm ignore rework it actually potentially could make it more likely we all also should question when we see the word value in evm what does it really mean what type of values really being managed there's nothing here about stakeholder value or the value of information or value at risk some concepts that we'll be thinking about a little more as we go forward here ultimately progress depends in projects not on the boxes we check but on what we actually accomplished it's not how we spin our wheels but it's the traction we get to move forward that matters somehow we need to differentiate our progress in projects from merely being able to say we've done certain things rather what have those things now allowed us to know and what value do they therefore add i'm going to introduce several concepts and in some cases add some quantification to them to give you kind of the tip of the iceberg of a framework called the project value risk and opportunity framework the first concept i want to introduce is project value and there's a couple aspects to this first let's realize that project value depends on what the project actually delivers at the end therefore until that point for sure we don't know exactly what that actual value is going to be the value for of a project depends on the stakeholders preferences for a combination of some attributes of the project and its results if this is a product development project then what are the attributes of that developed product that's going to go a long way to determining how satisfied the stakeholders would be with the project's result stakeholders generally want more of some attributes or aspects from the project than its result and product development or service development these could be some examples like features functions how reliable is it how large or small is it how fast or slow is it is it available when we want it what's the design aesthetic and so on some of these attributes are fairly straightforward to quantify others less so and of course stakeholders may want fewer or less of some other attributes they always would prefer to pay less both up front and in an ongoing sense they may want it to be lighter they may want to have it faster and so on so thinking about products that you purchased there's typically just a handful of key attributes that really in the end make that purchase decision what are the order winners in those cases that's a similar idea to this one for a project what are the things the stakeholders ultimately are going to really focus on as they decide was this project successful or not and so we can think of the overall project value as a composite of the salient attributes and i've found that in most cases even large complex projects there are fewer than 10 of these that are extremely important to focus on and that's good because that's a manageable number and i'm going to call these project value attributes or pbas for short in this framework we'll attach a value function to each pva this is where we quantify things a bit this is essentially a kind of utility function although the units we might choose to use on the y-axis are not necessarily utility in a zero to one scale but in this case in this example we're using revenue essentially a market size that a product in this case a drone aircraft would be trying to capture some portion of and here the value function is for the project value attribute endurance how long can this drone aircraft fly continuously before it would have to land clearly this is a larger is better attribute more is better so you see that as the endurance increases from left to right on the x-axis the amount of revenue expected from being able to sell a product with this attribute increases and market research focus groups discussing with customers this is what marketing functions are doing all the time there are a lot of important pitfalls to avoid and guidelines to follow in assembling these type of value functions this one happens to be shown here as piecewise linear of course they could be in a number of shapes or forms but the idea is for each of the five to ten pvas on the project to develop one of these value functions they might be larger is better like this one increasing from left to right for our drone aircraft example here are five other pvas and the first three of these other ones are also larger is better notice a few different shapes here we have one for unit price where some nominal amount is best and more or less is less desirable and then on the bottom we have one where smaller is better here it's delivery lead time as that increases the market's interest in buying this product also decreases i also want to think about project value in terms of four different types there's the type of project value that we would hopefully know at the project's completion point now technically this is not true that we know it fully and completely even at that point but for simplicity i'm going to say that that's the part of project value we'll know the most about when the project is completed what's its actual value meanwhile prior to project completion especially early on in the project there are other ways of looking at value that we should think about first what's the desired value what do stakeholders really want sometimes they can articulate this sometimes not sometimes it may be tacit so the desired value from stakeholders for the project and its result we want to distinguish that from what i'll call the goal value meaning certain goals and requirements and targets objectives have been chosen for our project what's the value of merely meeting those these may or may not be the same as the desired value because we may have deliberately or inadvertently chosen goals for the project that differ from the stakeholders ideal result and often it's hard to get the ideal result for all stakeholders often we're content to satisfy or somehow accomplish something that's valuable but of course stakeholders could always want more so there's another argument why the project's goal value might be different than usually less than maybe an ideal desired value from stakeholders and then a fourth way of looking at value in an incomplete project is something i'll call the likely value given the project's capabilities i'll talk more about this in a moment what is its estimated value at completion what is its forecast outcome for each of these project value attributes that we're interested in again that may or may not be forecast to meet the goals depending on how challenging those goals are let's talk more about this goal value of a project for a moment and we'll go back to the value function for the endurance pva for each project value attribute say the project sets a goal and for endurance let's say it's 22 hours well once that goal has been chosen or selected there it implies some amount of value and if we're measuring value in terms of revenue that we would expect from selling these units after they're developed and produced then here we could find what the value of meeting that goal would be in terms of revenue and these types of business cases and projections are done all the time of course they're fraught with many difficulties but they're often the best we can do and really this framework what we're trying to do is pull from marketing the best knowledge we have about these aspects and try to integrate it into a framework that systems engineers and program and project managers can all work together and make trade-offs as needed based upon the overall goal value for the project of course will depend on the combination of pvas not on just one of them we also want to think about how the project goals are partly responsible for determining the risks faced by the project here those three types of value that we are tracking or thinking about prior to project completion and that's knowing its actual value first we want to think about if there's some risk we won't meet our goals in our project if it's likely value is somehow less than its goal value this is typically what we think about in terms of project risk risk of not meeting the chosen goals or objectives we can also think about any difference between the goals and objectives we've chosen and what the stakeholders or the market or generally might desire and think about this as market risk the reason for distinguishing these two is because it begs a question that some projects face about how high is at the bar what goals and objectives should we choose and it's interesting to observe in many cases here that if we make the goals easy to achieve if we lower the bar then that reduces the project's risk however it typically increases the market risk as we lower that gold value towards the likely value the project risk will go down but the market risk will tend to increase similarly if we raise the bar make the goals more challenging then we can maybe cut into that market risk but we might make our project more challenging and make its risk higher i'm going to come back to this in a few minutes but for now let me introduce a second concept called project capabilities the idea here with project capabilities is essentially how likely is the project to be able to do what we want it to do this depends on the resources available to it by technologies expertise skills people processes and tools all the things that go into what we often refer to as capabilities what are the potential paths we might take to reaching the project's goals what obstacles or challenges might lie along that those paths what partners suppliers do we have and what are their capabilities and how well are we able to manage all of this all of these figure into what i'll call project capabilities and the idea here is that for each of the project value attributes each of these key characteristics that are salient to determining most of the project's overall value let's make a forecast of how good are we how likely are we to be able to get certain outcomes for each of these project value attributes pvas we're going to call these forecasts project capability distributions the idea with these distributions of probability distributions as our capabilities improve these distributions will shift in a favorable direction if larger is better then they'll shift towards the right and if smaller is better they'd shift towards the left if we're talking about movement on the x-axis for example also they capture uncertainty and the capability to achieve the project value attribute outcome so if we've got less uncertainty about our outcomes we'll have a fairly narrow distribution if we have more uncertainty we'll have a wider distribution going back to our drone aircraft example here's a project capability distribution for the endurance pva based on answers to three basic questions what is the best case what's the worst case and what's the most likely potential outcome so we're asking our subject matter experts more from the engineering and technical side the designers rather than the project value attribute functions where we're talking maybe more to the marketing people earlier now we're bringing in the perspective of the project's subject matter experts and we can do this in a fairly more sophisticated way like a delphi process we can use a frequency distribution of votes we get from even like an internal market to build this type of distribution but here i'm just taking a very simple triangle distribution based on answers to those three questions and inferring what is the relative likelihood of endurance outcomes in this case between the best and worst case outcomes and given that there's a most likely outcome somewhere along the way there similarly do this for each of the other project value attributes here in this drone aircraft example there are six and we see you know simple triangle distributions for each of these the idea though is that as we move through a project as we gain more information hopefully we'll be able to narrow these distributions as we learn more about the potential outcomes this slide flips the distribution on its side so now the pva is on the y-axis instead of the x-axis like on the previous slide and here time is on the x-axis as we move left to right through project time we've got essentially a third dimension of probability coming out of the slide here where these triangle distributions over time are getting narrower now there's no guarantee that they will shift in a favorable direction which here is upwards as they're shown in this example so we can see at the left side the probability of meeting our goal which is the dash horizontal line is fairly low but towards project state n plus two later in the project it's fairly high well this is very nice when it happens but it's not guaranteed of course but we could think about what causes this distribution to change shape and to move up or down as we go through the project first though let's think about is merely tracking the probability of reaching a goal sufficient here in this diagram i'm showing two distributions pcds at different points in project time the lower flatter wider distribution is initially in the project it's very wide there's a lot of uncertainty about what the eventual outcome will be in terms of the drone aircraft's range however later in the project we have a much tighter distribution with much less uncertainty in the outcome what's the probability of failing to meet the goal it turns out it's essentially the same in both of these cases so it should be obvious for this reason that probability of meeting the goal is not sufficient enough to tell us what's going on here what's really changed well later in the project we're in a much better situation because even though there's some probability of an outcome less than 2100 there's no probability of an outcome less than about 2075 which is meaning if we miss the goal it's not going to be by much whereas early in the project there were possibilities of outcomes as low as 1950 or 2000 that would have a much larger negative impact on the project's value so monitoring uncertainty is not enough we also need to think about the impact of these outcomes and this is where we start heading towards the idea of risk so to think about the impact of an outcome it could be positive or negative a bonus or a penalty with the endurance attribute we started with earlier if we chose a goal at 22 hours for example it would imply a certain amount of value here about 1.3 billion dollars in potential revenue for our project whereas if we miss that goal let's say we come up with eventually a drone aircraft's endurance only at 20 hours then there's some penalty here 334 million that we could read off of our value function if we look at 20 hours and its result over on the y-axis so for any outcome that fails to achieve our goal we experience some kind of penalty in its value on the flip side if we exceeded our goal for example 24 hours then we're going to get a bonus in our value here any outcome that exceeds the goal could have a positive impact risk will be quantified here in terms of uncertainty and its adverse impacts so we're going to sum over that region of outcomes that fail to achieve our goal and look at their relative probability and their value penalty so the adverse outcomes the probabilities come from the pcds the value penalties impacts are coming from those pva value functions and then we do this for each pva and we have to build a composite to look at overall project risk this risk index as we'll call it captures information about both uncertainty and its impact in a single scalar variable which is a useful construct that we can talk about as maybe the cost of our uncertainty or the expected value loss due to this performance outcome the portion of the project's value being put at risk by the prevailing uncertainty and its impact these are different conceptual ways we might have have an intuition about the meaning of this term risk let's i think put this in a useful context uh using maybe an olympic high jumper or pull vaulter as an example if we put our projects outcomes on a vertical gear better outcomes at the top worse outcomes at the bottom and we set our goal here at the bar at some point then we have a distribution of potential project outcomes where we'll actually end up with our project and the examples i showed before were triangle distributions here it's a normal distribution we can use any distribution that makes sense for a project value attribute or for the overall project the idea though is that some of the project's goal value or the project's goal value is determined by the height of that bar in the context of high jumping or pole vaulting you don't really get additional value by clearing the bar by a large amount but in our context here you might get a value bonus by getting over the bar at some higher level however the possibility of not getting over the bar the adverse outcomes are what put some of the project's goal value at risk and similarly if we've got a good outcome a good possibility of a value bonus then we could get over that bar and achieve some additional value at opportunity this means that we can think about designing a project for value tailoring it for risk at the very beginning based on the goals we choose and the capabilities we have essentially we're thinking about the likely value of our project depending on where we set the bar how high is it and what are our capabilities how good are we as a jumper in terms of each of these project value attributes we face these goals and uncertain capabilities are going to going to render a portion of our project's goal value at risk and they may also point to some additional value at opportunity sort of on the table if we increased our goals there's a quantitative framework for all of this in one of the papers mentioned on the title slide i'm not able to go in that in great detail but i would like to give you a high level overview of this pvro framework it's essentially the concepts that we've introduced already we've got a set of project value attributes each with a value function and each with a capability distribution we select project goals for these we use the goals and the value functions to determine what are the goal values for each of the project value attributes and we also use them to determine what portion of that value is at risk and at opportunity we need a composite model and i don't have time to go into those right now but the papers do that allow us to combine these pva values and look at the overall project goal value similarly an overall project portion of value at risk and at opportunity we then can use these to help plan the project to think about all of the dimensions all of the pvas and how much risk and opportunity there might be in each realizing that part of that's due to where we set the bar what goals have we chosen we may want to increase or decrease renegotiate some of the goals and requirements also could depend on our capabilities as a jumper maybe we need to pump more resources skills technologies into certain areas all of this helps give us a big picture view that helps us try to take on appropriate amounts of risk not on inappropriate amounts and think about how we're trading off and balancing among the important aspects that drive the project's ultimate value in addition to planning projects with this we can also track monitor control projects in an ongoing sense when we think about these capability distributions generally decreasing in size as we learn more gain more information this is useful for tracking progress and we can think about that in terms of individual project value attributes we can also think of it in terms of a composite for the overall project are we decreasing the portion of the project's value being put at risk by the likelihood of the outcomes and these project value attributes rather than going more into the quantitative aspects i want to focus more on the conceptual aspects in the few minutes we've got left as we move through the project let's return to these questions about how do we know for making progress what does that look like what does that really mean well as a project unfolds we're gaining useful information and we're moving towards that point at the end of the project where we'll know it's actual value remember at the beginning of the project we don't know that yet we can only try to forecast it so we're doing things in our project we're accomplishing activities that produce useful information and it's this useful information that's really adding the value well how's that happening it's replacing it's eliminating uncertainty it's converting potential value to actual value it's decreasing the portion of the project's value being put at risk by that uncertainty so how do we add value not by just doing things but rather by the information by the results produced from the accomplishments of activities or we can view this differently we can focus on the green trying to add value we can also focus on the red here and think about progress in terms of removing the threats to value what i'll call anti-value let's connect this conceptually to some other project examples uh here is some interesting data from an old harvard business school teaching case that looks at the difference between estimated and actual project duration as the project moves forward at the beginning of the project in 1984 they estimated how long would it be until this project was completed that estimate was wrong how much was it wrong by almost 1800 days years in other words however as they progressed through the project over time the difference between the estimate of the project's outcome in terms of its duration and the actual outcome continued to decrease so even though they were not ever really able to perfectly predict that outcome until a few months before the project completed their ability to hone in and get closer with those predictions is clear and that's the idea trying to replicate here that as we move through time we can't necessarily know what the most likely outcome is going to be certainly not yet what the actual outcome is going to be for each pva but we are often able to eliminate some of the extreme cases as we learn more about what will and will not work we are able to hone in on what the likely outcomes for our particular pdas will be on a project so as the projects evolve their outcome estimates evolve and this is largely due to the information produced by the activities that are done on the project this is even really interesting to me in a broader sense because i don't know if any of you are sports fans who noticed on the espn app and perhaps elsewhere these charts emerging over the course of a game seeing these for football for basketball for other sports where there's a probability of winning to outpry their team and this probability changes throughout the game each play run an american football game is changing this probability and it's very interesting very early in a game a simple running play that doesn't gain any yards doesn't nudge this too much but in the last minute of a football game a simple running play that gains no yards but uses up time could make all the difference in the world here and so the idea is that each play in the game is more or less affecting the probability of its outcome and that's changing over the course of the game it's not just the play but when it happens that also matters and that's the same idea we want to think about in terms of progress and projects each activity we do each outcome we get the information we produce revises our ability to estimate where we're going to end up with each project value attribute so we want to focus not on the whole ocean of information out there but on the information that really revises the things that matter the things that determine value for our project and its result and we're capturing that with these project capability distributions as they evolve over project time risk opportunity value also evolve and we can think about reducing uncertainty which is not itself enough we have to think about are we actually reducing the risk so as we think about as our project making progress what's really happening over the course of the project the uncertainty tends to go down project capability distributions are getting narrower design trade-offs may shift these pcds as we change priority as we reallocate resources we can shift these up or down we may change their shape as well as the pcds evolve the likely value the risk the opportunity that we're tracking for our project will change and of course any adjustments we make to the project's goals if those get renegotiated or reset those could also affect this the pvro framework that you can read about more on the papers i mentioned earlier quantifies and monitors all this for individual pvas and then as composites for overall projects key ideas here we want to think about reducing a project's portion of value being put at risk it's the value-adding work that we're doing in the project that's producing useful information that reduces the portion of the project's value being put at risk by these threatening uncertainties we're trying to increase our competence in being able to get over the bar that we're not likely to get adverse outcomes that are providing huge threats to our value and our goals here this leads to a different way of defining a project or conceptualizing a project as the work we're doing to eliminate the risk of not achieving its goals this is a kind of double negative in its construct and i realized that but what it's really saying is that we're trying to look at a project and its progress and it's adding a value maybe less as a painter with a blank canvas at the beginning to which we're putting stuff on it maybe more as a sculptor where we start with a block and we're taking things away to get what we want or to prove to demonstrate to verify and validate that we're actually going to provide something of value implications here as we want to manage a project for value we want to think about tracking not only the best current estimate of the project's situation or value but also think about the uncertainty bounds around that estimate there's a lot of additional benefit by accounting for those and their implications for risk and opportunity like i said earlier earned value does not do that it seems a point solution a point estimate and here we want to think about that point estimate could bounce around like a stock market ticker as the project moves there's no real way to know or predict exactly where it will end up but we do have the ability to essentially assume that the distribution around that point will get narrower and narrower as we learn more and more and we can actually choose activities that will produce information that deliberately caused that point of the narrowing of the distribution to occur sooner rather than later real progress then produces useful information that reduces the portion of the project's value at risk that essentially burns that down we're chipping away the anti-value the threats to the value to reveal the clearer image of our project's actual value and if we don't like the way that looks then we're going to have to control the project by recasting something changing the goals maybe raising lowering the bars and some of these pvas or reallocating resources whether that's adding resources to the project overall we're just shifting them to emphasize different pva as managers we can plan we can identify appropriate things to be doing in the project and when we do them in the project we'd love to fail fast we'd love to tackle the big risks first in our project to demonstrate viability rather than allow a project to begin doing a lot of simple easy things just because it can claim credit or quote earn value for them we want to focus on the things that really matter earlier in the project to create appropriate information reduce appropriate risks to force us to identify areas of ambiguity break those down not starting all of our project stoplights in green and then being surprised later when they turn yellow or red but rather kind of start red and assume that we don't know yet where we're going to end up until we can prove that we're yellow and then green we want to focus on key performance attributes and driving early estimates of those is useful so as managers whatever we can do to try to think about what's the relative likelihood of ending up with certain outcomes what are the implications of that that's useful this helps us emphasize the importance of uncertainty and risk reduction not just in product development but in projects in general it encourages designers and those working on projects to communicate in terms of spread in terms of confidence in terms of uncertainty not just assuming that if i give you a number that's guaranteed to be exactly what it will be at the end of the project so decrease emphasis focus on point solutions and this fits very well for those of you familiar with concepts like set-based design that we want to think in terms of ranges of possibilities and narrowing those over project time as we can hopefully converge on solutions rather than just going in and telling everyone it's going to be x but everyone knows by tomorrow or next week it will no longer be x this helps us link activities to results that they provide i think activities are a nice uh construct a nice building block really the nexus of methodologies here because they're already essential in project time and cost models however we need to think about that third performance quality technical dimension and tie that together with the time and cost and also think about the uncertainties in each of those and i think activities provide us a way to do that if we will think about them not just in terms of the time and money we put into them but the quality of the information and outcomes they're providing this provides a scaffolding not just for planning and managing tracking an individual project but for organizational learning across projects as we do more similar projects and generations of products and other types of things we get better at this we learn not just about the individual project but what to look for across projects and we can capture that useful way in this framework so to conclude i want to just make sure a few key points are clear first when we think about doing work and its relationship to adding value and projects there are a couple key intermediate ideas that matter a lot first it's not just the fact that we spent time and money to do work but what is that work actually produced and accomplished is that information valuable useful for our project does it reduce the risk of not getting what we want and can we think about that as the path to adding value another big picture idea here is that essentially project management program management these things are risk management because every decision made by a project or program manager essentially it should be reducing the risk that the project will not get what it wants that whether it's to do something now or later whether it's to put resources here or there these are all essentially trade-offs being weighed that will increase or decrease risk in various parts of the project so each decision could be viewed as a kind of risk management decision and ultimately think about a project as the work we're doing to eliminate the risk of not getting what we want not achieving the goals that we set out some portion of that project's value is being put at risk by the prevailing uncertainties can we chip away at those can we produce useful information that gives us a clearer picture of the project's actual value hopefully aligning with our goals so there's a quick overview this has been mostly qualitative conceptual i again would like to point you to the papers on this topic and those are again listed here at the bottom these slides are available if you'd like to request them and with that i would like to take your questions thank you so much tyson this was fascinating and i definitely feel that i personally learned a lot about what calculating risk can actually look like um while we're waiting for questions to come in which can be put in the live chat on our youtube page um i wanted to ask about potential outcomes might look like especially with new technologies or new i don't know if product is the right word but when you're dealing with things where it's much harder to figure out what those distributions will look like do you just kind of slap a bell curve on it and call it a day or is there a better way to do that good question the quick short answer is see the papers for more explanation and like i quickly mentioned in the presentation there are various ways of doing it and they range from yes simply slapping on a distribution hopefully with some educated uh framework for for why you're putting say the the mean or the most likely or the media at some particular point rather than another but often we can do much better than that even with fairly novel type technologies and projects the reason is because certain outcomes are just infeasible and because other outcomes are just unwanted and we would clearly abandon a project if we saw it heading towards that outcome so here we're trying to collect the wisdom of those who know about this project and that could range from a few experts to really everyone involved how we do that whether we can do that efficiently depends somewhat on the technologies we have best buy is a company for example that took an approach like this to forecasting demand for particular products and built it into their software where they pull all of their workers at all of their locations across the world and ask them what do they estimate demand for this particular product will be next week or next month and they just pool all of those forecasts into an overall forecast that's a similar idea that could be followed here we're looking for people who know about this situation to tell us what they think the outcomes could be and why and some people you'd think would mostly cluster towards the middle but others might be outliers they might have particular reasons for being especially optimistic or pessimistic so those reasons could be captured and all baked in here and the first time one does this yes it's the most challenging but there's a learning curve to doing this that makes it easier over time and as one does it with more pvas and projects that's really helpful thank you so we do have some questions coming in on youtube one of them from thomas eisenberg who asks what experience do you have where the project value is not measured in dollars where the outcome is truly binary yes versus no in these cases and by the way it doesn't have to be in dollars let's see if i have a yeah here we go uh this one happened to be for convenience i will say it's most useful if you can express all of the pdas and common units that's probably more important than that those units be in dollars they could be in terms of simple utility like a zero to one scale it could be a number of units sold they could be in something else but here i also want to point out this upper left value function for maximum range of the drone aircraft happens to have a step function built in where it's determined that for this particular market they're looking at there's just no value if it can't have a range greater than 2 000 nautical miles the no one in the market's interested in that probably because there exists other uh products that already do that and so notice that these value functions are affected by a variety of things stakeholder preferences wants needs market forecasts and also what substitute or competitor products or services might be available and so all of those things get figured in here as well so both these pva value functions and the project capability distributions are challenging to produce the good news is they don't have to be perfect they don't have to be exactly right but just putting something down and then being open to refining it as we learn has been very valuable because it's very useful to just kind of know where we start where we are and if there are particular areas we need to know more about or we find that our outcomes are particularly sensitive to we can then invest accordingly in learning about those specific areas and that's really been one of the benefits here is helping direct resources accordingly thank you uh we have a question from tom woodman asking about examples of using this to measure social impact that's an interesting idea and i haven't done that myself specifically but i don't see why it couldn't figure in here really any key attribute of a project or its result and some are easier to measure than others for sure but even something like social impact there have been a variety of metrics proposed for that and so choosing one or more and a composite could come up with some measure like that that could be useful and applied here i should also mention that when we are focusing down on say five to ten key salient pvas for a project clearly there's a lot of things that would be underneath those if we broke them down to lower levels and although it's not in this particular example another common one that's often used especially in software is just bugs or defects so bugs defects on one hand obviously smaller is better and then features and functions larger is better more is better those are two examples of very high level pvas that can apply there they're relatively easier to measure but others are less so especially like design aesthetics how do we measure how well we're doing in design is more or less better what is more or less how do we quantify that often this is done with focus groups and getting feedback from stakeholders it's obviously not a perfect science but we do learn more and more about it all the time and more and more can be learned in the context of specific types of products this is certainly where marketing expertise is valuable and so we want to capture the best information we can get albeit imperfect and put it here into our decision making framework so that we can design and manage a project with it so there's a question from michael winoker who says excellent talk which agreed but asks whether you have any thoughts on the use of machine learning to estimate outcomes such as for drone endurance or the demand estimate example that you presented short answer is i'm intrigued by the possibilities but it has not been done yet to my knowledge now that's not to say it truly hasn't been done because a lot of organizations using this type of framework i would say the number of organizations not that many but a lot of what they're doing is proprietary so i i don't know if they've gone so far as to make such an application but i could definitely see the potential i think that's a fair answer a question from don w who asks what if any is the role of the critical path diagram in reducing risk in complex projects so could you restate the the name of the diagram critical the critical path diagram ah critical path good so critical path is essentially trying to tell us which activities if they got delayed would end up delaying the overall project that's good information to know for every project manager to distinguish them from the activities that could experience some delay hopefully without delaying the overall project those critical path activities may or may not have the highest direct effects on these pvas and revising the project capability distributions therefore we might think about critical activities more broadly than just their effect on a schedule we might think about them in terms of their effect on value which activities are critical because they have the highest leverage to change the project value realizing that project value is partly determined by the project's duration and therefore the time until the project can be delivered or provided to stakeholders but that's only one aspect of its value certainly some stakeholders are willing to pay more to get it faster others not sometimes there are penalties for being late sometimes not some stakeholders would prefer better performance even if they had to wait longer these are the types of questions that we can weigh against each other and determine value-adding trade-offs with this type of framework so yes i do agree critical activities should be defined more broadly than merely their effect on delaying a project and that critical activities should be the highest value leverage activities thank you so we've come up to one o'clock so usually we enter these after an hour people are welcome to email us at sdmcommunications mit.edu if they would like these slides for the talk or to be put in touch with tyson thank you again to everyone who attended the webinar again the presentation recording will be available online after this session and we hope you'll join us next time on may 25th for a webinar with gene ellison assistant professor of analytics at alfred university thank you again to tyson browning for joining us today and on behalf of the system design and management program thank you to our attendees for joining us you
2021-04-19