Wesley Cole: Cool. So, the recording started. Welcome to our Augur training. This is gonna be extra exciting, 'cause Will, who's doing our training, he can't hear us, but we can hear him. So, I guess that's the most important direction. But, anyway, this is a third training so, you had the ReEDS introduction Tuesday.
Yesterday, we did the outputs using the _____ _____ tool, and today, we're gonna be focusing on the Augur Module. This one is kind of a more in the weeds piece than the other ones were. So, the Augur Module is the portion of ReEDS that does the hourly calculations so, it's the one that does the – and after ReEDS does a solve, it does an hourly dispatch. It calculates a lot of variability and other metrics.
So, that's what Will's gonna be talking about. Will wrote almost all the code that's in Augur so, he's a good person for this so, he's quite the expert with all these things. So, we're excited to have him share things with you. He's planning on not taking the full two hours. Like he said, he's probably gonna go a little over an hour so, we'll see where he actually lands up.
But we'll see if he's ready. We've gotta communicate with him via chat, so – Will Frazier: I can hear you now, Wesley. I – Wesley Cole: Oh, you can hear us now? Oh, perfect. Will Frazier: I exited, and I called in and I have my phone audio so, yeah, all set.
Wesley Cole: Okay. Well, we're ready for you now, Will. Will Frazier: Cool. I'm just gonna get one thing set up here.
Sorry about the delay. Cool. And we are – has the recording started? Wesley Cole: Yep. It's already going. Will Frazier: Okay.
Awesome. All right. So, as Wesley has likely mentioned, my name's Will Frazier.
I have been responsible for most of Augur and the structure and things that reside inside it so, we're gonna do this training today to hopefully make it a little easier for people to perhaps dig into Augur and work out any bugs they have, or at least have a better understanding of what's going on in there and why we have it and how it works with ReEDS. So, first off, what is Augur? So, an Augur was a religious individual – a divine individual – in ancient Rome, and these people were thought to be able to determine omens and predict the future based on the behavior patterns of birds. So, these people would observe birds and their behavior patterns, and make judgements based off of those. So, since this module of ReEDS is attempting to see into the future what these different technologies and effects are gonna have within the optimization of ReEDS, we gave it this name, and we have a couple of submodules in Augur that we have named after birds. So, the Augur then interprets the results of these submodules and determines some parameters that get fed back into ReEDS to inform the next set of investment decisions in the next solve year. So, we have Osprey, which is a production-cost model that we run at hourly resolution, and we also have Condor, which is a storage price-taker model.
So, why is Augur needed? I'm sure that we've been through some of this within the past few days of ReEDS trainings, but just to provide a little bit of brief background. Our aggregate time slice representation in ReEDS is insufficient for representing some certain components of the power sector. It's left unaddressed, but what we can do is we can take a deeper look at some of these effects that are only resolved when you look at a higher temporal resolution, and then, we can use that to inform the model to make up for these insufficiencies. And the main pieces that we're concerned with here are one – the net load of the model. So, here, I have a plot of the hourly load and net load, and this is compared to the time slice level aggregations.
So, the dash lines, in each case, are the hourly profiles, and the solid lines are how these are cast into our time slice level representation. I'm gonna see if I can pull up a laser pointer here. Anyways, as you can see, at the high end – when load's really high, our net load tends to be much lower than the actual hourly profiles, and same on the other side. So, this causes some issues with assessing system reliability and accurately modeling prices and everything.
So, the energy prices are the other main piece that we try to address here. So, here, I'm showing our energy prices from the 2020 standard scenarios. This is the mid-case in the summer, and we – for each line, we have a time of day.
So, our time slices are aggregating to morning, afternoon, evening, and night. And the difference in energy prices between these times slices is generally quite small – often within 15 percent of each other – and 15 percent is what we assume for our round-trip efficiency losses for storage. So, in the absence of something else incentivizing the use of storage, the ReEDS model would never see any opportunity for storage to operate based on the energy prices that come out of ReEDSs at the time slice level. So, this is why we have Osprey and Condor, our models, and we do some interpretations off of that to inform some parameters that go back into ReEDS for the next solve year. So, what does Augur do? So, as I mentioned, Osprey provides an hourly dispatch for the entire year and so, for each of our 134 regions in ReEDS, they're all represented as a node, and all of our generation and load and hourly generation profiles from wind and PV go into this production cost model so that we can get this generation in transmission and storage dispatch. And we also get hourly energy prices out of this model as well.
And we interpret those profiles for prices, generation, transmission, and storage to inform ReEDS of what curtailment is and how that interacts with these other components and also, the value of storage operation based on these energy prices. And we also assess system reliability. And system reliability we actually assess using seven years of weather and load data and looking at the peak demand hours there. My neighbor is doing some construction. Wesley, is that noise too loud and annoying? Should I shut these windows? How are we doing there? Wesley Cole: I don't hear anything. Will Frazier: You don't hear it.
All right. Cool. I'm gonna plow forward. So, the ReEDS parameters that are populated by Augur – I have them grouped into these three main categories that were informing ReEDS. So, we have Curtailment, Storage, and Reliability. So, all of these parameters for curtailment – you know, they include curtailment from existing resources and marginal curtailment rates for investments in wind and PV and how curtailment interacts with storage and transmission, and I have something on the next slide that'll hopefully make this a little more clear.
We also have a couple of parameters specific to storage, including the hourly arbitrage value and the minimum operation that storage needs to meet in order to get that hourly arbitrage revenue – and also, a few things on system reliability. So, a little more simplified of how these actually fit into the ReEDS model. So, curtailment in ReEDS is equal to curt_old, which is the curtailment from existing resource, plus the marginal curtail rate for VRE investments times the amount of VRE investments we have. There's a curt_mingen parameter that informs how much curtailment changes based on any changes in mingen generation – so, from retirements or new additions of thermal generators that have a mingen level. Curt_stor informs how additional storage charging – how much curtailment can be recovered from additional storage charging. Curt_tran informs the same thing for transmission investments, and curt_prod is for H2 capacity investments.
And more nuance here. Curtailment reduction from transmission investments is limited by this really long parameter name. It's just the net load.
Basically, what this says is that if you're building transmission to send curtailment to a neighboring region, that neighboring region actually has to have a load that needs to be served in order for that curtailment to be used to go meet that load. The mingen level is informed by a parameter called ret_frac that comes out of Augur. We send the hourly arbitrage value for storage directly to the objective function. So, the objective function is subtracted by this hourly arbitrage value based on the energy prices from Osprey multiplied by the amount of storage investments that ReEDS determines is optimal. The variable STORAGE_IN must be equal to or greater to strorage_in_min, and storage_in_min is equal to the amount of storage charging that we see in Osprey in the hourly production cost model.
So, that storage is actually operating ReEDS as much as we see in Osprey, and this hourly arbitrage value is supposedly more accurately represented there, because we wouldn't want to give storage that value but then, not actually operate in ReEDS because the price differential and that the time slice resolution is not great enough to incentivize storage operation. And the last couple of things here – the firm capacity of VRE is the sum of the existing VRE – firm capacity cc_old plus the marginal capacity credit rate times the number of investments, and our firm capacity storage is limited by our sdbin_size – our storage bin size. And there's more information on that elsewhere.
Reach out to me if you're interested there. So, how Augur actually integrates with ReEDS. So, ReEDS – we typically run ReEDS with a myopic sequential solve, which means we solve the 2020 system and then, we move on. We move ahead in time – say to 2022.
We optimize investments there and so forth, but in between those ReEDS optimizations, there's this little file here in the ReEDS folder – the Augur data dump – which dumps the information from that investment year out into a GDX file, which is read in by Augur and Augur does this thing and writes out the 14 parameters that I just covered into a GDX file, which is ingested back into ReEDS, and those parameters are used to inform the next solve year's investment decisions. Taking a deeper look inside Augur to the moment, it has a similar sort of sequential structure. So, the first thing we do is we format the data from ReEDS so that it can be used in these other scripts down the stream. The next thing we do is we run Osprey. So, this performs our hourly dispatch of generation storage and transmission.
This is – so, we have a file for running the actual model and then, their file's gonna dump out of a bunch of CFC files that are gonna be ingested by Augur so those results can be interpreted later on. The next thing we do is we determine all of the parameters relevant to curtailment. So, this is looking at those Osprey results to calculate how much curtailment exists and how curtailment is gonna interact with marginal amounts of transmission, storage, and other things. From there, we go to Condor, where we run our price-taker model using the energy prices out of Osprey to determine the value of energy time-shifting for storage.
And finally, we determine the contributions of storage in VRE to system reliability looking at net load across seven years of data. So, I'm gonna do a quick aside here. Within Augur, we have – just give me one second to log on to our remote machine here. So, within Augur, we use object-oriented programming in Python. So, we have class structures to handle all of the data in a structured and organized way. So, I'm just gonna go through a quick example of how to use classes in Python and – Wesley Cole: Wait, Will.
Question. So, this is – so, just want you to highlight here – so, you – this object-oriented piece – a lot of it has come into the 2021 version that's not yet what – Will Frazier: Yes. Wesley Cole: Can you just talk real quick about that and how the version that's out there right now – the version you're talking about with object-oriented interact here? Will Frazier: Yeah.
Sure. So, Augur was first introduced in the 2020 version of ReEDS, which is the open-source version that's available right now. I'm new to this type of programming so, when we first wrote Augur a year ago, it was – we used Python largely as a scripting language and we had these long sequential scripts that had some suboptimal methods for handling data and for passing that data around. So, it was difficult to make changes in Augur in the 2020 version of ReEDS because these scripts were rather inflexible. So, for the 2020-2021 version of ReEDS, I went through, and this is – I went through and tried to implement a better structure for Augur using classes and object-oriented programming in Python.
This is the first large thing I have built using this type of programming so, if there is anyone that does work on the model who is savvy with this kind of thing and has any suggestions for improvement or that type of thing, once the open-source version of the 2021 ReEDS model is available, I welcome your input. But yeah, most of what I'm going to go through now at this point, where I talk a little more in depth about the code of Augur and how it works and how it interacts with ReEDS in more detail, I'm also gonna go through an example of opening up Augur and trying to debug something. This is all applicable to the 2020 version of ReEDS, which I think, Wesley, is gonna be released in October publicly? Wesley Cole: That – it comes out with standard scenarios, which is targeted for October.
Will Frazier: Okay. So, with that caveat, I'm gonna move forward with a quick example here about object-oriented programming. So, the basic gist of it is that you can create a class in Python, and a class in Python is going to represent a certain structure or certain set of a thing that you want to be able to represent in your code. So, my example here is I've created a nice example called "Pet". So, we have this class called "Pet" and the class, Pet, has attributes "name" and "tricks".
So, if we – I'm going to use this interactively. So, here's my class, Pet. We're going to create this class, and then, I can create a pet called "My_Pet" and it's going to be an instance of the class Pet, and I need to send it a name.
So, this init function – or method, rather – this init method of the class Pet is called automatically when you initiate an instance of the class Pet. Now, here, I'm creating an instance of the class Pet. This instance is called "My_Pet" and I'm going to pass it the name "Ted". So, now, I have an instance of the class Pet called "My_Pet". So, if I look at My_Pet.name, it's going to tell me that my pet's name is Ted.
So, this "self" thing is generally pretty confusing for most people when they're new to Python. What "self" means is that within a class – if you have a method of a class, the first argument of that method is always the instance of the class itself. So, when I call the class "Pet", the first argument that gets sent is actually "My_Pet" and the second argument is the name of my pet, which is Ted. And when this instance of Pet is initiated, it assigns the name that I gave it to be the name – so, this attribute of my pet, whose name is now Ted, and my pet, Ted, at this point, does not know any tricks, right? So, we can get a little more in depth with this and use the hierarchical structure of Python to create what's known as a subclass. So, the first subclass we're gonna talk about here is "Dog", right? And so, now, a subclass inherits all of the information, all of the properties, from its parent class.
So, this class, Dog, is a subclass of Pet. Pet is the parent class of Dog and so, now, Dog inherits this init function. So, anytime I create an instance of the class Dog, it's going to initialize it the same way it would for any pet, except I can specify some additional things for my dog here. For example, I can say, "The max number of tricks that any dog can learn is three." So, this is what's called a "class variable". This class variable applies to all instances of the class Dog, and I've also given Dog another method called "Learn_trick".
So, this method will teach the Dog a new trick. So, the arguments for this method are first self. So, whatever the instance of the class that's calling this method – that instance itself will be the first argument.
And then, the next argument will be the trick that you want to teach it. So, this method will check to see if the length of its attribute, Tricks, is longer than the max number of tricks it can know, and if it's now, it will append to that trick to its attribute, Tricks, which is, right now, an empty list. And if it is longer than that, they'll say, "The max number of tricks has been reached. Cannot learn anymore tricks." So, let me go ahead and grab my class Dog here, and we can – now, we're gonna say, "My_Pet = Dog + Ted".
So, now, I have My Pet.name – once again, is Ted, and let's just say My_Pet.learn_trick. The first trick we're gonna teach Ted is to sit. Right. And so, now, if we look at My_Pet.tricks, it's gonna return a list which now includes
"Sit", because we've added Sit to that attribute there. So, next trick I want to teach Ted – so, this is Ted. As you can see, Ted has some boundary issues, right? So, let's say it's early in the morning. You're gonna take Ted for a walk and you're trying to put on your shoes and Ted's really excited. He wants to go on a walk. He's excited to see you.
It's the morning so, he's all up in your business and so then, a really good important trick to teach Ted is gonna be to back up, right? So, let's teach Ted another trick. Learn_trick('back_up'). And how about shake? Shake is another fun trick. Shake.
So, now, Ted has a set of tricks that he knows – sit, backup, and shake. And if I try to teach Ted any additional tricks, it's not going to let me, because let's say, roll over. "The maximum number of tricks for Ted has been reached.
Ted cannot learn anymore tricks." So, that's a quick look at how different attributes and methods can work. I'm waiting for this top bar to go away so that I can move this away so I can get back to my presentation here. So, this is really useful because if there's another sort of pet that we want to include – let's say we want to include a cat in our file.
Now, we want to be able to model both dogs and cats. We want to do it efficiently. So, I'm gonna create another subclass of Pet called "Cat", but obviously, cats are very different than dogs so, I'm gonna have to redefine this "Learn_trick" method for Cats, but it'd different than Dogs – that when we call "Learn_trick" for any instance of the class Cat, it's just gonna say, "This cat has no interest in learning the trick. General mayhem will ensue, followed by incessant fur licking." And we've also changed the init function here so that whenever we initiate an instance of the class Cat, what it will do is this stupor method.
This if actually a native method of Python – a native method of the native class, Object, in Python. So, anytime you create a class in Python, it's actually a subclass of Python's native Object class, which has this super method, which what you can do with this, is any – when I call stupor init, it will actually call the – this method of the parent class. So, whenever any instance of Cat is created, first, it's gonna call this and it's gonna set these values for the attributes of Cat, and then, when we get here, it's gonna amend that, and it's gonna change the tricks for Cat, and the two tricks that can know in this case are to "lick fur incessantly" and "cause general mayhem". So, once again, if I go back here, and I can grab my class, Cat, and both are subclasses of the class Pet. What have I done here? Wesley Cole: Deleted the last line of your class, Dog.
Will Frazier: Oh, did I? Oh, I did. Thank you. Thank you, Wesley. So, let's create this class and now, I can say my cat is – let's call him Benjamin, and the cat – his tricks are lick fur incessantly and cause general mayhem. And I am not able to teach this cat any tricks, because Benjamin is not interested in sitting.
So, this – and once again, I'm just waiting for this to go away so I can move this out of the way and get back to the presentation. This allows us to use a hierarchical structure for similar types of data to organize things well, and then, we could – if we really want, we could create another subclass of the class Dog. Now, I have a class "Australian Shepherd" down here, which is just like a Dog. Australian Shepherds are just like any other dogs except they can learn more tricks.
So, I have changed the class variable max tricks to be eight for Australian Shepherds rather than three for any other kind of dog. So, a quick review here, 'cause I'm gonna be using these terms "Class", "Subclass", "Class Variable", "Attribute", "Method", and "Instance". So, the class – for example, a Pet – defines a common set of characteristics for something you want to represent.
A subclass will inherit all of the properties of the parent class unless they are changed or overwritten. A class variable, which in this example, I used "max_tricks" – defines a characteristic for all instances of a class, whereas an attribute defines a characteristic for a single instance of that class. So, if I were to create another Dog and name it Fido, Fido and Ted would both have max tricks equal to three, but their "name" attribute would be different where Ted's name is Ted and Fido's name if Fido. A method is basically just a function that can be performed by an instance of the class on that instance of that class, and an instance is just a single use case of a class. So, within Augur – so, in ReEDS, there is a ReEDS Augur folder, and within this folder, there's a utility folder, and this utility folder contains a lot of meat of what's actually happening in Augur. So, we have the "functions", "generatordata", "hourlyprofiles", "inputs" and "switchsettings".
So, in our "generatordata" file, we – this defines a class for all generator technology types. And, as we'll see later, all generator technology types are subclasses of the parent class for all generator technologies, which I believe is called "Tech_Types". And so, we use this hierarchical structure of generator types so that we can easily handle the data between these and define new technology types and data rather easily in case there's a new technology you wanted to add to ReEDS. Then, you could just come into Augur and create a new subclass rather easily and generator data, and it should propagate through the rest of Augur quite easily. And I'll talk about GEN_DATA and GEN_TECHS a little bit later, which are dictionaries that are defined in this script.
Similar thing in "hourlyprofiles". So, we handle all of our hourly data in this hourlyprofiles Python file, which, again, uses a hierarchical structure to define the different ways that our data is stored and manipulated and those types of things. So, we have "HOURLY_PROFILES" and "OSPREY_RESULTS" are defined here.
"switchsettings" defines a class called "switchsettings" and this class is the parent class for all classes that are used in Augur, and this allows for the switches to be seen by all classes so, logic based on whatever switches used to read can be applied easily and consistently across all of the data in Augur. "inputs" is another file which is used to read in all of the data and the organize in consistent fashion. So, I'll go through that a little bit later as well, and we also have "functions", which just houses additional functions that are used elsewhere, but not necessarily specific to any class. They're not class methods.
Okay. So, the data flow between ReEDS and Augur. So, the switches from ReEDS get into Augur via three places.
The first is the cases.csv file, the second is command line arguments, and the third is value defaults.csv. So, when you do a ReEDS run – and I'm sure that we have looked at the "cases" file previously – so, we have all these switches that are defined and go into ReEDS and define how certain thing are handled in ReEDS. Some of these are relevant in Augur; some of them are not, but for completeness, we send all of these switches to Augur. So, I'm gonna open this switchsettings.pi file first. So, here's our class, switchsettings.
It has a few class variables, and then, it has a few attributes, right? Its attributes are one – the GDX file where we're gonna get all the data from the previous read solve years. The next and previous years that were solved in ReEDS, the name of the scenario, and then, the last one here – the important one – is this switches attribute, which is a dictionary. And this dictionary will include all of the switches from this "cases" file, as well as the next year and previous year and scenario name, which are passed direction to our Augur via command-line arguments. And also, inside Augur, there is this additional value default csv file, and this just includes some default values that are used in Augur, 'cause we didn't want to hard code certain things. So, these are the three places where switches will come into Augur from – the cases file, the command line, and then, value default csv file. The data for Augur comes primarily from two places.
It comes from the ReEDS data GDX file. As I mentioned previously, after ReEDS solve year finishes, it's gonna call that D3 Augur data dump file, and it's gonna dump out all of the relevant data for Augur into a GDX file that can be read into Augur and used. Also, some of the inputs come from csv files inside the "inputs" case folder when you do a ReEDS run, and all of these are read in via this input.pi file in the utility folder. So, I'll pull that up really quick. Inputs.pi has this first class, "Augur_Input", which, again, is a subclass of the parent class, switchsettings.
So, the switches can be used when data is being read into Augur. And so, the primary method for this Augur input class is get data – so, we get data. It will read the data and perform a few things to make sure everything is consistent. For example, we drop all regions that are not included in your solve so, if you're only solving ERCOT, for example, and you read in data that has data for the entire US, it will drop all the regions that are not included in ERCOT. A couple of other relevant things.
Let's say you're running with the cooling water switch on and you're reading in data that doesn't have the cooling text defined for that. This will look and check that switch, and if that switch is activated, it will go to this other method that you can map the cooling water text on to whatever you're reading in. So, it's important for all of the data in Augur to be self-consistent, and so, that's why whenever anything is read into Augur, it all comes through this same class method, "Get data here" so that everything is self-consistent. And Augur doesn't break when you add anything new into it.
This ReEDS data method is defined in some of these subclasses, which depend on the actual file format of whatever your input is. So, if your input's coming from a csv file, it's gonna call the ReEDS csv function. If you're reading from the GDX file, it's gonna use the GDX _____ class to read that in. All of these inputs are defined in this dictionary here at the bottom.
So, this dictionary inputs – houses all of the inputs that get read into Augur and gets used, and each key in this dictionary points to an instance that gets created of whatever class that the certain input belongs in. So, "available capacity" comes from ReEDS, which is in the GDX file that's been out by D3 Augur data dump, and a couple of these – these are the arguments that are needed whenever you initiate an instance of the GDX input class, which is a subclass of this Augur input class. So, if we go to the GDX input class, we'll see that when we initiate anything, it needs a key. So, the key is gonna point to the parameter and the GDX file, and then, this call name is the list of names for the different columns in this file.
And there's also this attribute DF, which we initialized to be none, and so, when we read in the data, the first thing it does when it reads the data is it checks to see if the data frame is none, and if it is, then it will grab the data from that GDX file and return that data. And so, if, for example, we wanted to build capacity and we call inputs avail cap, and it's these that – it's attribute DF is none, then it's gonna grab the data and it's gonna set the attribute self.ds equal to the data when it grabs it. Then, later on in Augur, if we call that again – if we call "inputs" "avail cap", get data one more time, this will see that we've already grabbed the data once so, we don't need to read it in again, and this saves some time for us when we're reading in large amounts of data.
So, it can seem a little cumbersome to use something like this so that whenever you need to add a new input into Augur, you have to come in here to the "inputs" data frame and add it in correctly and assign it to the proper class, but using this allows Augur to be quite flexible and easy to augment and integrate as ReEDS changes, because we're always updating how ReEDS works and the different capabilities that ReEDS has, and so, in order to keep Augur up to date with that, it's important to have a nice, consistent structure that we can use – that the two are easy to keep in sync. So, I'm gonna go back to my presentation here as soon as this top bar goes away and lets me move this screen. Just give me one second. There we go. Okay. So, we've gone through switch settings and inputs.
Next, we're gonna talk a little bit more about these dictionaries. So, we went through the "inputs" dictionary, which is a dictionary, and each key in that dictionary points to an instance of a class that defines a specific input property. We have a set of five of these dictionaries that are used commonly throughout Augur.
All these dictionaries are written as all caps. Whenever you see something written in all caps, you know it's one of these main dictionaries that are used. GEN_TECHS is a dictionary and each key in the dictionary GEN_TECHS points to an instance of a class for each technology group, and attributes of these instances are data frames defining certain properties of that generator. And methods of each of these subclasses define the different ways of how we're gonna set the data up.
GEN_DATA is a dictionary of data frames, which basically just collects all of the attributes from each of the different technologies and packages them into a single data frame that can be used. HOURLY_PROFILES has – each key defines a different profile that's used. So, we have load profiles. We have VRE capacity factor profiles. We have other things. OSPREY_RESULTS houses all the profiles that come out of Osprey so, we have transmission profiles for each region.
We have net transmission. We have generation profiles – hourly generation profiles, hourly price profiles, that type of thing are included in this dictionary. We'll get into all this in a little more detail – not a ton of detail, but at least to give people a general overview of how things work. So, as I mentioned, GEN_TECHS – so, here's our GEN_TECHS dictionary, which is defined in utility/generator.pi – generatordata.pi – and each key for GEN_TECHS defines a different class.
So, we have this parent class for all generators' tech data, and tech data has a class variable called "mingen size" and this is defined because generators smaller than this are not considered in Augur or in Osprey. So, anything less than five megawatts is dropped and not included in Osprey. So, this is our class variable for tech data. We also have all of these attributes for every instance of the class, and these attributes serve the different properties that are relevant to these generators in Augur. So, we have available capacity; daily energy budget – which – go back – "daily energy budget" is used for dispatchable hydro and certain other hybrid energy technologies in Osprey – energy capacity for storage; generator costs; hourly capacity factors for wind and PV; max capacity mingen – those sorts of things. So, if I come over here and let's go to "Generator Data" and just look at a couple of quick examples.
So, down here at the bottom is where we have GEN_TECHS defined – and I'm gonna zoom in a little bit 'cause I think this may be kind of small on the screen share – once again, generator data defines a class instance for each of these different technology types. So, if we go back up to "Tech Data" here at the top, we see our definition of the Tech Data class, which again is a subset of switchsettings so that we can have access to all the switches and reads whenever we're formatting the data for any of these technologies. So, let's look at, for example, "Format gen cost".
So, generation cost is the fuel cost plus the O&M, and this applies to nearly all generators. So, we have this defined at the top here within the tech data class so that the first thing is does is it calls a method of whatever object is being manipulated here. "Calc fuel cost" – this is gonna call a different method here and calculate the fuel cost, which gets the fuel price, and then, multiplies that by the heat rate.
So, we get the fuel price, we get the heat rate, and multiply the two together. "Get fuel price" is its own method because some of the technologies in ReEDS have their fuel prices determined in different ways. For example, we have a supply curve for natural gas, and the price of natural gas is determined endogenously by ReEDS based on natural gas demand, whereas for things like coal – coal and nuclear – we have the fixed price. So, if we – this method will grab the fuel price that's specific to that specific class. _____ fuel cost, then we get the VOM costs, and we filter for the technology and add those two together.
So, this filter tech is perhaps another key one that's good to go through here. So, what this is gonna do is for any – once an input is read in, this will filter on that technology's specific – that technology class's tech definition. So, I use "technology" like, 10 times in a row so, I'll try and be a little more explicit here and explain that better. So, the tech class has this attribute "Tech_Key" and the Tech_Key is the one argument that's required whenever you're initiating an instance of the Tech_Data class. So, if you notice, down here at the bottom, when we created all of these instance of that class – so, for "battery", for example, when we set battery – when we point battery to an instance of the storage data class and pass it the Tech_Key battery, it knows that one – the gen_tech battery is an instance of the Storage_Data class, and also, its Tech_Key's battery.
So, when it filters in the other data in when we're manipulating this particular object, it will filter for all of the technologies that are included in "battery". Maybe we can go into that a little more later. So, basically, there's a lot of data manipulation here set up in such a way that things are generic here at the top, and for different technologies that have specific – that need specific methods – for example, natural gas – if we go to "Natural Gas Data", it has – it's a subclass of Tech_Data, and it just has a couple of properties that it needs to handle specific to natural gas. So, we have changed the "Get Fuel Price" method to grab the natural gas price from our "inputs" dictionary, and also, the "calc fuel cost" is slightly different for natural gas relative to other technologies. So, that's defined here. I'm gonna go back to my presentation here.
So, moving forward just a little more so in A_prep_data, which, as I mentioned, is the first script that is called when we run Augur so that it can format all the data and send it to Osprey and the other scripts downstream that use the data, this little block of code here really encompasses like, the meat of what's going on there. So, for every tech in GEN_TECHS – so, this is for each key in GEN_TECHS – and then, for each key in GEN_DATA – so, as I mentioned, GEN_DATA is a dictionary with a data frame that includes the properties for every technology instead of just a single gen tech. We're going to get the data and append it to GEN_DATA in this sense.
So, when we call GEN_TECHS "tech", when we call the method prep "data" and give it the key – which is the different data that we're trying to format – it's going to go to this "Prep Data Method" in "Generator Data". This is the method of the Tech_Data class. And depending on the key, it's gonna call the different methods that the data can be formatted properly and appended to whatever dictionary here is relevant. And I think, at this point, it may be most useful to just walk through Augur the way I usually do it. So, when I am using Augur, I typically use an IDE.
My preferred one is Spyder, as we've seen before, where we had the Pets example. So, if I have a ReEDS run, I can pull the Augur.pi file out of my "Run" folder and I'm gonna drag it here into Spyder, and then, I'm going to set my directory to be in that "Run" folder. Actually, I believe we used July 28. That's not the one we wanted. We want this Augur test one.
Excuse me for just one second. Okay. So, we have Augur.pi file, which is the driving script for all of Augur. I set my working directory to be that "Run" folder, this way, when I get to the first bit here and I start importing things, it knows to go into this ReEDS Augur folder, and it's gonna grab from A_prep_data. It's gonna grab everything that we need.
And it's gonna go into "utility" folder and grab "switchsettings" and it's also gonna grab all of these dictionaries that I mentioned before. So, this is the main function in Augur. Three arguments are passed to it – the next year, the previous year in this scenario. There's some logging information here and then, we call the function "ReEDS Augur" which exists here at the top.
So, in my "ReEDS Augur" function, this is the actual procedure of most of what's happening in Augur – or all of what's happening in Augur. So, here – there's a little note here to debug, un-comment these lines, and update _____ as well as the run path. So, I want to run Augur, and I want to run it for my next year's 2024.
My previous year's gonna be 2022. This is gonna grab the base name from my working directory, which is the scenario name, and the first thing that's gonna happen is we're gonna call the "set_switches" method from this "switchsettings" class that we talked about earlier. So, we're gonna define all the switches, first off. So, once these switches are defined, then, we can move forward and start going through Augur.
So, the first thing we do is we prep the data for Osprey. This includes hourly profiles and generator data. So, this "prep_data" function was imported from our "A_prep_data" file so, I'll pull that up here so we can take a look at that. The first thing it's gonna do – it's gonna find a marginal step size for wind and PV. We need a marginal step size for wind and PV to determine things like the marginal capacity credit, and that is based on the amount of investments that occurred in ReEDS during the previous solve year.
We're gonna go through this "Format Profiles" method for every profile that exists in our HOURLY_PROFILES dictionary, which is in our "HOURLY_PROFILES" Python file. So, just like any of those other dictionaries, we have HOURLY_PROFILES, and for each key in HOURLY_PROFILES, we have an instance of a class here in this script. So, I'm going to go ahead and go through A_prep_data here a little bit slowly.
So, here's our "prep data" function, which is called – if I had the Augur file to get these ReEDS inputs – and also, in "prep data", at the end of it, we're gonna write out some CSE files as well as the GDX files that are gonna be read into Osprey. So, the first thing we can do is we can set our marginal VRE step size. We're gonna format our HOURLY_PROFILES. So, this is also what I would be doing if I were to – if there is an error in Augur. For example, I happen to run this test yesterday because I knew an error was gonna occur – if an error does occur in Augur, in this "List Files" folder, it's gonna spit out an "Augur Errors" file for the next solver year, and it's gonna tell you – here, it's gonna give you a trace back of what the error was. Typically, not a lot in Augur is changing other than the way data is read in and formatted so that it's constantly being updated with ReEDS.
So, if there is an error in Augur, it's typically gonna be on this front end, and I will go through how I would go about finding this error and debugging this. So, we see that this is – in this "prep_data" function, there's another function that's being called down here – "format_prog_load" and then, inside "format_prog_load", we're getting some sort of error. So, we're gonna find out what that is as we step through A_prep_data here.
So, I have – we'll just do a couple more class examples here. So, when I look at GEN_TECHs, right, GEN_TECHS is a dictionary so, it has a key for each of these technology types, and each of these is pointing to utility generator data, storage data. So, "battery" is an instance of this "storage data" class, as well as the rest of these. So, if I look at GEN_TECHS battery, now, this is gonna be just my instance of that storage class.
And if I remember that my attributes for "battery" are the different properties that "battery" can have – for example, "max_cap" when this class was initiated, the attribute "max_cap" was set to "none", right? But once I run this bit of code to populate all of the data, then we're gonna be able to see what the – we're gonna see that the attributes for all of these classes are then gonna be populated with data frames. So, now, when I go to GEN_TECHS battery, max_cap, now, I have a data frame with all of my capacity from ReEDS, and you can also look at "energy cap", which is gonna be in megawatt-hours – the energy capacity for each of these same generators. So, just for a quick example, if I look at my storage data class, storage data is gonna have a "format energy cap" method, and what this "format energy cap" method's gonna do is it's gonna grab the max capacity and then, it's gonna get the storage duration and then, multiply the next _____ by the duration and set that to be a data frame at the energy cap method. So, that's happening for all of the data in Augur, all of the generation costs, capacities, capacity factors, that type of thing.
There's some additional data formatting done down here for Osprey. We're not gonna go through all this 'cause there's quite a bit happening here, but we are gonna notice, once I get to this line, then we're gonna run into our bug, right? So, this is our key error that we saw earlier in our error file. So, this is in format_prog_load so, I'm going to come into my "ReEDS Augur" folder and go into my "utility" folder. Functions.pi is where format_prog_load lives. So, let's find – so, first, before we do anything, let's import everything at the top of this script and now, we're going to look for "format_prog_cap." We're gonna scroll down to "format_prog_load".
So, we're reading in a few inputs here at the top and then, this – I'm gonna check this _____ statement and I see that it is true so, I know we're coming inside this block of text here on this conditional, and I believe it was this line here where the error is [Inaudibly whispers to self]. Oops. So, get_prop is actually a function here. That's why it's not defined.
So, I'm just gonna define all these very quickly so I can get back to this. I'm gonna hit "prog_load". I believe this is the line where we were having the errors.
I'm gonna do everything up to that point. Okay. Let's keep marching through. So, here's our error – key error's megawatt.
So, we're trying to divide this megawatt-hour column and _____ by the hour column. And I see that the issue is that the megawatt-hour column doesn't exist, right? And that's because there's no data here. We've run into a few errors with Augur early on here where empty data frames tend to cause issues so, what I'm gonna do to get around this one, is "cons_flex" is coming from this "consumption" data frame so, I'm actually gonna move this outside of the block here and I'm gonna add another conditional and say, "Not consumption.empty" and – so, now, we're only going into this block of code if there's actually data that exists in "consumption".
So, I'm just gonna run this entire function to see – just to double-check, to make sure that my fix worked, and my fix did work. Let's check on "cons_flex" and "cons_no_flex" just to make sure. So, "cons_flex" has our day, our region, and our megawatt-hour empty, and then, "cons_no_flex" is actually gonna be a set of profiles for each region where we have our inflexible H2 demand is actually what this is. So, I would save this change and then – now, I've fixed my bug in Augur so, I could come back and either restart the console here and just run it again just to make sure it works and then, go through pushing a change in whatever else you would need to do to get Augur working. So, I realize that I have covered a ton of material and some people may or may not be familiar with classes. If you are, you know, hopefully, this is at least a little bit helpful.
If you're not, hopefully, this wasn't too much information to process. And I think, at this point, I will open it up for questions. If anybody has anything, I'm happy to go through anything. I do think we have some time left in the webinar here. And, if not, I'm happy to end it early and people can just refer to this recording.
Hopefully, it's helpful later on. Wesley Cole: So, yeah, there's just a quick question in chat, Will. Can you explain the notation – the "And" followed by the slash when you had the multiple lines for the _____ statement? Will Frazier: Yeah.
Sure. So, this backslash in Python just means I'm gonna continue my line on the next – I'm gonna continue this line of code on the next line here. And all the "and" does is it makes this conditional depend not just on this condition, but also on this condition. And since I have another "and" here, it's also on this condition, right? So, I can say, you know, "If true, print 'yes'" and so, it's gonna print "Yes."
And then, if I say, "If true and false, print 'yes'" and so now, since I have the false condition, in addition to my true condition, it's not going to print it. Similarly, I could use "or" here. "If true or false". One of these two is true and so, then the condition is true and it's gonna perform the operation. Wesley Cole: Okay.
Thanks, Will. I think that's everything. So, yes, so, this recording's gonna be available so you can come back and check on it.
And, just as a reminder, you can follow up with – on the discussion page, too, if you have questions or comments there and then, let's see – okay. A couple of questions in chat. I'll refer these to you here real quick. Do you want to talk a little bit about Osprey? And you didn't open up the Osprey code – [Crosstalk] Will Frazier: No, I can.
If we – Wesley Cole: – which maybe just a high-level piece. So, Osprey is the portion of the code that does the hourly dispatch so, it's in GAMS rather than Python so Python calls the GAMS code that does the dispatch 'cause it's solving an optimization problem to do the dispatch. So, I don't know that we want to go into much more detail. It may be helpful, Will, just to show where it's at in there. Will Frazier: Yeah.
I will show, just very quickly – so, I'll share my screen and I'll just keep sharing. Wesley Cole: And you're welcome to follow up if you have other questions beyond there. Will Frazier: Okay. Male: Thank you.
Will Frazier: So, right, yeah, once we prepared the data, which includes – as I mentioned, we're gonna write out a bunch of CSE files here with available capacity, daily energy budget, netload, these types of things, and then, all of these properties that are needed in Osprey also go into a GDX file, which the "ReEDS Augur" folder has this "Augur Data" folder, and this has the ReEDS data, which is used in Augur. And then, it would also write out all of the additional CSE files, and the GDX files that are gonna be used in Osprey are gonna be read out here. And then, the next thing that happens in this main Augur.pi script is that we're gonna run Osprey. So, here's where we call the "B1 Osprey GAMS" file. We give it a few switches, which change the operation based on how it running ReEDS, and then, the next thing is we're going to call this GDX dump file and all this GDX dump file is gonna do is just gonna grab the results from Osprey and write them out as CSE files.
And maybe that's about as far as we want to get into Osprey right now. Wesley Cole: Can _____ your question about ReEDS Augur, dealing with flexible load – that, as you go through and look at the Osprey constraints, you'll see some of that where we'll have daily energy budgets for different resource types and the ReEDS will dispatch those flexibly. So, that's one way that's captured there. If you want to talk through more details, [Inaudible]. Will Frazier: Yeah.
Flexible load is also hints in our HOURLY_PROFILES file. We have a class for load profile, subclass of HOURLY_PROFILE, and one of the methods of this load profile class is "adjust_flex_load". So, this is gonna adjust the hourly load profiles in Augur based on the optimal load flexibility determined by ReEDS – if you're using ESF flexibility in ReEDS. Wesley Cole: Okay. Cool. Well, thanks, everybody, and again, Ed.
Feel free to follow up with us or to use the ReEDS discussion page on the GitHub to add more pieces. Or, if you find something that's broken or wrong, you can certainly submit an issue. So, thanks, Will, for walking us through this. Much appreciated. Will Frazier: Yeah. Of course.
Hopefully, it wasn't too much all at once. Hopefully, some people benefit from this. Wesley Cole: Cool. Well, thanks, everybody. Have a great day.
2021-08-17