Create better Editor and game interfaces faster with UI Toolkit  | Unite 2023

well hello everyone thank you so much for being here today good to see you all my name is benoa I am a senior product manager at unity and very excited to be here today to give you an update on your y toolkit so we have a packed agenda I'll start with an update on the state of UI at unity and then we'll Dive Right into what's being released for Unity 20232 and then we'll wrap it up uh with some insight about what is currently in development and we should have time at the end for for some Q&A so back in 2021 when mechanist Tre were developing their Beaver team City Builder game called timberborn they got to a point where they asked themselves if they should be using youri toolkit for their project they used yuui in early development of the game which back then was the only solution Unity provided and supported and they were pushing its limits as timberborn grew more complex a lot of times building elaborate screens trying to get everything aligned together just bro layout and reusing assets what's challenging since logic is tightly coupled with the view and because the UI assets are not human readable they were continuously running into merge conflicts some of them can be resolved automatically but other times uh they're not and they have to recreate the prefabs from the ground up so they felt like they were being slowed down by the system so they wanted to release timberborn in Early Access on Steam and when you do so it's critical to be able to iterate fast and improve the game based on that early feedback and you probably know that city builders are quite complex and they wanted to make the game more approachable and and and just more intuitive so they could keep growing that that player base and you can achieve some some easy win by improving the user interface of the game so me can we looked at UI toolkits road map and decided to take a deep dive in the technology to better better understand its capabilities but also its limitations they thought it would be a good fit for their project um and was worth migrating the entire UI before they released the the game on team and this would also be you know a good opportunity to clean up some of the code architecture and pay off some tech depth along the way so a programmer and an artist sat down and recreated more than 60 screens in widget over the course of a 9 month period And soon after building UI went a lot faster so no longer have to deal with merge conflicts it allowed them to have better SE separation of concerns for their UI logic so now they can work on different parts of the UI simultaneously and also test components in isolations or resulting in fewer broken builds it's also easier to create more complex so they've added things like U uh scalable UI and hidden menus into their game but best of all because they improved the UI but also every aspect of their game they achieved overwhelmingly um positive rating on Steam and that's that's a great achievement when we asked bavel to share with us his experience with UI toolkit in torn he said that in yuii a work making and working prototype can take one two or 3 days depending on the complexity but with UI toolkit you can have a work working prototype in a matter of hours um and that makes a huge difference in productivity when especially when you build a lot of UI and you want to be moving fast in this highly competitive market so if you want to learn more about their experience in using your y toolkit we've made this case study um that gives you all the detail about it and make sure to visit their website and check out timberborn it's a beautiful game and it's a lot of fun so it worked for mechanist and and other games um but it will will it work for your project and I get this question quite often and we're not really the ones that are able to answer that for you uh all I can say is UI toolkit has been released and supported for many years now and even though it's not yet at it with yui it remains a viable alternative for many projects but we always and always recommend for you to do a thorough technical assessment uh and evaluate if the technology is a good fit for your project you can also keep an eye on our road map we're continuously releasing uh improvements performance optimization um and adding new features and speaking of new features now we're going to go over is being released in 23.2 so for this release uh we focus on improving the foundation of UI toolkit before we continue growing the feature set so that work is divided in three main categories we wanted to make it more efficient to build Dynamic user interfaces and make it easier to create custom controls that are also um more easier to configure we also felt some of the system were bit complicated and complex and difficult to grasp so we updated the evance system which is now easier to operate and behaves more consistently the new input system now has runtime support integrated and no longer requires Yi in your project to use and we we've also added a few uh new controls to the standard library and we improved the button so now it can display uh icon with or without text but for today though we're going to focus on the first category of features that will increase development velocity and we'll go for them by building a very simple yet very common uh UI found in many games just a player bar displaying a player's health and it's u a player's name and health in most cases you need UI to display Dynamic information reflecting these state of the game right you could be pulling uh data from a server or just updating game stat like a player score or player help points which will most likely change a lot throughout the session of the game so here we have our uxl defining the UI and how you would normally connect data to it is by using U query to retrieve each element by either their name or their type or a USS class and assign new value to them and in the case of a player's Health you also need to ensure it's continuously updated throughout the session of the game right and the feedback we heard is that this can be a bit tedious uh for for two main reasons first when you have many elements it requires a lot of boilerplate code to write and even though strings can have some advantages string references uh have some advantages it makes the contract between the UI and the logic a little bit weak like you only need to make one simple change in a uxl or USS and the whole thing breaks and those changes are usually challenging to track down and fixed so what if instead you could Define what kind of data to connect to the UI directly into the uxl so this is where you will declare a relationship between the model and the view and have a much better separation of concerns and at first glance you're like well it's a similar amount of code but what if all that new uxl code could be generated for you then all you need to do is connect your data source and everything will be uh resolved automatically so that's what we did we built a new data binding system uh it works for both runtime and editor you why you're not limited to use data types that are supported by serialized property any plain CP object can use can be used as a data source and you can Target multiple properties on the same element ranging from visual element attributes to style properties and it's fully extensible you can create your own binding types and attributes so many of you also find it important for artist and designer to work closer with programmers within a project so we made sure the UI Builder can be used to create and modify most aspects of data binding using visual workflows so you start by assigning a data source to a visual element which will then be made available to all of its uh all of its children and you can decide at any moment to um to override that data source in your hierarchy if you need to and this will take precedence over its parents so when setting a data source you have um two choices you can set a type which is just a type information there's no instance being created at this stage so no binding will be resolved however this is um this can inform the UI builder of what type is expected so later on when setting bindings it can provide assistance for you or alternatively you can set a script mple object assets from your project uh and this is an actual instance uh so all the bindings you define in there later on would be resolved live in the editor so this is very useful if you want to uh work with a representation of your data and play around with some values uh and see this changes live in the UI Builder and optionally you can set a data source path um if you want only a certain part of your model to be made available to your bindings and to set a binding it's as easy you can eag eager those e either go through the the triple dot menu or just right click on an attribute or a style property and you select add binding and you will be presented with this new dialogue that has everything you need uh to properly set a binding this is where you can override the data source if you want to and then you set the path of the property in the data source so if an object or type was specified as a data source in the previous step um the UI Builder can suggest a list of parameters to choose from it provides autoc completion and gives you useful information about um about them there's also an option for um specifying a binding Mo binding mode and you can um you have several options to choose from so you could want a an input f bu in the data source to be uh to remain in sync whenever there's a change in either of them or just have the input uh update the source um in our case we want the opposite right we want the source to uh update the UI and there's also an option for if you want the um the UI to be updated once and only once if you have for example uh localization strings then we'll go and click on ADD binding then you can notice the affordance in the inspector now informing us that the value is coming from a data binding it's also telling us The Binding is resolved because it's coming from a scriptable object but the binding could also be unresolved and that's a perfectly valid State the system is just informing us that even though the binding is properly configured it doesn't have any data source connected to it yet so it it's on us then to provide a data source at run time and when we do so everything would be resolved as expected we we're also using that new aord affordance for communicating when the value is coming from other sources such as uh inherited from a parent coming from a selector or a variable and the tool tip provides a very useful information about its origin so like I mentioned before any change to a connected script mple object uh data source is immediately reflected in the UI so and that's true for the UI Builder or the game view you can def also Define um your bindings only using CP code if this is what you prefer so if we recreate the same UI in code this is what it would look like there's an API for creating modifying and setting bindings just as you would do through uxl or the UI Builder to define a data source for runtime Bindings that can also be used for authoring or caliz serializing purposes you can use a common pattern like this one so the serialization goes through the field and The Binding goes through the property instead of the field so this allows you to do um validation change notific notification or any other operation and the third one is similar but for uh autogenerated properties so we didn't want the binding system to make um any assumptions on how you want to manage and the updates from your data source so by default it will pull the data and update the UI on every modification without actually knowing or asking is if anything has actually changed uh since the last update so this works well for very simple projects but doesn't scale efficiently um especially when you deals with a lot of bindings so to increase performance you can integrate uh both versioning and change tracking into a binding data source and so you can tell the system what to update and also when to update it these are optional features but if you see bindings taking a lot of resources and the profiler you may consider um implement ing them to minimize the processing overhead to integrate versioning you just implement the I data source view hash provider interface it enables The Binding system to skip updating certain binding objects if the source hasn't changed since the last update the interface also buffers changes um which is very useful when um the data changes frequently but you don't want necessarily the uh the UI to reflect those changes so in our example uh for the health of the player could change many many times during a single frame but that information doesn't need to be conveyed to the UI on each of those occasion so you buffer the changes then you call the commit changes method to inform uh to notify the binding system that data source has changed and now I want to update my UI and by default UI toolkit will update all bindings Associated to it data source so if you need more granularity you implement the I notifi bindable property change interface this allows you to instruct uh the system to only update certain properties and not all the bindings and of course if you want to achieve Optimal Performance you can Implement both inter both of those interfaces uh in your data source sometimes you need data to be converted between um the source and the target of the UI you might have data that is completely different uh from the UI you could be using an integer or in inum to do a lookup in a table and retrieve some texture to be displayed in the UI or in some other cases you might have um might want a different representation of the data for example if you have uh angle that you want to display in degrees instead of radians UI kit comes with a list of pre-made converters but you can also Supply your own so a simple example for our player Health UI would be to convert a numeric value through a string and a pen a percentage symbol to it you can Define individual converters or group converters like this one which can which are very useful to handle um permutation on the same conversion the method for adding a converter requires a delegate um to convert a source type to a Destination type and the delegate must be a raft delegate to enable U bidirectional usage and then all we need to do is register our converter group and once that's done they can be assigned to a binding using an API or directly from the UI Builder so if we go back to edit our binding and go into the advanced settings now we can assign our newly created converter and because the UI Builder knows is aware about the types of the source and the targets it can propose only the converters that are compatible and once assigned the value is sent to the UI converted as as expected right so as you see the value did not change right in in our data source but now the UI has a different representation of it so so far we've been looking at data bindings and it's possible to also have other kinds of bindings The Binding system comes with an extension class that provides what's needed to create for creating your own Uh custom bindings they can be useful if for example um you're dealing with custom types that are not natively supported by unity or when you don't need a data source so using an example uh oops no sorry it's using um Unity is also using custom bindings uh to create other types of binding as well one example is the serialized object data binding system for the editor UI which is still available we rebased it on on a custom binding so we can have a shared implementation for all binding systems and it's worth noting that since we have that now we can uh continually improve it uh so we're planning to have a deeper integration of that system with the new binding system and add supports like custom converters for example we're also developing a localized uh localization bindings for adding support for UI toolkit in the localization package uh so this is a perfect example of a binding having a different set of requirements than regular data bindings so an example of a custom binding implementation when The Binding system tries to resolve a binding it's going to call the update method and expects a binding result hopefully successful but let's say our localization system makes asynchronous operations to load string tables and then assign strings value to uh for a given local then if those operation are still ongoing when the update method gets called we just return a pending result to ask The Binding to try again later then when when it gets called again and everything is ready we can set the property on the element U to the translated text and we do so using the converter groups because we want to make sure all the registered converters um are taken into account when the value is assigned and this time we can res we can return our result as a success and at the bottom here we have an example of a localized string binding in the uh in a uxl document see it looks a lot like a regular binding but with its own specific attribute for uh fetching an entry into a localization table also notice it sits under a bindings element just like data bindings and you can have a mix of different bindings under that that element node and for those of you with a keen ey you may have noticed this new uxl object attribute and we're going to cover this in a moment so our UI is bound to a data source this displaying converted values functioning as expected but I think at this stage it's fair to assume that most player would expect an actual HB bar um instead of a textual representation so I'm going to go ahead and create a custom element for that so we'll we'll create a new custom H bar by extending the visual element class we only need two elements a background color and another another bar for displaying the actual representation of the player's health and whenever the player's Health changes we need to make sure to update and update the width right of the Players Bar accordingly and then to allow our custom element to be used um using uxl or the UI Builder we need to implement what we call the uxl factories and traits so a factory is what creates instances of an element it also takes care of generating all the inherited attribute of visual elements and then for each property you want to expose to uxl as an attribute you need to create a trait by specifying its type its name the name that you want to uh in the XML attribute and a default value and then finally during initialization you need to get the value for each of those attribute and assign them to your element so you told us that this requires also a lot of setup and more so because uh you can't really predict what's going to happen into that inet method and because all of that setup requires a lot of strings to be pared we're kind of limited on how we can optimize instantiation times so we improved how uxl gets serialized we wanted to make it faster and more intuitive to create custom elements and also while while also providing U The Familiar Unity features you rely on such as serialization uh property drawers decorators and so on so with ux XML serialization this is how you would expose UI elements to your XML so we're going to go over a few differences first to declare a custom control you now use a uxl element attribute which replace places the uxl factories each attribute now only requires a uxl attribute no more uxl trait and the attribute name is automatically derived from the property name although you can if you want specify a custom name you can do so as an argument in the attribute and the default values are directly assigned to the field and it requires the class to be marked as partial because we're using um some Source generator so when you co when your code compiles we detect all of those attributes and we add new code to your classes that handles the serialization so now we can replace our standard label with our new health bar and we just need to make sure that the data binding um reflects those changes we add a binding to the same Health property in the data source but now it targets the um the the value property of our H bar instead and that's pretty much all we need to do there's no need to change any code or modify the data source in any way now with that we can do even more so let's add the ability to customize the held bar so it changes color um when reaching a certain percentage for example it would start green when I the player reaches 60% Health it turns yellow and the red at 30% and for maximum flexibility we could have any number of those Health markers which could be of any color so a regular player could have three colors and and an MV bus maybe go through five of those so let's go ahead and create a new class to define those markers which will name Health marker and only needs a position and a color and then we add a new property in the hbar element that will hold a list of those markers so this all this is all fine but you to Kit has no idea how to handle the conversion of our heal markers into uxl so because when a system uh sees a uxl attribute it knows that the field or the property uh is linked to an attribute in ux XML and unity will automatically handle the conversion of values to and from uxl attribute strings as needed and this conversion is done through um the use of What's called the uxl attribute converter um so this is great because we can create our own custom converters to instruct the system how to go ahead with that conversion and to do so we create a new class that extends the uxl attribute converter and then we just need to override two methods that instructs how to go from a type to a string representation and vice versa and then at the bottom uh you can see what that representation would look like in ux XML we have our three markers separated by by commas with their positions and color and of course these can be picked up by the editor and they can be edited like any other attributes so this is great right we can extend custom data types by defining our own uxl attribute converter but um you can run into certain limitations as your data types become more complex or you're dealing with large lists of objects which would make the generated string uh either you know less readable or more complicated to manage so the notion of a uxl object uh which is essentially an element that can be a child of a visual element um has already been part of unity well in UI toolkit so if you have ever authored a multicolumn list view or a multicolumn Tre view you may have encountered them um they're essentially uh a like you would notice a columns element which holds a list of columns so we enhance this feature and made it available uh for defining our own uxl objects so here we're going to convert our health markers to use uxl objects instead so as you can see they look a lot like uxl elements we're using uxl attribute for attribute declaration and the uxl object attribute instead of using uxl element and just like uxl elements it also requires the class to be marked as partial also northworthy is instead of using the uxl attribute now um when you using a uxl object Fields um we introduce what's called a uxl object reference this allows us to specify a name um that will uh indicate that uxl object will be parented um to an element with that same name but it's optional if you don't specify one they will just be serialized at the root of the node so that will generate that so that's the uxl um with now our new health mark markers which is much more readable and scales a lot better right if we keep adding additional Health markers it's much more convenient to manage and of course you can edit uxl objects directly in the UI Builder so then the last thing we need to know uh to do is update the value property of the hbar we here we can do uh validation like an early exit if the value hasn't changed uh then we look for some Health markers that would match our new value and if we find one we assign its color um of the H bar background with the color of our help marker and lastly we notify the binding system that the value has changed and we can update uh the data source so there it is and now our H barar is able to you know shift different between different colors any number of colors and those markers can be authored directly in the UI Builder so now that we are using serialized Fields uh through this new uxl serialization we can support custom property drawers just like a scriptable object or a Model Behavior those are very very useful if you want to improve the usability of custom elements and custom types so let's go ahead and add some predefined property drawers to our custom element as you would expect they can be used to decorate types properties uxl elements and uxl objects so here we're going to use um a header decorator for making a dedicated section for our health settings in the inspector so they're easier to find and I'm a big fan of Sliders for changing values so we're going to add those to the um position property of our health markers uh we don't have time to go in depth on how to create custom property drawers I mean they they're not a new feature they've been around for a long time though there's a lot of existing material if you want to learn more about them I just wanted to let you know that they are um are Now supported by the UI Builder so let's add a final touch to our UI I found the color change between those marker to be a bit choppy uh it would be more pleasing if the shifting of colors was a bit smoother but this is a perfect use case for USS transitions which have been around since 2021 so again this is not a new feature um but they're great for for this kind of of UI they don't require us to specify a start value or an end value anytime the value changes it animates uh the transition between the existing value and the new one and so for this case we just want to set the background color to transition over4 seconds so again same UI now with more smoothness so it's one of those simple changes that can have a big impact on how the UI looks and feel okay so let's recap we looked at how UI toolkit now makes it easier to build uh and customize UI with the new data binding system uh makes it easier to connect live data to your UI which also works at runtime and the new new XML serialization streamlines the process of creating custom elements but also provides better control on how they are exposed um in the UI Builder so I hope you find these new features useful and improve your day-to-day as you work with UI toolkit and please let us let us know if you enjoy them or if you see anything that you would like to be improved about them right so we're going to spend the remaining time we have to look at what we are current currently working on so what has been released in 20232 concludes the second phase our uh road map now we're shifting our Focus to provide the features and capabilities to make UI told kit our best solution for creating you know immersive and dynamic UI for games but also XR experiences so we're not going to cover everything in the details but I wanted to highlight uh the most important ones with which were prioritized based on your feedback which we always appreciate so thank you um so we're working on a solution to animate complex screens like a startup screen or a result screen and that for that you need the ability to create key frame animations and sequence them together uh and even combine them with other kind of animations the second area of focus is being being able to create more sophisticated UI styling by using custom shaders so we can produce Advanced visual effects like blur glow drop shadow or any other kind of visual enhancement and thirdly we also noticing an increase in requests for making games available to more players um so we're working to add uh full multilanguage support which includes um right to left languages like Arab and Hebrew and like I mentioned earlier we also want to make um UI toolkit work with the new localization package so we're adding support for that and unity also recently released a new C API to interface with Native screen reader on mobile devices so we're going to build on top of it so UI to Kit um will produce UI that works with them out of the box and will add support for desktop as well and lastly you requested to have UI that can be rendered in World space so in our internal Dev build we're exposing this capability uh via the panel settings which now has a new render mode option so When selecting World space UI can now have 3D transformation applied to it and is fully part of the game scene the implementation is not using render textures or similar methods it's fully integrated into the render chain so this allows for textual sui to remain crisp and beautiful at any size or distances from the camera and of course all other UI tokit features works as as expected such as styling animation and input we're of course super excited to be working on this and can't wait to make that available and one last thing before we leave um we also have a new sample project in the asset store we've heard that it's not not obvious to uh how to get started with UI toolkit and especially how to create a code architecture that will uh scale well so this project shows how to use certain design patterns like the the uh with patterns with youi tool kit like the model view presenter um the state pattern and how to manage many many screens uh in a project and many other things so feel free to check it out all right that's all the information I have for you today so thank you very [Applause] much

2024-01-06 09:11

Other news