Lighting and AI techniques for your 2D game | Unite 2023
- Hello folks, my glorious 2D people. So nice to see you again. It's been a while since we've had a Unite Amsterdam, and I'm happy to be back. So let's get started.
We're gonna talk about lights effects, ai, hot new techniques for your 2D game. My name is Russ Scammel. I am the product manager at Unity for 2D as well as generative ai.
Now, I want to do a special call out here. A lot of the amazing stuff that we're gonna see in this presentation is the work of the tech content marketing team. These folks are pivotal in the creation of all the eBooks and samples that share so much of that useful production wisdom by experts, four experts.
Okay, so today I'm gonna talk about a host of new techniques and technologies that you can apply to your 2D game productions. We'll be looking at dynamic lighting in two DVFX graph, or 2D projects, 2D improvements in the next version of Unity, and a special sneak peek to some of the work that we're doing to deeply integrate AI powered Sprite creation into the Unity Editor. We're gonna see a lot of this in the context of our latest sample Happy Harvest.
Now, this is a great example of a genre that lots of game developers are working on, and it's full of wonderful art and features that need to come together to make a compelling game. Let's watch a trailer of Happy Harvest. Now. All right, that's a glorious project, right? There's so many things that come together, so many different techniques, and we're gonna jump into a lot of those. We're gonna see how they were put together so that you can use some of those techniques in your projects as well. Okay, to begin with, let's talk about dynamic lighting in 2D.
Now we're gonna cover a couple of features and some techniques that are useful to achieve believable runtime lighting in 2D, and we're gonna look at secondary textures and light blending styles as part of this exploration. Now, an important point I wanna make is how the development of Happy Harvest really considered this, considered the fact that it was gonna use two delight from the very beginning. This is the kind of decision that you want to make early in your production, or possibly consider as part of pre-production, since it has a significant impact on your art pipeline. For example, every 2D asset in the game can respond to lighting, and so they all use secondary textures. This includes the characters, tile maps, and of course, just plain old sprites.
So here's an example of how every texture or every 2D asset is made up of different textures, right? Of the different maps that it needs. So this one is the market stall, and we can see here on the left we have the main Sprite, right? There's very little, or maybe even no light information or shadow information in the Sprite itself. I guess the one exception here is that there's a little bit of ambient occlusion where it makes sense. We have a normal map of course, and in this case it's used by the two delights and it contains all the surface direction information or the apparent surface direction information. And then finally on the right we have the mask map.
Now, the way we use this in Happy Harvest is to basically define where we want the highlights to be in the sprites when we're doing lighting. Okay? So there's a lot of techniques that come together to build all of these things. And if you wanna learn more about the techniques for creating, especially the normal maps and the mask maps, don't forget to get our ebook on 2D game art, animation and lighting for artists. You can also grab the Dragon crashes and Happy Harvest demos.
These are full of examples of implementations of these techniques, and you can grab those from the asset store. Okay? The next feature I want to talk about that we need to understand is the light blend styles. Now, in every project in, in every 2D renderer, you can set up four light blend styles. Every 2D light in your scene uses one of these, right? It needs to choose a blend style. Now we can see the default on the left, which is additive, so fairly ordinary blend.
And then in the middle you can see a multiply style if you want a more modulated result. And finally, on the right subtractive or like negative lighting. So this is when you wanna remove lighting from a Sprite.
Now, this negative lighting effect can be good for creating kind of false shadows or otherwise control those dark areas in your level with a pretty high level of precision. Now, looking back at those mask maps that we saw earlier, these blend styles can also work with the mask maps that we mentioned earlier to mask which parts of the stripes actually get affected by lighting and in what way. Now, here again, if we look at the top, we can see you have different options for the mask texture channel. So if you choose none, we just ignore the mask and we light everything, right? We don't, we don't modulate that lighting at all. If you choose R-G-B-O-A, then we light the Sprite in the mask area. In the mask map area.
Or if you choose one minus R-G-B-O-A, then the Sprite is lit in the negative area of the mask ask. Okay? Switching gears slightly. In this project, we use a couple of simple techniques to control the large ambient or the large environmental light that we want throughout our levels. So first and foremost, your your, the thing that you can use is a global 2D light.
And this can affect the entire scene, right? This makes it easy to affect the mood of the whole world, right? So maybe a wizard has cast a poison spell over the town, and so everything should be a shade of green, right? Or in this case, maybe there's a battle nearby. There are dragons flying in the sky, and you want to create that apparent effect on the level by just controlling a simple thing like global light. On the right, we're seeing sunlight as simulated by a very large spotlight. So this is how we do it in happy harvest. Now, this is attached to the camera and it rotates via a script. And this is to simulate the motion of the sun over the course of the day.
Now, again, this is a very stylized game, so we're not going for realism, but we're going for a look, right? We're trying to develop a look. And so we're using the features in very creative ways. And so we're gonna see more of that as we look at more of the more of the examples. So this is part of the day night cycle that we see in the demo Here on the left, we can see normal maps. Now these are being used in the bushes to create an illusion of volume.
And on the right, we're using the multiply blend style to darken areas affected by a light. Again, this is this thing that we see on the right here is not a shadow, it's a 2D shape light that's being used to simulate a shadow caster, right? And we're doing this because we wanna maximize the creative control on the look and shape of the shadow, again, because we're being stylistic. And we'll discuss a more advanced version of this effect a little bit further on in the talk. Okay? So that's a good segue into shadows. Now, when it comes to shadows, a really old technique, an oldie, but a goodie is the blob shadow technique.
So here we've got like a blurry circular shape that is placed between the object and its environment, and we stretch it or squash it. And this is a good standin for ambient occlusion between an object and the ground that it's on. Now, of course, we can also use the standard shadow caster component that, that we provide when we need projected shadows, right? So we can see here this is a pretty, it looks pretty impressive in the game when the character has a shadow caster on him, it's on different parts of his body.
And as he walks past the street lamps, you can see that the shadow gets projected right as he occludes, as he occludes that shape. Okay? So this is where it, where it gets really interesting. So there are some techniques that we use for some of the larger objects in the scene, and we got quite creative here. The concept of the blob shadow that we talked about earlier gets extended when we talk about the trees. What we do here is we use a Sprite, a blob shadow Sprite, and we stretch it and rotate it based on the time of day. So this has been integrated with that time of day, that day night cycle system that we have.
And that creates a soft kind of cast for the trees. You can see there's, there's no hard edges to those shadows. Now, the shadow for the cabin presents a especially tricky problem, right? So if you think about it, we're emulating a 3D object there. And so the projected shape of the cabin shadow changes over the course of the day.
Now, because of this, the team came up with this idea of using several freeform lights at different times of the day, right? So you have one for the morning, one for the afternoon, and one for the evening. And then the final piece that comes bring ties it all together is an interpolator. And this thing tweens the position of the light vertices over time, effectively animating the freeform shadow and simulating the changing shape of the shadow of a 3D object over time. Okay? Let's talk about the special problem of using dynamic lights to make a game that has objects that are fairly static or characters that are in fixed positions more interesting. We'll discuss that in the context of the upcoming sample project, which is a match three puzzle game with modern gameplay mechanics.
Now, when a game has a lot of moving characters, lighting from static lights is sufficient to make them look really good, right? So if we look at the stuff on the left, the lights are static, but the characters are moving around. So in games with this where, where the objects are basically still like a lot of match three games, there's not much motion apart from the tiles moving. It's important to make the tile to make the lights more dynamic by moving them around to create the same level of appeal. Now, we can see this clearly in the contrast between the more dynamic level here on the left and the static one on the right. Now in many 2D puzzle and card games, lighting and shadows are pre-baked into the art, right? As we can see in the pre-rendered stuff on the left, it's baked in, or sometimes developers choose to make the game in 3D, right? While retaining the gameplay in in a plane, basically.
But there is still more that you can do while remaining in 2D. So let's, let's break this up, right? Let's look at the stages. So here we can see that we have an example of a matching game, right? Again, a pattern matcher.
And typically the sprites themselves, even though there's a little bit of lighting in them, they're fairly flat and right, so here we can see some of the early dynamism gets added through bit of VFX, in this case a placeholder particle system, and some of the motion of the sprites. Okay? The next layer you could do there is to add a little bit of 2D lighting, dynamic lighting here, the board on the left uses a standard 2D light. You can see it by the way, it affects all of the tiles together. And the board on the right uses a slightly more advanced effect by implementing something called a custom Sprite lit shader. We're gonna dig into that one later. Something that you can do if you're using URP and if you are, if you're using a shader graph.
Now, finally, what you can do over the top of this is you can really put a cherry on the top of everything by using a full screen effect. In this case, we can see that we've got a bit of bloom on the level, a bit of post-processing that emphasizes the lighting even more. And the shader that we see here is something that uses the camera sorting layer texture. Now that simulates this rippling shockwave that passes through the entire board.
And it's this kind of effect that really separates your game from the others, right? It's the visual impact that your players are gonna see and remember when they play these kind of games. And again, if we take a look at the textures that make up each of these sprites, we can see a familiar pattern, right? We've got a main Sprite or diffuse color, normal map again, and a mask map. And for these ones, we have one additional layer where we basically take the Sprite, we take the, the mask version of it, right? In this case, a white mask against black. And then we blur it a little bit and we draw this behind the tiles again, to simulate a shadow, right? This is the same technique. We're using blobs over and over again.
Okay, let's dig a little bit into that shader that we saw earlier. That was the one on the right earlier where there was some custom lighting per tile. Now, in shader graph, you have an option to create a Sprite custom lit shader.
This is for more advanced effects, especially when you want to control how 2D lighting is sampled and processed. So to sample the light texture, we use the 2D light texture node in shader graph. Something to know here is that 2D light textures are essentially render textures that are created by the 2D renderer.
And they contain the visible lights in the scene for each of those, for each of those blend styles. So there are up to four textures each representing the, the blend styles that you have defined in the 2D renderer as you can see them here. And the output of this node is basically the same as the output of a texture 2D, which means you feed it into a texture sampler, and then you can use it as you would anywhere else. So he, what, what this means is that you can capture that lighting in the scene, process it, play with it however you want, and then finally you can plug it into a custom Sprite lit node at the very end. And so you can do really interesting effects with, with the lights in your scene and break all sorts of rules of reality on how lighting should work.
So why did we use a Sprite custom lit as a substitute for scene lighting in this game? So we wanted to control the lighting on each puzzle piece. And really the Sprite custom lit meant that we could move the light position data into the shader and eliminate the need for an actual light object in the scene as you see above. So it keeps the scene nice and neat. So the scene above has a light for each tile or two lights for each tile. And here, all of that light position data is actually just defined in the shader.
It encapsulates this per object lighting. So you get better isolation and editing at scale. Like you wouldn't really want to edit a scene with so many floating lights in it. It's a little bit more fragile. And basically where possible you get improved batching, right? Improved performance when batching.
The other thing I wanna dig into is there's a really nice glare effect or a shine, like a reflection effect on the, on the gem set. How do we do that? So here again, it's a, it's a glare effect on each of the gems. And to do this, we sample the normal map.
So this is something else you can do. You can sample the normal map, gather information about the direction of each pixel, and then we use the very useful dot product node to apply simulated lighting to each pixel. Again, we're trying to, we're going for simplicity and control, right? So this way, again, we don't need that light for every piece.
We can keep the scene organized and performed. Now, the real trick is this dot product node very commonly used in 3D, it is mainly used in 2D when you need to do some lighting. So dot product here measures how kind of, how close to vectors align.
And in this case, we're comparing light direction to the apparent direction of each pixel in the Sprite. Now, the resulting image can be used then to add light to the Sprite, and then we can update those values at runtime for all the pieces using the same shader. Okay? There's a bit of gif compression on this, but this is what the final effect looks like if we zoom in on it for each of the tiles. And once again, there's no actual lights in this scene. This has been done through custom lighting. And just to see how performant this is, we fed one of the gems into AVFX graph and were simulating thousands of lit sprites without a hitch.
There you go. So because of all of these interesting advancements and techniques that you can just apply to your 2D games, it means that 2D games don't actually need to look like they have, right? So three clear opportunities exist when you are trying to create more advanced 2D lights in VFX. And this could really, you know, elevate the visual appeal of your games. So if we look at the games on the left, for example, you could play around with lighting and effects to create the illusion of volume like we saw in the examples with the normal maps. And this will give any piece in your game a little bit more, you know, it's more tangible, right? It feels a little bit more like it's present. Now, of course, any game genre that is static, you can throw some 2D lighting and, and effects into it again, to, to raise the visual appeal.
And finally, it can actually help in the art direction of the game when you're trying to enhance glowing or shiny effects. And you need a little bit more oomph to that final look. Okay, so we've talked about VFXA few times, and so let's once again dig in and let's look at VFX graph for 2D projects. So what's really exciting recently is that VFX graph now supports 2D lit particles, right? And all of the, all of the 2D features.
So this means, this means that we can use some of these effects, and we're gonna dig into the visual effects that we see in Happy Harvest, typical examples of them in 2D games. And then we're going to break down the pieces in Happy Harvest and talk about what, what we did to put them together. So there's a bit of VFX graph in there, and there's a little bit of shaded graph together. Okay? So the VFX in Happy Harvest come in many forms.
One key aspect is animation, right? You often want to play like a frame by frame animation, like the one on the left. This is very effective, very performant of course, and very suitable for effects that have this kind of look a little bit more toy, more stylized. Just a simple animation of transforms or tweezing animation is also very pleasant. This is the one in the middle. And in some cases, we can just offload all of this animation to the shader itself to create an effect where we can just drive shade of values, either via script or in the case of some of these effects here, just animation clips, right? Which is animating values over time. We also use particle effects as a, as a part of all of the VFX.
We see in Happy Harvest, we use the good old particle system as well. And VFX graph, when you need vast numbers of particles, you can go up into the millions, of course. And these are really good when you need complex effects, right? Where, where the number of simulated particles rises into really high numbers.
Now, typically, of course, we'd compose all of this into a game object or a prefab. And the reason for that is you want to have an effect, right? Think about an effect as a game object, and then you can just spawn it whenever you need it, right? Or you can pull them of course, and then, and then spawn them out of pools. So when building these visual effects in 2D, we actually think about a bunch of things before we get started, right? Some initial considerations, this is important to think about earlier rather than later, so that you're building the pieces that you need. And you don't have to then redo all of this later on. So one, one thing that comes up from time to time is should your, are you thinking about a game where the camera is basically orthographic? Or do you have perspective? Now, this is important when you're authoring a lot of these effects because you wanna know whether to think about just a fully co-planner effect, or whether you need to consider volume as part of your, part of your particle system or your VFX graph.
What hardware, of course, we always think about this, but this is important when effects need to scale, right? Or you need to have multiple versions of an effect for different platforms. So you need to think about what you're targeting early on, because you'll, you'll want to address that. Do you want to add post-processing? Now, this is important to think about earlier, again, because if the goal is to put some post-processing over the top, it's a similar thought process for lighting.
You want to prepare that base art such that it responds well to post-processing, right? And not bake too much so that you have these double lit or double effect final results. Now, some, a fair number of optimization techniques probably also need to be considered earlier on if they end up affecting your art pipeline. I always think it's a good idea to check the asset store because it's very often the case that someone has built an effect that either you can use outright or you can start from, right? You can use it as part of your prototyping, as part of your con conceptualization, and maybe later on you could then take those, be inspired by them, and use those in, in your projects as well. So always start with the asset store. I find it's a really good place when, when you're trying to get started quickly. Okay, so some specific examples.
Let's look at the fireplace inside the cabin. So in Happy Harvest, there's, there's this really cozy interior scene, and we've got the fireplace that that has been, that is completely procedural. So the fire has been animated with both VFX graph and shaded graph, and sorry about that, with both VFX graph and shaded graph. And the VFX graph here controls several particles.
So the main fire is a single Sprite that has been masked to give that flame the oval shape that you see. So that's the basic mask. And then the rest is created procedurally, you can see some of it down here With a OID pattern, right? So we use noise and tint color for the flame and the background, and that's how we get the result that you see there. So we've also added some shaded graph effects to the water tiles in the lily pond.
Now here we have a tile map renderer. So you can apply it, these effects to any of the 2D renderers that exist in your scene. And in this case, the tile map renderer is using a custom material with a water shader on it. And all of the tiles inside the tile map are rendered using this material.
So we have this oscillating motion that is a product of the tiling and offset of the UVS over time. And then we add a bit of a noise effect to it. We continue to add to this effect. I really like the next one. I think it's pretty, pretty cool. And we basically give the water this refraction effect.
And again, we're using a similar technique to what we saw earlier with a ripple effect over the gems. You'll see the character go behind the water in a minute. There you go. And this is done by blending into the surface texture, right? With a LRP node.
And again, the refraction here is using the camera sorting layer texture, which you can define in your 2D render asset. So what is this camera sorting layer texture? If you haven't used it, it's a little bit like a grab pass. But what it does is it's a snapshot of all the sorting layers that the camera has rendered up until a defined sorting layer that we call the foremost sorting layer. And all of the elements, everything behind that has been rendered into that buffer. You can then sample that texture in shaded graph and then do whatever you need to do with it, right? Mix it up. Here we are,
we are mixing it into the noise texture. Cool. Now, the watering can spray effect here, that gets triggered by a animation event, a tool animation event, and, sorry, an animation event. And that calls a function inside the tool animation event handler.
And this is a component that has been placed on, on the, on the object. So this script just simply enables or disables the corresponding instances of VFX graphs at the given time in the animation. Now, happy Harvest is really full of small techniques, wonderful little techniques that really help to bring the game to life.
It's really enjoyable to explore as a developer, but it's these small touches. Again, if you look at the sample, if you, if you grab it from, from the asset store, that again, your players are going to enjoy, right? When you brush into the bushes, the tree, the leaves fall out when you're planting the crops, again, there's little pieces, lots of secondary animation, little after touches that really makes the game. Again, you can dive into VFX graph by grabbing this ebook, right? The VFX graph techniques, the, the advanced visual effects in Unity ebook, sorry, again, this has been done in collaboration with professional artists and experts at Unity, so that you can get a headstart on all of the techniques that you can apply when you're trying to create advanced visual effects in Unity. Alright, let's take a look at some 2D improvements in Unity. Unity now comes with some really powerful new improvements that I'm really excited about.
And we'll start with probably one of my favourite a Sprite importer. So I'm really happy about this one because every time I talk to 2D developers or people making 2D games, so many of you are making amazing games with art that you've authored in apr, right? In apr, it's this lovely, this tool Made with Love, and we wanted to support all the awesome work that was being done there. So what does it do, right? So it, it doesn't just flatten your ACE Sprite and say, there you go. There's an image. This importer supports a lot of the main workflows for getting a Sprite art into Unity.
But most importantly, it makes it easy to iterate on on that, right? So you can just drag an ACE or ACE Bright file into Unity, and you get support for all the color modes, but you also get support for all of the layer import, right? So for some of the layer import pieces here, for example, you can get support for hidden pieces, for hidden layers. I mean, this is an option whether or not you want to see those construction pieces that you had in APR or whether you want to leave them out of your Unity project altogether. All the layer blend modes are supported with import mode, merge frames, all of the layer and cell opacity, because people are often doing some work in the cells, in the cell opacity linked cells, as well as tags. This is being actively updated as well. So you can follow the development on on this as it goes. And this is available today, I would check it out if you're using a Sprite.
This could be the workflow enhancement that you've been looking for. Very exciting. Finally, I didn't get to the end. We support individual frame timings. This is, this is really cool.
If you're training, if you've, you've worked out where your hold frames are, how long you want to kind of be on a frame, you know, this is, this is great stuff. And of course, layer groups. So very exciting, super happy about that. We now have a light batching debugger.
So if you're using 2D lights, this will help you visualize how your lights and shadows are batched, because this can be a little opaque, will be a little bit hard to understand. And they're usually organized by these laying sorting layers in the scene. And so you want to know which sorting layers or which lights and shadows can be added or removed to optimize batching and a host of small things or as I like to think about them, small, big things. So we've made sure now that all the 2D renderers support SRP batching, this has been requested many times, and this is, this works now. So Sprite masks can now use Sprite as they always have, but also Sprite shapes and tile maps as input masks, right? So if you want to mask something by a Sprite shape or mask something by a tile map, you can now do that with a Sprite mask. We've added ways to pick brushes and white box tile palettes to the tile map system.
The inverse kinematic manager for 2D now supports camera rust and cuing. So all that expensive IK that's happening off screen is no longer going to contribute to your overhead And a, a nice little workflow accelerator. You can just open the Sprite editor window from any Sprite renderer inspector now, and get very quickly to the Sprite that you were trying to render. Alright, cool. So I promised you a little sneak peek into some of the work that we're doing in ai. Now, the goal is to enhance your workflows, right? And we we're trying to do this again, this is, you're seeing a sneak peek here.
This is not going to be available today or tomorrow, but it's something that we're working on. And we want to enhance your workflows with deeply integrated AI Sprite creation. The goal is you should be able to train on your style for your projects. From that you should be able to generate sprites, create variations, refine your generated images, and apply these results easily to your projects and scenes.
So here we can see a Sprite generator module, for example, that's been integrated into the Sprite editor window. And when you're done generating here, you get a Sprite that you can use immediately in Unity. Let's, we've got a little video to show how it solves a particular problem that we heard was, was a common one that a lot of people had. So I'll narrate you through the story of, Of this next video.
Oops, there we go. Little too fast there. So the team has been working on enhancing your workflows with deeply integrated AI spray offering. Now, in this scene, we imagine a studio looking to release the winter version of their game, right? They're trying to launch for the end of year holiday season. Now, seasonal content is an area where improvements can have a highly desirable and significant impact on the business of a studio. This example, we see an artist, he's using a style, trained on the art of the main game, and she selects a tree and then opens the Sprite generator module.
She configures some generation settings. Now we're using one where we're using the existing Sprite as input. Then we write a prompt to bring a snowy version of the tree into existence. Now she's gonna explore multiple variations and then she can preview them in the scene by selecting them. And then when she finds something that's good, she can just apply it everywhere across the level. Now, because the tooling is built into the editor, she can also make adjustments easily 'cause it's not gonna be perfect, right? So in this case, she's changing the pivot.
She hits supply and she's done. Now this provides a simple, yet deeply integrated way to create and refine, enhancing your existing workflows with the power of generative AI directly in the Unity Editor, the workspace that you know well. And there you go. Now, this is an early step.
We're still exploring many ways forward. There are many other opportunities in the editor and in the workflows where we could be injecting generative flows. But as always, keep authoring beautiful and powerful VFX in your games. Build on the features you know and love now more robust than ever, and hopefully really soon. Create with AI as we infuse your existing workflows with AI powered creativity and productivity boosts, and of course, keep making awesome 2D games and Unity. Thank you very much.
2023-12-19 14:54
