[STEF 2022] テクノロジーを用いたコンテンツ制作の最前線 / The Frontline of Content Creation by Cutting Edge Technologies

Hello, I'm Takashima from Sony Pictures Entertainment. Today I will be hosting a session at the STEF2022 conference entitled, "The Frontline of Content Creation by Cutting Edge Technologies" Thank you for joining us today. Today, many engineers from all over the Sony Group are here.

I'm with Sony Pictures Entertainment. Mr. Tanaka from the R&D Center. Mr. Kobayashi from Sony PCL.

And Mr. Yutaka from Sony Interactive Entertainment. Thank you for your time. In the first half of the day, I will be introducing some of the interesting things we're doing. In the second half, I would like to discuss together, in particular, the fusion of game and movie creation. Let's get right into it, shall we? I'll start us off.

I'd like to introduce "KILIAN'S GAME," a short film production project utilizing virtual and remote production. I'm responsible for the evaluation of technologies at SPE, as well as technological collaborations within and outside the Sony Group. As a Distinguished Engineer, I've been driving the convergence of entertainment and technology for the past 3 years, and this short film is just one of those activities. KILIAN'S GAME is a unique blend of a relatively classic style short film and Japanese yakuza-style action.

It was released on YouTube in April and can already be seen there. Let me give you some background as to why we created this piece. The Sony Group calls itself a creative entertainment company with a solid foundation of technology, and I've always thought what significance it actually has, and I wanted to find ways of expressing it to make it easier to understand. That's why, together with Mr. Yutaka, we launched an activity called the "Content Technology Strategy Committee" last year.

This committee includes the leaders of our entertainment group, including music, games, movies, and Sony PCL. Engineers from the R&D Center and Sony Electronics also joined to provide support for the project. We have about 100 members. There are few opportunities in the entertainment field to try out new R&D technologies for commercial productions.

The engineers, on the other hand, are eager to try out their technologies in actual content production. This committee can provide a forum for collaboration, where the creator side can learn about new technology, and the technology side can try it out. The committee was formed last summer and immediately began to work on a short film, which was actually filmed on location in Los Angeles in November and continued in February using PCL's virtual production stage. The video and audio were finished at Sony Pictures in March and screened at NAB in April. There were two main challenges here. First, the pandemic made the travel between Japan and the US impossible.

However, we wanted to create a seamless connection, as if we were working as one team on one project. The other was that we wanted to make this a serious content production, not a technical test. Therefore, the main focus was on content production, with a framework to use any technology available. To that end, we collected many products, including prototypes, as well as technologies in R&D, to show to the creators. We showed them to Saylor, a short film agency based in Los Angeles.

Then we let them think about what they wanted to make. The result was a film in a very classic style, but in fact, it is rather difficult to create a classic film with quiet color tones by using the latest technologies, such as virtual production. So it had a challenging aspect to it. When choosing the locations and writing the script, we discussed at length how to seamlessly cut and connect scenes shot in Los Angeles and Tokyo. In the end, we formed a plan. The only thing left to do was to execute it.

For three days in November, in the suburbs of Los Angeles, we shot some secnes with the drone and car driving scenes first, then the outdoor and indoor scenes, mainly using conventional filming techniques. One of our goals was to learn these conventional shooting techniques, so we learned a lot about doing them. The technology we used, such as the Airpeak S1 drone and the new VENICE 2 camera, were just released last fall, and were not commercially available at the time.

We found them to be very useful, and are already using them in new films. The biggest technology was virtual production. I'd like to play a video of that now. As you saw in the video, it took three or four hours during the initial filming to capture 360-degree 3D data of the room. Then we took data of actual props such as desks and chairs which were scanned in by a scanning company called Gentle Giant.

Our CG company partner, FuseFX, created an Unreal Engine project for us, which we utilized in the virtual production stage at Sony PCL. Sony PCL opened a new stage called "Kiyosumi-Shirakawa Base" in January this year. Since "KILIAN'S GAME" was filmed in early February, we had the opportunity to test it first. It was a learning experience, but it turned out very well.

Thank you very much. "KILIAN'S GAME" was a joint project between Japan and the US. We held many web meetings about the colors of the film. One proposal from Japan was to use CG in virtual production to set a building on fire, which the American side didn't think of. We ended up doing that with CG flame effects.

After repeated testing, we finally confirmed that we could create a seamless image with very similar color tones between the virtual production stage on the left and the real location on the right, as shown here. Another thing that was very helpful was the remote production tool. As it was a joint production, we wanted to share footage as best as possible. We used Xperia PRO, an Xperia device with HDMI input that allowed us to stream live video between the US and Japan simultaneously. When we shared the video from Los Angeles to Japan, the local network was not very good, but we were able to send three kinds of HD video by combining 4G and 5G networks. As you can see, we connected devices and sent the images to Japan.

And this helped us prepare very well for future virtual productions. With good connection from Tokyo, we could send up to 6 HD videos. We sent main camera and BTS videos, as well as Unreal Engine's screen video. The American director gave detailed instructions on lighting and such. By the time we finished, it was late at night in the US, but he was very pleased with the seamless video that matched the backgrounds. I also noticed here that when Japanese actors make a film in Japan with a Japanese crew, they can give a truly Japanese performance.

It is very rare to see truly Japanese acting in Hollywood productions, but by having the filming done in Japan in this way, I could see the potential for Japanese acting to be included not only on a global scale, but also on a creative level. These logos are for commercial products and services actually used in this film. For details on how they were used, please see the making-of video on YouTube. We are serious about creating a variety of content, not just films.

Each year we work to utilize technology and will continue to expand more proactively in the future. That's all from me. Next, Mr. Tanaka will talk about Sony's volumetric technology. The floor is yours, Mr. Tanaka.

Thank you. I'm Tanaka from R&D. I would like to talk about volumetric capture, which is a technology designed to capture an entire space and express it. Before that, I would like to look back on the history of spatial video itself, or rather, I would like to give some of my thoughts on the subject. I have been interested in VFX in movies since I was a child, and I was fascinated by the many worlds depicted by VFX in science fiction movies. For example, the weightlessness in "2001: A Space Odyssey" or the holograph of Princess Leia in "Star Wars." Or the opening of "Blade Runner" and its depiction of the world.

Then there is "The Matrix" and its famous bullet time scenes where you wonder how they were filmed. This shows how VFX has evolved through the years. VFX techniques originally started with overlaying on film, but they gradually evolved into the use of CG.

CG and live action have merged to a large extent since, and we're seeing more approaches of converting live action into CG. This filmmaking process is advancing in many areas, and it is often intertwined with research, and there is a movement to create new methods of filmmaking in collaboration with academia. For example, the Institute for Creative Technologies, or ICT, at the University of Southern California is a prime example of this. They're making 3D models of the human body to use in the filmmaking process. In the first place, recording images, not limited to film, is a natural desire for mankind.

It started in ancient times when people painted pictures in caves. And then photography came along and moving pictures were made. Then came the emergence of television, which enabled us to deliver images to different places in real time. Then our company, Sony, for example, developed a camcorder called "Handycam," ushering in an era where people can record video anywhere.

There have always been attempts to change the viewing experience itself, and this is where spatial video came in. Until now, content has been passive, with the viewer watching what the camera decides to show, but from the viewer's perspective, the need for more active viewing and interactivity has been a longstanding issue. For example, we now have omnidirectional video which is taken in 360 degrees. Or, as has been done for a long time by Professor Kanade at MIT, free-viewpoint video from multiple cameras has become more widespread. On the other hand, what about Sony? Sony has long had the concept of transmitting and broadcasting space itself. We have been researching possible best methods for each era.

As you can see in the diagram here, we started with the concept of broadcasting space, and by 2003, we had developed a 360-degree camera called FourthVIEW, which enabled spatial video to be viewed on PlayStation. The technology we have been working on more recently is volumetric capture, which aims to capture the entire space, using multiple cameras surronding the subject, and create 3DCG from the captured data. so that the images can be viewed from different perspectives.

We've been working on this R&D project for a long time, and recently we opened a volumetric studio next to the virtual production studio at Sony PCL's Kiyosumi-Shirakawa Base, and we are talking about creating new expressions that combine virtual production and volumentic capture. We have created a sample video to show what can be done with the footage shot in that studio, so please take a look. As you can see here, shooting with volumetric capture allows expressions that couldn't be done before. There are two main characteristics of Sony's volumetric capture.

One is what we call "photorealistic rendering," which reconstructs images from the captured images themselves, so the images look more like they were shot on film than like they were made of polygons. Another feature is that it can do pose estimation, which estimates pose based on each camera and then accurately captures the 3D movement of the body by synchronizing with the video. These two features make it easier to use motion in video production.

As for the future, what we are shooting now is just capturing the light as it is there. This alone would be worthwhile, but I think it is very important to add to this the estimation of animation and materials when producing new videos in the future. For example, the video shown here is white balls just falling down, and you don't know what is going to happen, but by adding physics simulation and materials, the balls look as if they are really there. So, we're also working on technology that can estimate materials and make physical calculations for video, including animation.

This has been my introduction on volumetric capture and its future. Thank you, Mr. Tanaka. Next, Mr. Kobayashi from Sony PCL will talk about the second generation of virtual production, also involving volumetric capture. The floor is yours.

I'm Kobayashi from Sony PCL. I would like to talk about the next generation of virtual production, which combines volumetric technology and virtual production. Sony PCL is a video production company, which delivers the technologies you develop to actual clients. Now let's get to the content. Virtual production, as you may know, has been getting more and more famous. There is a large LED panel on which a 3D background is projected,.

It's critical that the background is 3D, which is connected to a camera. As the camera moves, the background behind it is rendered frame by frame in real time, making it appear as if the performers were there. This is a sample clip, and what you see in the background is actually LEDs. Since the car itself is live action, the light reflected off the car is supplied by the panel behind it, creating the effect that the car is actually being filmed in this space.

Next is volumetric capture, which can capture real space in motion as three-dimensional digital data. This is the same technology that Mr. Tanaka talked about earlier. On the left side, we have a traditional 2D camera image.

This is flat, so the angle cannot be changed later. But since we are shooting in 3D and recording for real time playback, we can freely move the camera position later. If you view this with a 2D device, you can view it from any perspective. If you view it with a 3D device, such as 3D-capable display, or a VR headset, you can view it from the direction you want to see it.

Please see this video for a sample. This is a real example used in a music clip. The top one was shot at the Volumetric Capture Studio, and the bottom one is the finished product using the data shot above. So the viewers are free to choose among many viewpoints from just a single shot, and because it is 3D data, it can be processed and output in multiple ways. Above all, it's photorealistic. The fluttering of the clothes here, for example, is captured very beautifully, which is a clear departure from conventional modeling techniques.

Here's what actually happens when you combine these, using volumetric data with a virtual production background. So, when the camera moves, the background actually moves in tandem. The three-dimensional volumetric capture data is placed in the background, so that the background can be recorded with a 3D effect and parallax. This new technology will allow us to create more highly dynamic camera work than ever before. And it's so photorealistic.

I believe that virtual production will continue to evolve with new technologies, and I have listed four trends that I see on the horizon. First, more camera freedom. And, while most productions today use a single camera, multiple cameras allow many perspectives and switching between cameras.

Also, high-definition and motion background will help create a very immersive space for the actors to perform. Another important thing is that the VFX team can work together with the art and lighting teams, and share their know-how on production. This is a major change from today, where film production and CG creation are separate. If what we have done so far can be called the first generation of virtual production, the second generation is a natural progression in terms of increasing the flexibility of the camera as the panels become larger. This allows us to do backgrounds of large battle scenes or large crowds.

On the other hand, I believe that technologies will emerge that will make virtual production more compact. For example, by using a turntable and rotating the background in the same way as the turntable rotates, we can create the effect that the camera is constantly panning. There is also a technology called set extension, which uses AR technology to extend the space beyond the LED. There's been an increase in such use of XR in filming. The multi-camera system can allow for virtual production to be used in conversation scenes on TV, or like the conversation we're having now.

I expect that by incorporating AR into live events, we will be able to combine realism with a large amount of information and deliver richer images than ever before. Above all, it will be important to have collaboration with the lighting and production desingers. Bringing digital technology in will be a step to digital transformation. By adding volumetric data, the backgrounds will become richer and richer, making them more vivid. I hope that new technologies will emerge and be integrated in the future. That's all from me.

Thank you very much. Mr. Yutaka will introduce latest game production techs, including game engines. The floor is yours. I'm Yutaka of Sony Interactive Entertainment. I joined Sony in 1988, and I was involved in the development of the original PlayStation. I've followed the PlayStation's evolution for almost 30 years since.

Today, I would like to talk about game engines in light of that history. First, let's talk about the key points of interactive entertainment. Interactive entertainment differs from traditional entertainment, such as movies, in several ways.

One is that it must provide real-time reactions. The appropriate scene must be created in response to the user's input, which is a very technical challenge. In the case of a movie, you can shoot the film in advance, but in the case of games, the scene must be created accordingly in 30 or 60 frames per second or something like that. Very difficult from a technological perspective. The other is called real-time reconstruction.

In other words, scenes need to be created according to the creator's intention as well as the user's input. This is another difficult point. Such reactions are generated through software, or programs.

In the case of game graphics, as I mentioned earlier, we started out with very simple pixel graphics. However, as technology has evolved, we eventually have to create these very photorealistic worlds. It's become very difficult to create them as desired in real time, technologically speaking. It also means that a lot of work goes into producing these very high quality videos.

Now I'd like to talk about how these graphics are made. There are two key points to creating high-quality graphics. One is that the assets, such as individual cabbages in the cabbage field shown before, must be of high quality. The other is called the runtime, which involves adding movement when the final graphics are created. Therefore, high quality graphics require good assets and runtime.

In terms of recent game graphics, assets are created by artists, but there is some technology involved in the process. As graphics become more photorealistic, capturing the real world is one area where technology is involved. The use of AI-generated images is also a hot topic these days. Then there's the challenge posed by the runtime.

As I said before, graphics have to be created in 30 or 60 frames per second. This requires effort from the programmers. Game engines contribute to this, and I'll come back to that later. Many technologies are involved here.

For one, we use simulation technology, such as physics simulation, to add movement. We also use 3D audio to add environmental sound effects. We also use AI technology. There are two objectives in using these technologies.

One is to produce even higher quality graphics. The other is to reduce the load on artists and programmers, to optimize and reduce costs. As for how the assets are designed, up until PS3, they were done with raster graphics.

This is a method of making a game look realistic by, for example, applying textures to 3D polygons. The challenge from there was to make graphics photorealistic, so from PS4 onward, we introduced physical rendering. Basically, what we humans see is light, a physical phenomenon. By simulating that light, we are trying to make it very photorealistic.

In order to make this change, the asset creators have to do a number of difficult things. In the old days, you only had to make about three texture maps, diffuse, specular, and normal. In order to do photorealistic rendering (PBR), we need to create much more data such as base texture, roughness, height map, normal map, IBL, and cube map. This is one example of how very difficult it is.

Next is runtime technology, which also requires physical simulation in the sense that graphics must have natural motion in order to look realistic. When objects move, they basically follow the laws of motion. Therefore, the equations of motions must be solved to make things move.

Simulating light, like physical rendering, can create realistic graphics. Then there is character AI, which is the technology that determines how the character will move. All these technologies must be created in runtime. This is where game engine comes in. When incorporating such a wide variety of technologies, it is impossible to create them for each game from scratch. Therefore, it is necessary to put several technologies into a single engine so that many games can use them.

As for the runtime, it must run at 60 frames per second. This requires multiple technologies to be accessed at the same time, and these must be coordinated in order to ensure timely processing. A game engine makes this process more efficient. The game engine also makes it easier to run physics simulations, and easily control movement using scripts and parameters.

Thirdly, since assets take a lot of time and effort to create, an engine makes them interoperable, so they can be used on many games. A game engine also allows interoperability between platforms, say, porting a game created for PlayStation to a PC platform. Therefore, game engines are a very suitable method for modern games. There are two major game engines. One is the Unreal Engine, which is also used in the virtual production mentioned earlier.

The other is called Unity. At SIE, we've created a game engine called Decima. This was made by SIE's first-party company called Guerrilla Games.

Guerrilla Games creates high-quality games, such as "Horizon Zero Dawn" and the latest title "Horizon Forbidden West," both of which use Decima. The engine is also used for other SIE first party AAA games. The engine is unique in that it is designed for PlayStation and optimized to maximize the PlayStation's performance, resulting in very high quality graphics. It also includes a wide range of cutting edge technologies, such as the PBR renderer I mentioned earlier, physics simulation, character animation, climate simulation, and other natural objects.

And also procedurel generation, which generates many things automatically. I would like to explain the features of this game engine with a video. At SIGGRAPH 2022, there was an event called Real-Time Live, and I've brought some demonstration video that was shown there.

We are honored to say that Decima won the Best Show award in the competition. First, the real-time performance. As you can see, it moves very smoothly at 60 frames per second, rendering a more beautiful world. To schedule this properly, several processes are running simultaneously. We optimize them so that we can see at a glance how they are running and where the processing load is heavy. This makes real-time movements at 30 to 60 frames per second possible.

I would like to go on to the next video. Next is a physics simulation, and procedual generation. As you can see in this scene of moss, you can probably tell that we use procedural generation to automatically grow moss from where there was nothing, so we can create different scenes at will. In the next scene, you can see grasses, and each grass has bones to be animated with the characters, and a physics simulation was used to make the grasses sway realistically. Let's look at the next scene. Next, recreating the natural environment.

You can see that the waves are moving very realistically. The physics are calculated so that each wave moves realistically. The transparency of the ocean surface and the way the character swims in the ocean are also calculated to be realistic. Next is a weather simulation. As you change these parameters, the world will gradually become stormy and thunderstorms will appear, and so on.

We can control the movement of waves and water, as well as the weather, in this way. Since it is raining, the character naturally gets wet with rain. Here are some close-ups. You can see that the surface of the skin is wet from the rain. The game engine can reproduce such high-definition graphics. Finally, the character flies over on a dragon, escaping the storm and seeing the sunset.

This shows how the entire world can be created with a game engine. To summarize Decima, the SIE game engine, it is an engine that can render beautiful worlds in real time. The climate simulation, breaking waves, automatically generated plants, and character movements are realized very well with the cutting-edge technologies. As mentioned earlier, the weather can be controlled with simple parameters and the images can be viewed in real time. These are key features of the engine, which allow us to easily check very high quality graphics. Next, I'd like to talk about "Beyond Game" Game engines are capable of rendering high-quality virtual worlds in real time.

But this technology is not only limited to games. We are thinking that it can be used for virtual production, as mentioned earlier, and in the future, it can also be used for the metaverse or digital twins. I think that game engines will become much more important in the future. That's all I have to say. Thank you very much. I am moderating today remotely from Los Angeles, but having learned how each technology is used to create such high quality content, I was able to understand the importance and necessity of these technologies.

Let's get right into the discussion. I work for Sony Pictures, and recently, we have been seeing a great deal of interest in utilizing game production technology. By using real-time motion capture, it makes it easier for actors to act with something actually moving.

The director also needs to make sure they got the footage they want, instead of waiting for months for CG, after which it might be too late. Therefore, we need tech that allows us to check in real time whether we are capturing the footage we want. Are there any changes or new developments in game technology in this regard, Mr. Yutaka?

Well, in that sense, as I mentioned at the beginning, real-time is a very important element of games. If you don't do everything in real time, you can't respond to the reactions. In that sense, we have been pursuing real-time performance ever since the PlayStation 1 launched. Of course, the images were not so high-definition in the beginning, but as computer technology has evolved, they've become much higher definition, and gradually real-time capability has come to be used in many fields. I have one question for Mr. Takashima.

What are the benefits of real-time capability, and what do you think will improve as a result of real-time technology? I am sure it will contribute to film production in many ways. What are your thoughts on the current real-time trends from the standpoint of the film industry? Well, I think there are two things. For one thing, it will be possible to know in real time on set what was previously not known until later. The in-camera VFX and virtual production using a LED panel that Mr. Kobayashi mentioned are the right example of it. You can see everything during the shoot, so if there are any problems, you can re-shoot on the spot.

What is said to be lacking there now is when the background is relatively static, and there's a fixed set or animation, such as a robot walking. It's not easy to change the movement of the robot, or the movement of an object in the background. This can be done in real time by a person using motion capture. Secondly, there is pre-production, which is the stage before the filming of the movie begins, in which the shooting plan, scenes, content itself, and concept are decided. Until now, CG artists had to repeatedly create, check, re-create, and hand-draw the images, an extremely time-consuming process.

Here, too, the process can be done in real time before the director's eyes. By showing the 3D data to the director in real time, the director's imagination is able to expand and the concept of the work becomes more and more solidified. It is very effective to use real-time technology to ensure that there are no mistakes. In any case, I think there is a growing expectation for tools that allow creators to check what they are thinking in real time. Thank you. In that sense, I think real time is very effective not only in improving the efficiency of production, but also in improving the quality.

In this sense, game engines are capable of controlling many things in real time, and I think they will be used more and more in the future. I think that even in virtual production, there are still aspects that need to be run in real time, and I would like to know more about the use of game engines in virtual production. You mentioned earlier that you are using Unreal?. Yes, we live in an age where we can produce usable image quality, with technological advancement, image quality has greately improved to the point it can actually be shot, recorded, and played back as a work of art. As Mr. Takashima mentioned, its advantage is that

interaction will occur as it is real-time. Up until now, video production has been based on a solid script, with predetermined movements, and the performers acting according to that script, but now there is a little bit more range. We create a rough story where the camera can move freely, and we can have unique motions, which don't have to be improvisation, but a bit more humanness in them. I think the beauty of interactive and real-time is that we can now put people at the center, whereas we used to think based on CG and post-produciton. In that sense, some interactions are controlled by things like game engines, but how to control them may be a new area to consider. That's right.

Assets are very important in games as well, and the key is how to create them. Capturing is one of the key points that has to be addressed when attempting to be more realistic. What do you think volumetric capture and 3D capture will be like in the future? As to the ability of real-time rendering, more things can be done with increased PCs having higher signal processing capability and stronger computational power, so the assets are becoming higher in resolution when viewed in 3D.

As Mr. Yutaka and Mr. Kobayashi mentioned earlier, it takes a lot of time just to create the background assets. The question is whether we should make that process harder at this point. When it comes to creating content, social implementation are necessary.

We have to think about what is practical, not just what our research makes possible. From this perspective, the ability to easily convert live-action video into CG would be more convenient than working hard to create CG. In virtual production, for example, it used to be common to use CG to create background assets, but now there is a technology called NeRF that has recently emerged. This can create complementary views from images captured by multiple cameras. If we can estimate 3D from live images using machine learning and display them in real time, the asset capture can be done similarly to location hunting, and can be immediately replayed after being shot.

We are getting better at doing this with stationary objects, but with people, for example, current volumetric capture is limited to playback of what they have just performed. In contrast, we've made leaps in quality when it comes to scanning objects in 3D, and adding materials and animation to them. such as digital humans in video games. However, the higher we goes, the more costly it becomes, although we were able to clear the so-called uncanny valley. It's relatively inexpensive to make a volumetric capture.

But right now, we can only play it back, so it is still difficult to get a particular person to respond in real time, as I mentioned earlier. A digital human made with CG is easier to react, but is also more expensive. Therefore, the key to our research is to maintain this balance, which is what we are currently working on. Our group is working on both right now, and we're doing both digital humans and volumetric capture. But I think that if you make the digital human, you can drive it with volumetric capture.

If you have both of those technologies in place, you can use them. For example, if machine learning data is captured by volumetric capture in 4D, and if there are 3D assets, the image can be reproduced from the learned data in response to slight movements of the performer. This is an area that will be the focus of future attention, and we are currently working on it. The greatest point of volumetric capture is that it can capture complex objects that are difficult to create with models, such as plants full of leaves, or humans in particular, which are difficult to create with CG. This is where the realism comes in. As was mentioned earlier, there is a big jump to reconstructing the image and adding movement to it, but I think it would be amazing if we could apply AI technology to this, and if 4D capture, which involves capturing over time, could be used to add movement to the volumetric capture, and if we could move the image as we wish.

I assume that's the kind of movement you want to create in film. If you don't mind me jumping in, films usually shoot the same scene about 10 times from different angles. "KILIAN'S GAME" was no exception. The actor repeats the same performance 10, 50, or 60 times until the cut is OK. If we can shoot the performance photo-realistically and volumetrically, we would only need to get one good shot, and compose camera angles freely later. So, even without animation, technology that can shoot a real performance in 3D volumetrically would be very useful as well. In today's virtual production, a 2D image is fixed once it's shot.

You need to use CG if you want to change it. The volumetric capture gives you the freedom to change the angle later. Mr. Takashima, in that sense, what you are talking about now is the demand to see different angles in the virtual production. Are there any other demands? Is there anything you want to control, like weather changes or situation changes, besides angles, in real time? I think you have a lot of experience with this.

Sometimes, once you start filming, you realize you want it in the evening. Normally, you have to wait until the next day, or when the next storm is coming. With virtual production, we have lots of freedom, so that makes it possible to think about the scene you want to shoot, the kind of lighting you want, and the kind of time of day you want to shoot it. Can virtual production be used for that kind of thing? Nowadays, it is possible to change the weather conditions, or time of day, to some extent, by using a climate simulation.

However, in pursuit of photorealism, it is still better at the moment to film morning scenes in the morning and night scenes at night. If we can get more times of day from a single asset, it would be very useful for virtual productions. In that sense, is there demand for relighting or changing the lighting of those 3D assets? As Mr. Tanaka mentioned, it is in the area of materials. At the moment, we are still relying on textures.

If we can create proper textures based on physical calculations, and if they move in real time, we will be able to create truly realistic backgrounds, and we won't have to worry about whether certain objects are real or virtual. I think it will be exciting to be able to create pictures where everything in the background is on an LED panel, but you don't notice. We'll see things that have never been done before.

In this sense, I believe that our game technology, like relighting, will gradually come into play in this field. We could go on and on, but time is short, so I will end the discussion here for now. In closing, I've received the impression that new types of content can be created by combining game technologies, volumetric technology, and traditional production experience in this field. I'd like you all to give a closing thought from that perspective. I will go last, so let's start with Mr. Tanaka.

I am sure that as technology advances, more things will become possible, but the technology I am currently working on is aimed at creating and manipulating images in space. We can now just capture things and express them photorealistically. But game technology, especially physical simulation, as mentioned, will allow us to do things that were not possible before. I hope to focus on creating something that has never existed before. Thank you. Mr. Kobayashi, please.

I believe that new techs will continue to be developed and more will be possible. But it is the creators who actually make use of them. I believe that it is important to identify what creators want to do, and combine technologies that make it possible. We need matching of "what is possible" and "what is required." So, I would like to establish such a position within the Sony Group and work with a view of bringing more Sony technologies to the world. Thank you. Mr. Yutaka, please.

Games have been pursuing real-time interaction since their inception. As the quality improves, they will expand their applications beyond games. Especially when it comes to virtual production, you need to move things in real time, as well as relighting. I believe that game technology will play a very important role, and I hope that the Sony Group will further focus on this area to make advancements. Thank you. From my experience with new technology in my studio, I know that while technology makes new things possible, traditional work by skilled creators is sometimes actually faster.

In order to use both, I think it is very important for the two groups to discuss and find the best way to work together. It is crucial to always show a variety of technologies to creators. Thank you for your time today. I look forward to working with you again. Thank you very much.

Thank you.

2022-12-10 10:29

Other news