Sarah Taillier: This is The Future Of, where experts share their vision of the future and how their work is helping shape it for the better. Sarah Taillier: I'm Sarah Taillier. From explosions to fictional landscapes and 3D recreated human beings, visual effects are prevalent across much of today's onscreen entertainment.
As technologies advance and audience engagement changes, what could movies look like in the future? In this episode, I was joined by researcher Dr. Stuart Bender, and industry expert, Brendan Seals. Stuart is a major coordinator in the screen arts program at Curtin, and researches psychological responses to high emotion media, while Brendan is a Curtin graduate and a supervisor for visual effects studio, Luma Pictures. Brendan has worked on several blockbusters, including the 2022 Oscar-nominated Spider-Man: No Way Home.
We discussed when and how visual effects are used, how audience expectations are changing, and how the industry is utilising new technologies, such as deepfakes and LED StageCraft. Sarah Taillier: If you'd like to keep up to date with the work of our guests, you can visit the links provided in the shownotes. Sarah Taillier: Brendan, visual effects are used on screen more than most of us realise. Can you give us a few examples from your work? Brendan Seals: Yeah, sure.
I think what's great at the moment is it's really used for everything from telling story to covering for some of the fallout from COVID, with being unable to go on location, to actors not being able to be in the same place at the same time. So it really is used in all matters of speaking. And of course, the traditional big hero explosions and stunt performances and superhero battles that we all love to see on the big screen. Brendan Seals: So as far as what we cover at work, it's really that and everything in between.
So we've worked on, for example, Jojo Rabbit, which was very much invisible work for Taika's dark comedy. So lots of period piece, World War [II] set extensions, clothing, flags, crowd replication, tanks, explosions. A lot of things that otherwise you'd expect to be in-camera, but couldn't be, potentially for safety reasons, particularly because the cast were predominantly children. Brendan Seals: And then all the way through to the other extreme, which is all the Marvel fanfare of Iron Man, Hulk, Batman, Black Panther.
So we've worked recently on Doctor Strange 2 and Spider-Man: No Way Home, which was obviously a massive hit worldwide. That really was that case of production primarily taking place during COVID. For example, the opening sequence of the film, we were in charge of and it was taking place in New York, and they never visited New York, in terms of the crew and the cast, because of COVID reasons. So in that case, visual effects was not just used to heighten the story and the drama and the action, because of potential obvious things that Spider-Man can't do in real life, but also the fact that the actors weren't able to make it to New York. So that entailed full CG landscapes of Times Square and Madison Square and everything in between, you name it.
Sarah Taillier: My goodness. What a feat, Brendan. Can you take us there for a moment? What were some of the points that you really enjoyed creating those visual effects for? Brendan Seals: Yeah, I think Spider-Man, for us it's the third outing. We've actually been very lucky to work with Jon Watts and the Marvel team on the previous two efforts. So we've got a really good relationship with Marvel and the client on those particular franchise properties.
I think just being able to bring back characters from all the Spider-Man movies that we grew up with from the Raimi trilogy, it was a real delight for the crew. These movies are often, for the crew, the reason why they're in the industry to begin with. Brendan Seals: So we had the opportunity of bringing Electro and Sandman back to life, back on screen, giving it a bit of a new twist and take. A bit of a refresh in the look and in the quality of the visual effects. And we were afforded a really great opportunity to collaborate on the previous level. So even coming up with ideas for the choreography of the battle, the staging of the fight, and even the performance of Spider-Man himself.
And Jon was very, very collaborative and gave us a lot of leeway to pitch ideas, which is super exciting for the crew. And third time around, we're kind of becoming experts at how Spider-Man moves, so that's really fun. Brendan Seals: And to see the film do so well was a really, really nice way to cap off what was really a tough year last year, because like many industries, we mobilised completely from home and we actually started and finished Spider-Man completely working from home.
In fact, there were many people I never even met in person working alongside me on that project. Sarah Taillier: That's incredible to imagine, the freedom that you've had with being able to fill in those voids but also use your creativity on so many different levels. Stuart, just turning to you, how do you think audiences' appetites for visual effects has really changed over time? We just heard Brendan talking about the evolution just within a few movies there. Dr Stuart Bender: It's really interesting hearing Brendan talk about, I guess, the range from invisible effects through to very obvious spectacle-based effects. And it's probably worth considering the two traditions of cinema that over a hundred years of cinema have kind of started to collapse with a lot of the visual effects work.
So the early traditions of cinema were either a more seemingly realistic tradition, the famous Lumière Brothers shot of a train arriving is the classic here. And this sort of tradition privileges the idea that the image is showing you something that really happened and you're witnessing this. And the other tradition, developed at exactly the same time or roughly the same time, was the Georges Méliès tradition, which was clearly a trick film.
It wasn't meant to take place in reality, it wasn't meant to be naturalistic. And these were very, in their times, special effects and in-camera effects heavy films that were very obviously so. Dr Stuart Bender: Those two traditions have continued until today, but we're starting to see a bit of a collapse of those, where Brendan's talking about some invisible effects that might be ... Whether it's background replacement or maybe the day that you're filming, the sky isn't what you want, so you change it to something else. And the audience is not meant to notice those things.
But then the audience is obviously meant to be aware of the spectacle-based, type visual effects work. Dr Stuart Bender: So I think it's interesting to keep that in mind, but also we're seeing even a change within visual effects films. To prepare for this, I re-watched the 2014 Planet of the Apes film, which I love and I think still holds up, but there were a couple of parts of it where the motion of the apes just looked a little bit odd, but only a couple of parts. And my memory of watching it at the time, seven or eight years ago, was none of that. My memory was that it was superb, it was flawless.
We can do anything now. So what's changed in the last seven or eight years or so. Dr Stuart Bender: And there's really great research on, I guess, the changing scores on IQ tests of the population. This is called the "Flynn effect", where James Flynn noticed that, okay, every 10 to 20 years, psychologists need to change the questions on IQ tests and change the way that they analyse the data.
And it's not the case that people are getting smarter or people are getting cleverer, there's a very specific part of IQ tests that people's responses are changing to, which are what's called the Raven's Progressive Matrices. This is where you get a complex visual shape and it progresses from one ... It changes a bit over, say, three iterations of the shape, and you need to predict: "What is the fourth one in this series?" And people are getting much better at that. It doesn't necessarily mean they're getting smarter, but they're better at extracting visual information very quickly and making predictions about it. Dr Stuart Bender: Now, the research into this, particularly by important researcher called James Cutting, has argued that movies are part of this cultural evolution in terms of how people's ability to quickly extract visual information has changed over basically every ... Well, every 10 to 20 years, people become, I won't put the numbers on it, because I don't have them in front of me, but people become almost twice as smart at picking up a visual information or twice as quick at extracting visual information.
So movies are partially responsible for this, so is what people are doing with telephones or cell phones and exploring all sorts of computer graphics in everyday life and so on. Dr Stuart Bender: So it's interesting in terms of films to look at if I go back seven or eight years, my ability to extract visual information has improved over the last eight years. So things that I couldn't notice eight years ago, now stand out very prominently to me. Sarah Taillier: So we've seen so much change already in this sphere. Like you're saying, these rapid changes on either side from audience and from the people behind the screens.
How do you expect that visual effects will change in the future? Dr Stuart Bender: It's a really great question and it's difficult to answer because, of course, the fact is that eight years ago, I thought that we'd reach the limit, and then just last night, I rewatched the film and I can spot things in it that didn't occur to me eight years ago. I think I might actually answer the question by asking Brendan a question instead. Brendan Seals: Sure. Dr Stuart Bender: Which is that I rewatched Spider-Man: No Way Home very recently. And obviously, the quality of visual effects are superb, but what I'm interested in is what discussions or planning might go into the execution of those sequences to keep the audience oriented. Because the shots are cutting very quickly, there's huge amounts of visual activity in every frame, which is kind of different to another film you worked on, The Tomorrow War, which has a very ... Not a very slow pace, but a slower pace than Spider-Man, and the
shots are a little bit wider or quite wider in some cases. So I guess I'd be interested in how you discuss, plan, think about how the audience is going to be able to process the visual information in shots, particularly those that are perhaps entirely computer-generated. Brendan Seals: Well, Marvel's quite well known for their visualisation process. They spend a lot of time and resources on really getting the film into a ... It resembles
almost an animated version of the film itself well before even the shooting. And at that point, what they're able to do is go through test screenings, test audiences, demographics, people that ultimately are the consumers of the product. And it gives the director and the execs at Marvel that chance to take a step back and reflect on those kinds of conversations. Because that is often one of the most important questions we have to answer is: "Is the audience oriented? Do they know where they are?" Brendan Seals: It was certainly one of the main things in one of the sequences we worked on, which was a nighttime battle in the forest.
Obviously, it's nighttime. There's a singular light source, being the moon. But because it's dark, it's very easy, with the fast pace of action that you find in a Marvel film, to lose yourself. So in that case, we had excellent previs to start from, and that is the case most often with the Marvel sequences.
But it is a case of vendors like us, Luma, coming in and taking over and absorbing that previs into our own pipeline and workflow, and starting to update the visual quality of some key scenes, and then updating some of the choreography that didn't test so well or the director wasn't feeling so confident on. Brendan Seals: I think what's changed over time is that previs used to be this kind of product that was delivered and it was finished and it's done, and then it moves into a very next definitive state. And then that becomes postvis where you're implementing plates of photography, and that's a semi-finished version of the film. And then you move into post-production, and then it's execution to final. There were very definitive states. Brendan Seals: What we're seeing now is that it's just a seamless living, breathing organism that just continues to evolve all the way to when the film releases.
Previs gets absorbed into the studio's hands, they update it, they show the director. Plates get shot and captured and delivered to the studios. We update those previs shots with plates.
And really, now there's just no seam or end or start to this process. So what used to be certainly treated and looked at as this post-production process, now really is just a production process. Visual effects engages at the start of the film and stays with it until the end. Sarah Taillier: Brendan, if we're looking at how that's actually captured as well, some TV shows and movies, like Disney's The Mandalorian, are using large LED screens with visual effects on their film sets instead of blue screens. What does this mean and how will this technology impact the visual effects industry? Brendan Seals: Yeah, I think this is a profound step in the progression of visual effects.
And with the advances of visual effects in the '90s, we obviously saw come the 2000s, many would say an overuse or overreliance on visual effects to cut corners, let's say, that were potential problems on a production level that could be solved with visual effects. But maybe the visual effects technology or resources weren't quite up to the challenge at the time. So we saw that period of pushback of, no, the audiences don't want to see a heavy reliance on visual effects to solve problems. We still want to have our filmmakers thinking and engaging about the art and the design and the language of a film, and making choices on set.
Brendan Seals: So I think what is really special about technology like the LED screens is that it's bringing us back to a bit like what Stuart was saying, traditional means. We're talking about filmmakers making artistic decisions on set in pre-production, in production, in design with real imagery up on screen before their very eyes, and a cinematographer reacting to that, framing for that, as opposed to being on a blank green screen stage and guessing and interpreting what might happen in and around the actors. We have now a case for that landscape changing back to what it used to be, with directors and production designers really being tasked with locking in decisions on the day. And I think the results speak for themselves in the quality of The Mandalorian and such TV series, which are really at the same standard of cinema now. Dr Stuart Bender: I'd love to know what Brendan thinks about Top Gun: Maverick, which goes out of its way in the marketing to demonstrate– Brendan Seals: [Laughs.]
Dr Stuart Bender: That they didn't use this kind of the light stage and that sort of a virtual background. Brendan Seals: Yeah, I mean, I'm all for ... I think you would find many supervisors, I mean, I wouldn't go on record to say "all", but maybe "all" would prefer that as much as possible can be done in-camera. Because at the end of the day, that's what we're striving for. We're striving to reproduce what could be done in-camera.
And if it is done in-camera, then the job's done. And we are looking for that perfection and realism, and what better product than getting it in the lens. Brendan Seals: As for the rhetoric around marketing and the talk of it being done in-camera and how that's better, we've worked on films that involve visual effects in quite a profound way, but because of the nature of the film not being, let's say, a spectacle or visual effects driven film, that topic was kind of taboo. And certainly, many studios on these kinds of films aren't allowed to discuss that angle, because it almost deters from that marketing topic of all the effort that was put in on set. I guess we're at this precipice right now where audiences feel like the product is going to be better instantly if there is this rhetoric of it being in-camera. I think in the case of Maverick, there is a strong case for that because of the intensity and the action that you really want to capture.
But I wouldn't be underestimating the number of visual effects shots that are in films that are purported to be wholly in-camera. Let's just say that. Sarah Taillier: Controversial.
[Chuckles.] Dr Stuart Bender: It's a really good point, because the charm of Maverick is that the film is meant to be nostalgic. The narrative's nostalgic and the approach to the style of the film is attempting to replicate a lot of the things that were used in the original in terms of the long lenses and the types of framing and the pacing and so on, which is actually part of ... Can we do a spoiler on here? The narrative being that the younger pilots can't match what the older pilot in Tom Cruise was able to pull off. So there's a explicit rejection of contemporary technology in the narrative of the film, which is part of the way that it's made, part of the way that we're told it's made, part of the way that it looks, part of the way that it feels.
It's a great film. Brendan Seals: Yeah. I think one thing to add, Stuart, would be that with that much intent, and obviously, Tom has always been very vocal about the experience of film and theatre and the experience of trying to do these on set, I think what that speaks to is that there's that real desire to preserve that essence of real filmmaking. I think what's key here is, actually, that effort and truth goes straight into the post-production.
Because what they've done is they've really made an effort in every case to really get it in-camera. So if they don't pull it off in-camera, 9 times out of 10 with a production like that, with the way it's been handled and that truth, you'll find that the visual effects studios have got immeasurable data and reference to match into. So the shots that are visual effects shots: you won't know they're visual effects shots because they've had such good material in surrounding shots to begin with, because of that intent at the inception of the design process. Sarah Taillier: Stuart, another area where special effects is really kind of breaking new ground: can you talk to us about the increased use of deepfakes? Dr Stuart Bender: So deepfakes is a technique where you can take an image, or in this case, maybe a video of some person and replace usually their face with a computer-generated face or a different person's face and essentially make it look like you have authentic vision of someone doing something or saying something that they didn't really do or say. So it's possible for people to make videos that maybe show a political figure doing and saying something that they may not have really done or said, or even just an average person.
We see not exactly the same thing, but for a person to imagine what this might be like if they haven't seen it is maybe a Snapchat filter. These sorts of things where you can alter the visual appearance of yourself and it's mapped onto your face and your facial expression, to some extent, as you move. Dr Stuart Bender: A good deepfake looks pretty convincing.
They can be more or less convincing. Stick with the Tom Cruise theme, there was a really convincing set of Tom Cruise deepfakes recently that someone made. Dr Stuart Bender: But these require huge amounts of reference imagery of the person that you're trying to take their face and put it onto someone else's face. So if you've only got three pictures of me at this stage with the technology, you're not going to be able to take those three pictures and map my face onto something else.
But political figures have plenty of imagery of their faces out there, plenty of reference material that the artificial intelligence deep learning can use to generate, I guess, a digital version of their face to replace someone else's face with. Dr Stuart Bender: I feel like there's three potential concerns to this. The third one, I think, is a legitimate concern. The first two I'm unsure about, and I'd love Brendan's input on it, because the first concern that often comes up with these is that average people at home might make deepfakes as a form of bullying.
Now, can the average person at home access this technology in any way to do anything convincing yet? Dr Stuart Bender: And I guess another concern would be that average or even above average bad actors and state-sponsored PSYOP programs might make deepfakes of political leaders and other public influencers. Again, how likely is that? How achievable is that? Or is that something that's coming 10 years from now or two years from now? So those are questions I'd love to hear Brendan's input on. Dr Stuart Bender: And then I have a third concern that I think is more legitimate, which is that I guess that the public awareness of these things might lead to an increased distrust of what is a real image. Is this image of this person doing something legitimate, or is it a deepfake? Brendan Seals: On point one, I think thankfully for now, the technology does require a little bit of technical prowess that I think the average computer user is not going to have. Additionally, it does require technological resources that the average computer consumer would not have access to, both in terms of the power of their computer and the time they're allowed to have their computer commit itself to processing the deepfake. Some deepfakes can take three weeks to process and output something convincing.
So I think in that regards to it, we're in a bit of a grace period there. However, I would say that it won't be long before these become wrapped in very user-friendly interfaces and offered as apps on the App Store. In fact, there's already derivative versions like that. So I don't think it's far away, unfortunately, in terms of the average consumer being able to do this sort of thing.
Brendan Seals: Point two, I'd say there's probably a case to be said that they could be doing it already with their resources and money. They could leverage any knowledgeable team around the world to do very, very convincing photoreal deepfakes of political figures today. I wouldn't say this is a case for the future.
Whether they would actually want to do it, knowing the ramifications of such a thing, I'm not sure, but certainly they have access and the ability to do it now. Sarah Taillier: What can you see as some of the concerns in that space, Brendan, in terms of the risks for, you mentioned trust, but in the visual effects scene, in your world? Brendan Seals: Well, thankfully, glass half full, we've only seen really the positive use cases of it at the moment. For example, it's really helped speed up certain workflows that are very time-intensive and very manual. We have seen a whole explosion of period drama and history revisiting type sequences, which I think are really special to behold. Obviously, within that category of visual effects, there's a lot of conjecture about what's right and what isn't right.
Brendan Seals: Let's say in the case of The Mandalorian and [The Book of] Boba Fett series, where we're getting to see Mark Hamill on screen again as a young Luke Skywalker, which is pretty special to see. But of course, there's the flip side where we are resurrecting those who have passed and it's a very complex topic. And I know in many cases, the estates have approved such a use of the likeness, but it's a very strange arena to wrap your head around. But certainly in terms of visual effects, it's a very exciting technology and I kind of see where it's going. Actors actually getting ahead of this and capturing their likeness in certain periods of their life so that they can extend their careers. Sarah Taillier: That's a really interesting concept.
Are you already seeing that taking place? Brendan Seals: Yes, yes. I mean, it's, like I said, an explosion of use cases. And I think like many groundbreaking advances in technology, it's opening up directors and filmmakers and screenwriters to really novel ideas behind it.
We're seeing biopics like Elvis, which we contributed work on, which in the early days, this was very much looking into that technology. We have used it on Marvel films for stunt sequences where you would never know it wasn't the primary actor doing all those stunts. Before deepfake work, that was a very hard thing to do, very manual.
Brendan Seals: But yeah, I think the future is a little bit nerve-racking with its use. And I do share Stuart's concerns, particularly with point three. As a father myself, how it will be used in this next generation of children who are so equipped with this technology at their fingertips. Sarah Taillier: Thanks, Brendan. We're just going to take a quick break.
We'll be right back after this ad. Jessica Morrison: Whether you're optimising energy consumption or exploring the depths of the oceans, we invite you to work with us to find realistic solutions for your business or industry. Curtin University can provide assistance from product testing and benchmarking to investing in new technologies. You'll have access to specialised facilities and expertise backed by groundbreaking research. To find out more, visit curtin.edu/workwithus.
Sarah Taillier: And we're back. Before the ad, Brendan, you were talking about the impact on the next generation. What about the next generation of visual effects artists? I'd love to hear from the both of you, starting with you, Stuart, about the advice you would give to new artists who are listening to this podcast. Dr Stuart Bender: Well, in the Screen Arts program, I still see students coming in that have an interest in visual effects not as their particular area that they want to go into, but they get inspired through some of the work that we do and they start to say: "What else can I do after this?" Dr Stuart Bender: But what's interesting is this connects to a point Brendan made about some of the visual effects workers have been inspired by the Marvel films and so on. The students that I still see are still inspired by Star Wars, and they still want to do a visual effects class because they want to make a lightsaber and they want it to be footage of them holding a lightsaber.
We still see a lot of that, and that was never what interested me in getting some skills in this area and then moving into teaching and so on. But we still see that. So maybe I guess for the next generation, keep watching Star Wars, keep being excited about that, because it does get you in the door. Dr Stuart Bender: But I think the greater imagination that you can develop and the greater your ability to embrace, I guess, your creative possibilities so that you're not locked into, "I've got to point a camera at something, and that thing in front of the camera has to be what's in my final film." Think outside.
What else could be added to the frame or what could be taken away from the frame? What could be changed about the frame? We start to see student filmmakers now understanding that they used to be limited on maybe props or maybe they want to have a scene with a police car, and that was very difficult to do, so they try to imply it and have it take place off screen. And now they start to see: "Well, actually, if I can get my hands on one police car, I can replicate that digitally and turn it into three, four, or five or ten, and make a much more exciting sequence." Dr Stuart Bender: So I think the more that you don't feel limited by whatever you can stage in front of the camera and just start to think of other opportunities, other things that you want to do. And think about what got you passionate about making films, because it's unlikely to be that you just wanted to point a camera at a person. It's likely to be you wanted to construct an exciting story, an exciting scene, and some sort of ... I'll use the word "fantasy", but I don't necessarily mean The Lord of the
Rings–type fantasy, but some human fantasy you've got that you want to see captured on film. Sarah Taillier: What about you, Brendan? What tips would you give to future visual effects artists? Brendan Seals: Yeah, well, in my role at Luma, I've been here 10 years and I'm fortunate enough to not just supervise the work, but do a lot of the interviewing for the next generation of talent coming through. Just in the 10 years that I've been doing that, I've seen the education sector and the tutorials and content for education just get out of control in terms of what you have access to. I think what that's created is a little bit of a generation of artists who have lost touch with the fantasy, a bit like what Stuart was saying, and why are you doing it in the first place. There's so many tutorials online that we are now seeing just droves of students reproducing the outcome of the YouTube video or the outcome of the Vimeo link.
I think what we need to bring it back to is, yeah, there's so much out there that you have even free to study and learn, but at the end of the day, what we're always going to be looking for is the application, the concept. Brendan Seals: I'm certainly never looking for button pushers. I'm not looking for someone who can tell me that they know how to move a slider.
In fact, in many cases, you may not know a software that we have at our studio, and that is the case for many studios. That's kind of the reality is that when you do get into the industry, you do tend to get into the seat of a pipeline that has proprietary software and tools. So at the end of the day, it doesn't matter what you knew on a purely objective level, it's really about the subjectivity that you can bring, the craft that you have, and the imagination that you have.
So I'd say, yeah, sure, learn the software to a point that you can start to bend the rules, but try to come back to original ideas. And I think that's always going to be what gets your foot in the door and gets you noticed. Sarah Taillier: Stuart, over to you for a second.
You've already achieved so much in your career and unpacked some very fascinating research. What's next for you? Dr Stuart Bender: Well, thanks for saying that. I now plan to maybe move back a little bit more to I guess my original passion, which was English education in high schools.
But I'm very interested in exploring what can we do next in terms of critical visual literacy for students, because there's a lot of concern and talk at the moment about information, misinformation, deepfakes, fake news, these sorts of things, and it will ultimately fall upon English teachers to be asked to do something about this. Whether it's their responsibility or not, they'll ultimately be the people that are asked to do something about this. So I'm interested in, I guess, doing some action research and some theoretical work, looking at what can teachers do functionally in this space to actually prepare students' critical visual literacy skills to encounter what is being thrown at them on a daily basis through digital media, and to turn some of, I guess, the theoretical and experimental work that I've been doing over the last 10 years back to feed into what can education take from that and what can we do for the next generation of citizens. Sarah Taillier: And for you, Brendan, what's next? Brendan Seals: Actually, this year post the craziness of COVID and Spider-Man and all the other films that we've been working on, I've actually been, would you believe, upskilling and taking a look at Unreal Engine 5.
So I've been really interested in seeing how the games industry is really influencing what I perceive to be the next big change and shift in the visual effects landscape. Brendan Seals: And we've been employing the technology in the last three films. And with each film, seeing just how much it can produce faster, better results and get us closer to a real sense of craftsmanship that ... It's otherwise just been a very long, intensive task in our industry, and I'm really excited about being, I guess, closer to that. As a supervisor, being quite time poor, I'm enjoying being on Unreal Engine and playing with really beautiful imagery in real-time. I think it's a really exciting arena.
And obviously, it's the technology that's driving LED screens, so I'm fascinated about that as the outcome of it, but I can't wait to see how it really takes over the visual effects industry in post as well. Sarah Taillier: Looking forward to watching the space for both of you. Thank you, Brendan and Stuart, for coming in today and giving us a feel for a really interesting world that's playing out on and behind the scenes in such a rapidly evolving space. Dr Stuart Bender: Thank you.
Brendan Seals: Thank you. Sarah Taillier: You've been listening to The Future Of, a podcast powered by Curtin University. As always, if you've enjoyed this episode, please share it, and don't forget to subscribe to The Future Of on your favourite podcast app. Bye for now.
2022-10-20