Google I/O Part 2: Show Logistics - Day-of-Show Technology (VMIX), Streaming , Captioning
(upbeat music) - Hey everybody. Welcome back to another episode of Tim and Tim Talk. As always, I cannot do this without my wonderful co-host Tim Kerbavaz. - Hey Tim, great to be back. What are we talking about today? Are we continuing our saga of Google IO? - Yeah, that's exactly what we're doing. We are, this is, if you missed episode one, please do us a favor.
Go back and find that previous episode, because we give you a high level overview of a project that we both worked on, Google IO, the largest developer conference that Google puts on every year and 100% virtual. So we talked about why we made over, like high level decisions. And in this one, Tim, let's talk about, let's go into the nitty gritty, to the details here. - Yeah. So let's start at the beginning of like the (mumbles) process.
This is where the video comes from our presenters, as I knock my microphone outta the picture, video from our presenters into the studio, what do we use for this? - By the way, that's what happened with so many presenters, AV challenges. Mics getting knocked all over the place. This is, thank you for replicating a real world environment. - I'm showcasing what we had to deal with here. So what do we use to bring people into our studio? - So this is, if you've listened to our previous podcast and video podcast, by the way, we're on video, audio, it's fantastic. Find us everywhere.
So, yeah, and like, as we talked about earlier, is that the best technology and the best tools are not necessarily the best broadcast tools, they're the best tools for your specific client. So for this project, this being a Google project, Google Hangouts was absolutely a no brainer, slam dunk. - Google Meet it's called now, Tim, they changed the name, remember. - That's right, that's right. It is now, thank you, it's Google Meet these days.
I think when we started, it might have been Hangouts. - I think it was, I think it was. They seem to change the names of their products every couple years (chuckles). - They really do, and thankfully though, they did some good upgrades with Meet and it really made for not only sort of a great solution, but the presenters use this tool in their normal workday. And I think one of the biggest compliments that we received without them even knowing that they were complimenting us was like, "Oh, so where do I go? What's next?" We're like, "That's it, you're here.
That's it, we'll take everything out from here." So let's talk about that. Let's see here, the client needs their, so yeah, once they were in, Tim, what did they do? - So we wanted this to be, as you said, a familiar platform experience for them, they just join the Meet and then we take it from there. So, Google Meet is at its core, a conference calling tool, but it's a video conferencing tool. And so we were able to use a process that we've developed over a couple years of using Google's tools to do production, to essentially extract the video and audio and getting each person as a separate video feed into our video switcher so that we can then composite those in the ways that are needed for the show. - If producers of other events, it's a very similar technology, excuse me, technology, or the way that we extricated the video using Zoom.
- Yeah, it's a similar workflow. - Similar workflow. - I mean, you could even do this with like WebEx.
I mean, I don't know who who's still using WebEx. There are still companies that are using WebEx, if you're using any of the number of BlueJeans, WebEx, GoToMeeting. You can make this technology work or this workflow work, but certainly with Meet, with Zoom, we have specific experience and tools to make it work really well. And basically, a technology stack that we're building in our studio to really make sure that we're getting the best broadcast quality, audio and video out of these tools that we can.
- And that was part of the checklist, right? So we made sure that they were sending full HD as possible, making sure that the right microphone was selected. And as you can see, and that's what we refer to as our station over here. You also see, share some pictures of actually, of it live. And this is our inbound station. So essentially like over here, it comes in and then we split those out into individual challenges and sources.
- Right, so we've got those, we got those four or more presenters coming into the studio. But one of the other things we had to bring in for those workshops was screen share content. And we teased this last time, at last episode, Tim, we had a plan, like a really great plan for how we were gonna get the video, the screen share video in. - OBS, right, OBS is a fantastic product. It is a very robust product, and the nice thing is it's absolutely free to download. What did we talk about Tim, in the first episode of this deep dive? Downloading is a problem.
- Downloading is a problem. We had a couple of challenges here with OBS, OBS, Open Broadcast Studio, for those who aren't familiar with it. It's an open source streaming application, is used by a lot of focus who stream on Twitch and things like that. And it really has a really easy way of sharing your screen. But the big challenge was.
- Is it an easy way to share the screen? - Well, easy for us, I guess. That was the issue, right? That was the issue. - I'll show you some of these times where it took an hour. (mumbles) - So we had these aforementioned rehearsals we talked about, we were talking folks through how to install this in their computer, how to share their screen.
And it was a chore, to get folks on board, getting them sharing the screen, getting it set up. And then of course it was gonna be a week or two before the show and they were gonna reset it all, not a working solution. The other challenge, Tim, this is Google, remember, and a bunch of folks were joining from Chromebooks. - And in case people don't know what a Chromebook means. That means it's not a MacOS, it's not a PC OS. It is Chromebook OS.
- It's not even like a regular Linux install, right? And OBS will actually run on Linux but it's literally just a Chrome browser. That's all you get on that computer. And so all you can install is a Chrome plugin, right? Installing software that required installation at all was a problem and the configuration, but also because it wouldn't work on the Chrome devices. So we ended up using a product at the time, it was called OBS Ninja, it's now called VDO.Ninja, which is a browser based point to point screen sharing application, essentially, that let us share their content directly into our studio. What were some of the ways, the benefits we got from that? - Well, benefits, but we're also nervous about using this product and distributing this so widely.
It's a free tool out there, which is fantastic, but also means that we don't have control over its ecosystem, we don't have control over its servers. There's other ways that we thought about, but definitely we were like, "We're gonna go with this. This is our choice. We're gonna make it, we're gonna own it." And to its credit, it was absolutely fantastic.
And what we loved about using this tool is that we got to do the heavy lifting. We can put the heavy lifting in our technicians and not the speakers, but the output of what we received was a beautiful 1080p image at its highest quality. We're talking every single pixel mattered because this is, where they're doing coding examples. And if you can't read what they're writing, then it's a frustrating viewer experience.
So we can deliver this. And we don't care what computer you're on. If you were using OBS, if we were gonna go that OBS route where we installed, and we walked through it, all that information with you, and then something happened to your computer, or you now changed your setup, we were in trouble. Here, it's just a URL browser link.
It made it one click, so simple. - And I think we should back up here a little bit, 'cause we didn't mention that of course, Google Meet has a screenshare function, built in. It's built into the tool, the challenge we ran into was it wasn't high enough resolution. It's really designed as a conferencing tool for sharing PowerPoint slides. And so when we were testing it with the Google Meet screen share, it was often very hard to read the code specifically in those examples. And so these other tools let us get 1080 video, that said, we still had Meet as a backup and as a tool to help walk folks through the setup.
So we would have them join Meet, share their screen in Meet, walk them through the setup. And then we essentially had a backup feed as well, in case one of these other tools didn't work. - And normally in this environment, this is a multi-desk setup. A conference and your presenter, multiple computers are being set up. But now the fact that we can use OBS Ninja, what is it now called? - VDO.Ninja. - VDO.Ninja, fantastic.
Now that we're using VDO.Ninja that's our primary source. And then like you said, we can use either, the screen share function as a backup or it's perfect for sharing slides.
And so oftentimes presenters were bouncing between their slides and their live demo. We had full control of it, they had to do nothing. So it's actually a really, really pleasant user experience. Again, Tim, zero downloads. Let's keep emphasizing that, zero downloads is the key to success.
- And again, using tools that are easy for the talent to use, make them comfortable and are (mumbles) with what they're already used to. So we've gotten now the cameras into the studio, we've gotten the screen sharing into the studio. So that's our inbound position, right? That we called bring the studio in. We talked a little bit last time about the life Q&A, but I wanna talk- - Actually, Tim, before you go to the Q&A, let's one important thing with the audio that I wanna mention. - Oh yeah. - Is that the best tool is essentially a broadcast tool.
But as we talked about earlier, we just didn't have the ability to use a broadcast tool at scale for everybody. And the one tool that we were missing for Google Meet was volume control, specifically volume control within the browser, but we found a sort of nifty way to solve for that and essentially turn Meet into a broadcast tool. - Right, and so by using some plugins, both in the browser that we were bringing the video in, as well as in our audio mixing solution, we were able to adjust volume individually and really make sure that we were getting good quality audio from each system, consistent audio, exactly, from every person. And so we've got all these folks, we've got a panel of folks talking, answering questions. And we again, we talked a little bit last time about how we managed that Q&A, the documentation we used, but let's go in a little bit deeper about what that looked like, how we used Adobe Premiere, which is a video editing software to bring those questions on screen and what that work would look like. - Absolutely, so Adobe Premiere being an editing software, credit to the Google team because they showed us this trick, which was absolutely nifty.
There is an NDI function in Adobe Premiere. And so essentially using the MOGRT function, you can use Premiere to render out live Q&A, now there is a little bit of render time. And I mean a little bit, I do mean just a little bit. It ranges from five seconds to 10 seconds or 15 seconds. So we did have to prep the speakers to say please don't answer yes and no, if you do, that's fine. We just won't be able to show you the next question.
We need a little bit of time to render it. So now that we have the question rendered. So how do we get that question there? You have to think about the workflow and the communication flow that's taking place. There's a project manager that's managing the questions on the Google side. That's bringing it from one platform into a doc because the speaker's doing all of this live.
And so how do you know, keep that person, the host engaged with the panel, but also while communicating to the live Q&A and getting these questions. - And we should say, these questions are coming from multiple places, right? There's Q&A in live chat on the platform. There's questions coming from Twitter. There's questions that were sent in advance. Folks had sort of seated questions with questions that had been asked online in the past for past events and past questions. And so we've got essentially sort of multiple streams of question information coming in.
And as you said, the presenter, the host of the Q&A session, can't be looking at like five different places at once, trying to decide what questions to ask while also looking into the camera and being engaged with his or her fellow panelists. And so with, as you said, having an outside moderator, someone who's not on screen, essentially, project manager taking these questions from all these places, creating an order. - Centralizing it.
- Centralizing it in this document, that Argus created for this purpose. And then as you said, last time. - Let's show that now, actually, let's bring this up to screen.
And so as you can see, this is a very, it's essentially a bare bones. It just tells you only the information that you need to know. Next, loaded and aired. So as you can see in its default state, there's zero colors involved in this.
And this is because these questions are just being sent by the moderator to this doc. And now the host gets to decide how they're presenting the questions. Because one thing we learned through this process is that different hosts basically communicate differently.
Some folks just like to do a question, they like to go top down, and in that order. Some folks like to group questions together, they're talking about a specific feature set. They like to talk about all those questions. So question that might be the next relevant question could be down 10 lines. So what this document basically allow us to do is do a next, once they click next, our technician now knows to load that into Premiere. So now they communicated to us.
Now we need to communicate that to them. And specifically, once that render's done, once our render's done, we're gonna click loaded. Green means go. How simple is that, it's very simple, which is why it was successful. Once they asked and answered that question, we just switched it to aired.
So they now know that that question went to air and rinse and repeat. You have a very simple custom created document that, what does it do, Tim? - Right, gets the jobs done. It uses, like we said, tools that the talent are familiar with. Simplifies communication and really just makes everything appear in one place.
And again, that kind of simplifying unifying views, creating dashboards really helped make this show possible. And speaking of those dashboards, we had those dashboards for every position. Let's talk a little bit more about what positions and crew positions we had on this show, who was doing what, where. - Yeah, absolutely. And this is one that we discussed like how do we distribute the workflow evenly? And so let's talk about, so the positions, but also every studio was essentially live going back to back.
So while a studio was live with one segment, there was no rest for the weary here. We went right from one segment to the next segment, which we had, it turned it around, what, 10, 10, 15 minutes. Sometimes even less than that, right? - Yeah, right.
- So the way that we accomplished this was we like to use the terminology inbound station, basically any feed that is coming inbound to our control room that is virtual. And so, as you can see with some pictures here on the screen, this is our inbound station. This position brings in the presenter and also their content. We're walking them through the experience, we talked earlier, I believe in the previous episode about our preflight checklist.
This is where we're going through with the preflight checklist. And we're making sure that all the steps are done. Tim, you wanna take, what are we doing next? - Well, I was gonna say we should mention that with this, we had this station because we had these back to back presentations. We actually had two inbound stations per studio. So when inbound A is on air with talent presenting live, inbound B is getting the next set of talent ready to go. So while the panel or the presentation is happening on inbound A, our inbound technician is onboarding the next set of presenters during the previous presentation.
So that they're ready to go, and then. - This is where crew training was so important where we emphasize and doing a checklist so important and emphasizing this right. - And unifying that experience for the talent and the crew. And then we basically set up the studios using a video router to flip flop the inbound stations when we changed presentations. - Let's talk more about that because you made it sound just so, hey, flip flop.
That's so easy, but I mean, that one button did a lot, like this is a deep dive, Tim, talk about that. When we hit one button, what happened? - So we created a studio control system, essentially, where we had using an Elgato Stream Deck, a button panel. I think I can't find one right here, it's buried, but a button panel that let us essentially program macros, basically, where during that changeover, there was a button on this button panel that rerouted inbound A inputs, rerouted inbound B inputs into the switcher input.
So essentially the video director, the technical director who's switching the show has one set of inputs always consistent in the switcher. And then we essentially use this preset to change the routing in a video router to remap the second set of inbounds to that input. And essentially with that button press, that switch. - How many like different commands do you think that was? How many button pushes? Probably a dozen button pushes that were truncated down to one button push.
- It's switching multiple different changes in the router, we also had buttons that started and stopped the stream. Buttons that controlled what the presenters were hearing, we had a whole system for talk back that allowed the director and the producer to talk to the talent, essentially what we call IFB, interruptable feedback, where we could essentially speak in their ear without the audience hearing. And so using this control system on the stream deck, we were able to have buttons for routing what they were hearing, routing the video in and outta the switcher starting and stopping the stream.
Even things like queuing the graphics. We talk about the Q&A graphics and the system we used for that. We had buttons to trigger those on and off screen on this button panel. So this button, single set of buttons was controlling, essentially five or six different pieces of technology and software tools. - And essentially this is, as we talked about earlier in episode one on-premise versus cloud, needing this robust switch over, we just found it more comfortable to do this on-premise, especially this switch over here. - Well, certainly these tools, you can have this control system for online tools, and it's not outta the question to have macros and things like that in the cloud environment, but it just was, as you said, a comfort level thing and comfort for us, comfort for our crew, tools that everyone was familiar with.
And speaking of, kind of those tools and positions, I think I do wanna get into kind of who was doing what. So we talked about having a director, right? A technical director, someone who's actually pushing the buttons on the switcher, who's switching the show, who's getting that content to air. And so that person was sitting in front of one of two different kinds of switches.
And I think we do need to acknowledge that we had two different sets of switching technology in the studio. - When it comes to equipment, we're device agnostic, we're manufacturer agnostic. We simply want to use the best tool for the job.
So in our control room, right behind me here, this is our primary control room. We're very happy with vMix and we use vMix for multiple machines. We have a custom built PC. We'll go into that.
Another episode of like all the specifications, it's completely overpowered that we're only using 10% of it at max most of the time. But also for the AMAs, the ask me anything and the panelists, we have TriCasters. NewTek does a fantastic job with their switchers.
And so we found it just being a little bit easier if we assign TriCasters to AMAs and vMix to the workshops. - And part of that too. I mean, the challenge for us is assigning the crew to the right technology that they're most familiar with.
And so we actually had, we divided the crew in terms of like those director positions, who was most familiar with a TriCaster, who was more familiar with vMix and make sure that they're in the studio that corresponds to their strengths. And that we're sort of shifting folks around. And the inbound stations were pretty consistent across all of the studios, but the TriCaster versus vMix was a big distinction, but even other things between those AMA and workshop, that checklist was different. And so we wanted crews, we actually remapped the client schedule to the appropriate studio for those sessions, so that we could really unify one type of show in each control room and we spent a lot time- - And that's really interesting, like to talk about, because the way the client schedule looked, it essentially went AMA, workshop, workshop, AMA, workshop.
We discovered early on, if we can send all AMAs to a specific crew, the cadence of them doing it over and over is gonna be, it's gonna be a much better show. So it took us a lot more work to figure out a schedule. And how do we move these around between these controls? How do we juggle the AMAs and workshop and distribute it? We did a pretty good job of sending 90% of AMAs to one studio, 90% to another studio. And then we did have a studio that bounced between back and forth of them. But we specifically trained them to be prepared. - And it helped with that, crew training helped with the equipment specification and it really helped unify the flow of our event.
And so speaking of that flow of the event. In every room, every studio, we had a producer. The room producer who was in charge of calling the show, but also really directing the orchestra there and making sure that all of these positions, other positions in the studio were all rowing in the same direction. Let's talk a little bit about that, Tim, what that producer did? - I like that, rowing in the same direction, because it's so true.
There's a lot of oars in each control room and they could have easily gone in many, many different ways and that's right. You essentially need to have one person sort of be the voice of leadership. And in that room, it was the producer. And so the producer had no technical responsibilities, but they had all the responsibilities specifically to, okay, we have the callers in now, where are we going next? Do we have the right person in, is the right person in the right room because we scheduled it and we built the calendars, it was pretty, that one was pretty efficient.
- We're talking a lot about all these positions, we're talking about this kind of orchestration of the show and really the position that was in charge of that orchestration was the room producer. And we had this producer in every room whose job was to make sure that all of the crew were rowing in the same direction, making sure that we're all going to the same end goal. - Rowing in the same direction, such a good analogy, Tim, because if you could imagine every individual technician rowing their own direction of what they think is the best direction possible. - Totally, and I think there's a lot of oars in this room, gotta make sure they're all going towards the direction of the show. And so this person really wasn't focused on technical execution. They were focused on looking a step ahead.
Their job, yeah. - Yeah, I mean their job, if you think about it, essentially, they're the air traffic controller in the control room. - Totally. - They're not necessarily pushing the buttons, but they're making sure they're routing, they're making sure the right plane is going to the right runway, taking off at the right place. And that there's no collisions because you wanna make sure you're sending the communication to the correct inbound station, that's not on air, disrupting potentially what's on air.
And then on top of that, Tim, and I think we'll talk about this later. One of their most important jobs was each stream started and stopped every single time. - Totally. - So are they pushing it to the right channel? Because we haven't talked about this much yet, but every segment went to a different product channel.
- And the big challenge with that is we have dedicated links for each event on different channels, different systems and really what if you start streaming to the wrong event, it essentially burns that link. - You burn it, you burn it. - Right, and so the producer, one of the producer's jobs was to, in addition to checking that the talent were in the right place, that everyone's ready to go before we started. To double, triple check, are we on the right event, the right place before we start streaming? - So Tim, you can see now how by having the Q&A doc and the MOGRTs, how that system works really efficiently. Okay, so we just talked about the inbound station, Tim, let's go to the next chair, director. Let's talk about the director.
- The director. So the director's job or technical director in sort of broadcast (mumbles) is to control what the audience sees. They're operating the switcher equipment to really switch between all these various sources that are coming in from inbound. And so we had all these different kinds of content.
We had the AMAs that are really a panel discussion, you got four or five people on screen having a conversation, and they have the workshops that are one or two people giving a presentation, really heavily reliant on screen share content, on slides, on live coding demonstrations. And so we had these kind of two different needs. And so one of our challenges was to kind of divide this content between different studios so that we could really focus the crew, both the director, but also the whole team checklist process on one kind of events. - And I mean, I think we sort of discovered, not discovered, but through our experience in running these productions that we found that.
First we're device agnostic, we're switcher agnostic. We want to use the best tool for the job. So is that a Ross Carbonite? Is that a Blackmagic switcher? We love those switchers, and we have those. And the switcher behind me is vMix. And this drives our master control room because we found it to be the most robust and flexible for just a wide variety of needs. But we also found out that during the AMAs, that TriCaster was gonna be the best route for the job.
So with that, we also now need to find the best technicians that know how to drive that. The best tool is only as good as your pilot, Tim, keep coming back to this pre-flight. It's just so important. They need to know how to use the tools.
The tool is only as good as the pilot. - Totally, and so by dividing up this content into the different studios, by putting the appropriate hardware in each studio, by putting the appropriate operator in front of that hardware, we were really able to kind of streamline each of these studios for the particular content we had there. And now, we've got the director switching the show, we've got the inbound station bringing folks in, we've got two inbound stations. We've got a inbound that's live and an inbound that's sort of on deck to bring in folks that are coming up next because the schedule is so tight. - The green room of it, essentially.
- Exactly, exactly the green room. But there's one other person in this control room and, and they're kind of orchestrating this whole, the conductor- - Kind of a big deal. - Yeah. - Not only are they the orchestra, Tim? May I use yet another air traffic control analogy? They're the air traffic controller. I just ruined my own analogy, but essentially. They are the ones that are coordinating, who's taking off, who's landing and they're making sure that no collisions take place.
So let's take that to our control room. And what does that mean? They're making sure that inbound A is on air, that we're not manipulating inbound A in any way, because it's live, that we're working specifically with B. So they're making sure that the audio and video channels are closed to the stream that we're rehearsing or that we're tech checking. I mean, such an important job. And also they're making sure that the right person is in the right control room, that we have all the right people.
When we're dealing with panelists of four or five and they're coming in multiple times per hour, we have to kind of track them down. So it's the producer's job to know, "Hey, we're missing presenter X, Y, Z, let's move this up the chain and alert potentially the handler to say, 'Hey, can you please reach out to them?'" We also have the phone numbers. The producer had the phone numbers to each person, which we gathered in tech checks, just in case they had any problems. So it was a really, really important dispatch job. - Their job is to be one step ahead of the production.
They're looking at the checklist. They're looking at the show flow. They're looking at what's coming up next. They're not focused on the technical details of what button to push when.
They're focused on what's happening, in what place, at what time, they're making sure the whole crew is rowing in the same direction. And they're looking ahead across that bow of the ship and seeing what's coming up, where are we going to be at the next step of things? - I see that you don't want to use an airline analogy for that, Tim, you want to use a rowing technology, but what a great analogy though. I mean, that's so true because everyone can row in a different direction and it's so easy unless there's clear and concise communication amongst the crew. - And, Tim, I'll say there are a lot of oars in that control room.
So you gotta make sure they're all going the right way. - There is a lot. - So speak of a lot, Tim, we've covered a lot in this episode, but there's so much more to talk about. We've talked today about what's happening inside the control room, but there's a lot going on once that video leaves the control room, what do you say we do another episode? - I think so.
I mean, this is a deep dive, we went deep. Let's go even deeper in the next one. So with the next one, Tim, let's, like you said, everything that happens outside, but what is, what did we take outside of the control room? What haven't we talked about? Let's tease the audience a little bit.
- Yeah, so we've got the video being switched on that equipment on the vMix, on the TriCaster, leaving the room. Where does it go? We've got records. We've got a master control situation where we're able to monitor all of the studios, all at once and manage what's going on air. And then we've got YouTube. It has to hit the internet and go to the stream. And so we've got those areas to cover.
- Layers of checks and balances, layers upon layers of checks and balances. - Layers of people who are checking those checks and balances. And then I think we're gonna do another episode again next, after this talking about some lessons learned, 'cause I know, you and I learned a lot on this show. There's a lot of things that we will do differently next time.
And so giving us the opportunity to really share with you some of those things that we'd change when we do it again. - Let's do it. Also, if you found some value out of this, if you think that there's any good tips and tricks, if you're trying to build a production or know a producer that's trying to build a production, feel free to share this. We love sort of our ideas getting out there and helping you and your production.
If you have a question, if you want us to go a little bit deeper, hit us up on the socials below that you see. That's @TimAndTimTalk on your favorite platform. So Tim, I think that about does it, let's wrap it up.
My name is Tim Kay, - And I'm Tim Kerbavaz and we talked, see you next time. (upbeat music)
2022-04-20 22:44
