The "VTuber" and Why Artificial Intelligence has Limits


Show Video

Let's talk about VTubers. Amygdala comes from the Greek word for  almond and it's the part of the brain   associated with recognizing emotion from  facial expressions or at least most of them   probably. It's an effective evolutionary  trait in recognizing emotions. In fact,   it's so effective that sometimes emotions  can be recognized in non-human face havers   such as animals or drawings. But determining the  emotion from a facial expression is a lot more   than just selecting one from a set. There are many  different facial structures and yet the same minor   muscular movements can determine the same slight  shift in intensity or nature of emotion. And so  

the question is how does the brain recognise and  piece together all these seemingly insignificant   details to understand what's being communicated  by a facial expression? For the past few decades,   a popular interpretation of the brain has been the  comparison to the computer. Both of them after all   are things that have parts in them that do  things. In conventional computer software,   a simple program follows a sequence of logical  steps. To find a red dot in an image for example,   you can check every single pixel until you get to  a pixel that is red with no red around it and to   find the face and all of its features you just do  the same thing but replace a red dot with a face.  

But people are different coloured pixels and even  if we weren't there's no difference between this   pixel and this pixel. So maybe instead we should  look at the combinations of pixels - at the edges   of the image. But even then there's really very  little difference between this edge and this edge   and so maybe instead we should look at the  combinations of edges - at the shapes. In 1964   Dr. Woodrow Wilson Bledsoe published a report  on a project for recognizing faces called the   Facial Recognition Project Report. The goal of the  project was to match images of faces to names and  

one of the strategies to doing this was first  looking at the features of the face. Features   can include key points such as the hairline, the  corners of the eyes, or the tip of the nose. These   features can then be combined and the distances  between the features are used to recognise and   classify the face against many faces in a dataset.  Of courses not all faces are always directly  

facing camera. Some of them are facing to the left  and some of them are facing the consequences of   their creation. To correct this, mathematical  transformations were applied to the distances   to face the face face-forward. Unfortunately,  not much else is known about this project due   to confidentiality but it is the first known  significant attempt at computers processing human   faces - so long as those faces are at a reasonable  angle and lighting and age and isn't wearing a   hat. Ultimately, due to limitations in technology,  the project was labeled as unsuccessful but it did   highlight a problem in processing faces in that  faces were just too different from each other   and different from themselves in different  settings. Any singular hard-coded algorithm   had to be very convoluted with potentially a  lot of error but that's okay though because   this was in the 60s so they still had plenty of  time to figure it out before I make this video.  

VTuber, short for virtual tuber short for virtual  YouTuber is a YouTuber that is virtual as opposed   to regular YouTubers who are authentic and  genuine. Being YouTubers, they are most commonly   found on Twitch and have the potential to be as  various in content as any other content creator.   In other words, swaying back and forth with their  mouths open. VTubers take obvious inspiration   from Vocaloid performances such as those  of Hatsune Miku but unlike Vocaloids there   isn't a singular history of VTubers since 1964 as  VTubers are not a singular organisation or even   a singular technology. Most VTuber historians  like to start the history at Kizuna AI though   who is in every sense of the word a virtual  YouTuber and the first virtual YouTuber if   you don't count Super Sonico or the Weatheroid or  Annoying Orange. Ai is the Japanese word for love  

but it also has the obvious second meaning A.I.  - the abbreviation for artificial intelligence.   It could also stand for Adobe Illustrator but it  does not because that is too many puns and also   would not make sense. The character proclaims  herself to be an artificial intelligence and by   artificial intelligence she means animation  software with a production crew but how   exactly the human operator and voice are mapped  to the model has been mostly left to speculation   due to confidentiality. It's also noteworthy that  unlike most traditional YouTubers she's somewhat   of a corporate mascot under Kizuna AI Inc made  apparent by her involvement with commercials.  

Nonetheless, her character and her format  of 3D animated voice acted video productions   is the first usage of the word the VTuber and  so it's a good place to start. Unfortunately,   Kizuna never left behind a formal definition for  the word VTuber and so its definition at least   as a medium has been left fairly open-ended. If  you take the definition to be the online video   form substitution of one's identity by animated  character then the concept of VTubing is not   very novel nor uncommon. On one hand, you have  static expressions chosen from a set of pre-drawn  

illustrations or PNGTubers and on the other  hand you have full body motion capture devices   connected to a model within a game engine such as  that of CodeMiko. These two extremes sit at either   end of the spectrum from most affordable but least  immersive to most immersive and least affordable   but they both solve the facial recognition  from image problem in the best way possible:   by not solving it. It's much easier to find a  dot on a screen than a face so what if we made   people's faces into dots or rather multiple dots.  In 1990, Lance Williams decided to track faces by  

not tracking faces and published a paper on  Performance-Driven Facial Animation in which   retroreflective dots that were easy to detect  by a computer were applied to a person's face   to be tracked and then mapped to a 3D model.  Williams performed this on himself for budgetary   reasons and not because he wanted to become  an anime character. This would be one of the   first instances of marker-based facial motion  capture for animation: a technique that can be   held accountable for The Polar Express. But it's  unreliable and bothersome both of which are bad   things and so it has nothing to do with the video.  If we ignore all the other parts of the body,   CodeMiko's facial animation uses iPhone  X's FaceID. By using projectors to project  

a light onto the face and then sensing the  depth of this reflected light using sensors,   a three-dimensional image is created thus  avoiding the problem of angle. And since the   projectors and sensors are projecting and sensing  infrared light rather than visible light on top   of the infrared light that your body radiates  naturally, the lighting around the face does   not affect the image. The entire solution is  thus in the hardware and it works pretty well   even on faces that are wearing hats. However,  how exactly a three-dimensional depth map is   achieved from a face with lights on it is  something that we're not going to get into   because hardware is scary but mostly due  to confidentiality though it doesn't take   an Apple engineer to make the observation that  light patterns distort themselves when reflected   on three-dimensional surfaces which could help  indicate the shapes of those surfaces. Apple's  

FaceID remains dominant in the IR camera facial  mapping market. Google's Pixel 4 had a similar   system called uDepth which used a stereo depth  sensing system otherwise known as two cameras   similar to how you have two eyes to sense depth  but this was discontinued and the other one is   Xbox Kinect. All of this wasn't developed just  for Apple's primary demographic of VTubers though.   The main selling point of FaceID is its biometric  authentication system and also Animoji. But where  

VTubing comes in is the tool that Apple provides  to developers: the ARKit. Developers can build   apps around this tool such as Live Link which  feeds the facial data directly into Unreal Engine   which is what CodeMiko uses. But what if you can't  afford an iPhone X or just despise Apple? Surely   there's another way to VTube from your webcam  or camera. In fact, it's probably the technology  

you've been thinking of since we brought up brains  and facial recognition. Microsoft Excel has a tool   that allows you to draw a trendline that best  represents a scatter plot. Most data is probably   not linear but a linear line can still be used  to predict y values given x values. Of course,   this prediction could just be terrible and so  Microsoft Excel has to minimalise the distance   between every single point and the line to find  the line of best fit. This process is called   linear regression. Linear means relating to lines  and comes from the word line and regression means  

estimating the relationship between a dependent  variable and many independent variables and comes   from the 19th century bean machine. You may have  noticed from that last sentence that there are   many independent variables. Linear regression is  useful for drawing lines through three-dimensional   and four-dimensional and whatever dimensional  scatter plots. Every new dimension is just another   variable or feature that affects the output and  predicted output in the y-axis. Using linear  

regression to predict how long a person is going  to watch through a video, the features may include   the length of the video, the age of the person,  and how much of the video is about statistical   theory. And to make predictions off of say images  of faces for example, the features could be every   single color value of every individual pixel on  the image. But making predictions of something   as advanced as an image of a face may not be  as simple as just drawing a line. A linear   fit might not be best appropriate for our  line or hypothesis to every single feature.   It might work better as a quadratic fit or  cubic fit. By adding more features or dimensions  

that are equal to the square or the cube or the  whatever of the previously established features,   we can do polynomial regression which is actually  just another type of linear regression because the   hypothesis is linearly proportional to something  that is non-linearly proportional to the original   data. You can also combine features and make new  features with them by multiplying them together   such as if you have a height feature and a  width feature you can instead have an area   feature. But making predictions off of something  as advanced as an image of a face may not be as   simple as just drawing a multivariate nth degree  polynomial. We know that we can modify and combine   features to make new features to optimally  fit our hypothesis to data but in what way   do you modify the features? Which features do  you combine? How do you even do any of that for   thousands of pictures that have hundreds of  thousands of pixels and millions of RGB values?   Who is Gawr Gura? A slightly controversial sort of  lie is that linear regression as well as all other   types of regression are a form of artificial  intelligence. In fact, if you sort of lie,   anything can be a form of artificial intelligence.  You yourself at home may already know a deal about   artificial intelligence, either from your own  extensive research and experience or your ability   to lie, but artificial intelligence isn't so much  of an algorithm as it is the idea of artificially   creating something that is intelligent or at  least seems so. Most of what you may know to be  

artificial intelligence is the method of machine  learning called the artificial neural network. A   neural network or a network of neurons is a system  of units that receive information, process it,   and pass it on to other units in order for the  entire system to make a prediction - quite similar   to the neurons of a brain. It's also quite similar  to Danganronpa in that regard. Neural networks and   all of machine learning is a big deal because it  allows programmers to do things they typically   can't do on their own such as play board games  at a grandmaster level or hold a conversation.   This is because unlike in conventional programming  where the programmer knows what they're doing, in   machine learning the program learns how to do it  on its own without the programmer really needing   to know how it was done. But machines don't have  feelings so how and what exactly is the machine   learning? The units or neurons of a neural network  are organized into layers. The first layer or the  

input layer is where the inputted features are  received. For every inputted feature there is   a neuron in the input layer. Each feature within  each neuron can then contribute by some weighting   to the features within the next layer of neurons.  The different weighted sums of all the features of  

this layer is thus the information received by  the neurons of the next layer. This next layer,   called a hidden layer, then applies some  processing to the information in order to make it   harder to explain. First, it adds a number called  the bias value in case the information is going   below a certain threshold that it shouldn't and  then it puts it all through an activation function   which is just some non-linear function so that  the features of this layer are not necessarily   linearly related to the features of the previous  ones. These newly activated features can then be   passed on to the next layer to repeat the process  and make more features off of these features.   Through this, the features of each layer are like  a combination of the features of the previous   ones: from pixels to edges to shapes to faces.  If there are many many layers that compute very  

very specific or complicated features then the  entire network can be called a deep neural network   because it is very long. Eventually, it reaches  an output layer which has the number of neurons   as the number of things you're trying to predict.  The values received here are the predictions that   the model is giving based off of the input. To  train a model is to figure out all of its weights   and biases which are altogether called parameters.  This decides how each feature fits into the next  

layer of features. To do this is just the simple  task of finding or creating hundreds of thousands   of pieces of data. The input data can be put  through the model and the predicted output can   be compared with the actual true value that was  manually determined by a human. The function that  

does this comparison and determines how wrong  the model is is called the cost function. We   can then go backwards through the model to find  out how each parameter can be changed in order   to lower this cost function. This part is called  backpropagation and if you know calculus it's a   quick way to calculate the partial derivative of  the cost function with respect to every parameter   and if you don't know calculus well it's the  same thing but you wouldn't understand. The   neural network relies on training with many sets  of data in order to improve itself with each set,   hence the name machine learning. Now  admittedly, all of that may have been   a bit of an oversimplification but it's the  backbone of machine learning and the model used   for computer vision and more specifically object  detection which would be true if I wasn't lying.   Different architectures of neural network can have  different activation functions, cost functions,   number of layers, and number of neurons per  hidden layer. The architecture for a computer  

vision model in which the input is an image matrix  is even more convoluted as it is a convolutional   neural network. An RGB image can be thought of as  three matrices: one for each RGB value of every   pixel. However, it would be a lot of weights to  put every single pixel into a weighted sum for   every single feature of the next layer. Rather,  the more efficient technique devised is to take   these matrices of features and pass them through  a filter that forms some number of matrices of new   features for the next layer. The parameters here  for the convolutional layers are the values that   make up the filters for each layer. There are  then also pooling layers that reduce the number   of parameters by throwing them away and hoping  it works and then near the end of the network we   may have some fully connected layers that are just  the same layers as before with weights and biases   to sort of check to see if there are  any relationships we're missing between   all the features now that there are less features.  Finally, the vector of features that we're left  

with is put through some regression function to  perform the actual classification or localisation   or detection or moral dilemma conclusion for  your autonomous vehicle. Just like with the   basic neural network, the convolutional neural  network or ConvNet if you're running out of time   is filtering for more and more specific features  with each layer. Also, this was once again an   absurdly oversimplified oversimplification that  is ignoring a lot of the math though this time I'm   not lying in any of the information I've given.  Computer vision is fairly big and so a lot of   research has been put into it as well as a lot  of abstractions from its bare mathematical bones   to the point where training and then running a  model could take the copy and paste skills of web   development. Given just a few hundred images, you  can train a model to perform object detection on   the body parts of Oikawa Nendoroids. Even you can  become a VTuber. In the case of VTubers which is   what this video is about, it's not actually  object detection but rather facial landmark   detection. The output for such a model may be a  number denoting whether or not there's a face on  

the screen followed by the x and y coordinates of  several keypoints along the face, eyebrows, eyes,   nose, and lips. You may have noticed that we never  answered the question of how a brain detects faces   and facial expressions. The answer to that is who  knows? It's not my job to teach you neurology. In   fact, I'm unemployed. If you take the definition  of VTubers to be "a" then its history is a pretty  

straightforward series of events. Following  the massive success of Kizuna AI in Japan,   many other Kizuna-esque VTubers started popping up  such as Kaguya Luna and Mirai Akari. It was only   a matter of time before agencies that managed  several VTubers started appearing to the scenes   such as Hololive. The agency Nijisanji broke  from the tradition of 3D models and used Live2D   which is like 3D but with one less D. Rather than  a 3D model of joints, meshes, and textures, Live2d   takes several flat images and layers them on top  of each other to move at different motions giving   the illusion of depth. Perhaps more importantly  though is that Nijisanji focused on live streams   rather than video productions with which Hololive  and other VTubers soon followed suit. And like all  

things Japanese, very soon there were fan English  subtitles followed by official English subtitles   followed by English speaking subsets such as  Hololive English producing English groups such   as HoloMyth including Gawr Gura followed by  entire English-speaking-based agencies such as   VShojo. This rise of VTubers from the debut of  Kizuna AI to the debut of VShojo is a relatively   short period of time from 2016 to 2020. In almost  all of the examples I've given thus far though,   the technology used for facial tracking is not  artificial intelligence. No matter how efficient   or accurate a neural network may be it has one  fatal flaw in that it is software. Having to put  

every frame or every few frames through the neural  network to get updated coordinates for our model   is a lot of processing. Even with a convolutional  neural network that does everything it can in   every layer to reduce the number of parameters,  image processing is going to be a costly process.   This means that in order for the animation to work  in real time with what's commercially available   today, the smoothness or precision is going to  have to be significantly reduced. Add on to the  

fact that computer vision is very dependent  on lighting, you can't process something from   nothing, and it makes sense why both Hololive and  Nijisanji provide iPhones to all their incoming   VTubers. The TrueDepth system of apple's FaceID  still uses software but the hardware part is   especially designed specifically for the purpose  of facial mapping. This means that rather than   being given some massive data and then finding the  features that it figured out how to find on its   own, the program is given the features of light  distortion or depth that coincides directly with   the coordinates of the facial landmarks using  just some conventionally programmed geometric   operations. As funny as it would have been though,  it's not like all that talk about machine learning   was completely irrelevant. There are still an  abundance of VTuber applications for webcam  

using ConvNets primarily targeted towards  independent youtubers who don't get free iPhones.   Luppet, Wakaru, VSeeFace, 3tene which comes with  bodily contortions, FaceRig which comes with not   being good, to name a few. VTube Studio which is  for Live2D is available for webcam, Android, and   iOS. For webcam, it uses a model from OpenSeeFace.  There it is. Whereas on Android it uses ARCore,   both of which are deemed to be of less quality  tracking than the iOS version. VTubing is not just   facial tracking though but since it's all about  tracking a human and mapping it to an avatar,   all the other aspects of VTubing short of  a mocap suit can use similar technologies.  

Hand-trackers such as LeapMotion use IR projectors  and sensors to track hand motions which is   very handy but also limited because you can't  cross your arms so no snarky remarks. Natural   language processing problems such as speech  recognition require a lot of feature engineering   and so neural networks are preferred,  inputting human speech and outputting   text which can then be either used as another  way for mouth tracking or be synthesized back   into speech via more neural networks to  mask your voice like VShojo's Zentreya. "Head fat". Neural networks, light sensors, and  even mocap suits, VR headsets, eye-trackers, and   the XBox Kinect are all methods of motion capture.  And if I didn't want people to see this video,   I could probably title it motion capture at least  for up to this point. But that still wouldn't be   entirely true as the motion capture required for  VTubing is still different than that required   for general virtual reality or film production.  There is an emphasis in the technology on facial  

expressions, affordability, and presenting to  an audience in real-time. What's more is that   VTubing doesn't have to be and was never meant to  be just a one-to-one direct transfer of motion to   an avatar. While this goes more into design than  development, VTubers can also employ things like   object interactability, keyboard shortcuts for  pre-programmed animations, or additional physics.  

VTubing is like a virtual puppet show or Luppet  show you could say or not say actually and just   because the puppet strings of motion capture are  necessary, doesn't mean you can't improve the show   with piles of corpses. Maybe it shouldn't even  be a puppet show. Perhaps the future of VTubing   should be a looser connection to the puppeteer  for more expressive or stylistic animation.   A paper was written last year in 2021 for a  VTubing software called AlterEcho. The software   uses facial expression recognition, acoustic  analysis from speech recognition, and mouse or   keyboard shortcuts to apply gestures to an avatar  on top of motion capture - gestures that the human   themselves are not actually doing. The nature or  mannerisms of these gestures can be configured   by what the paper calls avatar persona parameters  such as how shy or confident the VTuber persona is   supposed to be. How effective this all is is still  unknown though as the software is unavailable and  

the paper is still under double-blind review  at least at the time of this recording,   though the paper itself states that it was rated  fairly highly compared to pure motion capture and   VMagicMirror which is a keyboard-based software.  On the topic of new softwares for independent   VTubers, while 2D models are modeled with Live2D,  3D models are not necessarily modeled with actual   3D modeling software like Blender but rather  softwares like VRoid Studio which is essentially   a character customization screen which has many  sliders for incredibly unique customization,   though the official stable release has only been  out for a year. Currently, both 2D and 3D VTubers   suffer from a noticeably homogeneous design and  style that some say is reminiscent of Genshin   Impact characters whereas others argue is  closer to Honkai Impact characters. Perhaps   a much more unique easily accessible VTuber  avatar creator will be available for the next   generation of VTubers. It's unlikely that it  will ever break out of that anime niche anytime   soon. You had your chance. And it's definitely not  going to be whatever Metaverse was supposed to be.   But just like how Live2D models have been getting  exceedingly creative, 3D models could branch   off into as many styles as there are styles of  anime which has the opportunity to be aided by a   movement towards more motion capture-independent  animation-focused VTuber software. In regards  

to the future of VTubing, there is another  possibility that has been somewhat disregarded   since the decline of Kizuna AI and it has to  do with the AI part of Kizuna AI. Not love.   It's quite common for VTubers nowadays to come  with their own lore - the avatar is actually a   mythical beast or a magical being. Kizuna,  the self-proclaimed first virtual youtuber,   of course had the backstory of being an artificial  intelligence. This whole backstory stems from the   original idea behind artificial intelligence: to  mimic human intelligence. A neural network can   learn how to find the features it needs to find to  locate a facial landmark on an image which could   imply that given the right conditions in training  it can learn the features of human behavior and   produce content. And while in the case of Kizuna,  artificial intelligence was only used at most for  

landmark detection, there already exist machine  learning models that write scripts, interact with   humans, play games, and generate animations. There  are even neural networks for singing synthesizers   such as SynthV which has been mentioned to  me by all five of its users. It seems not   too far-fetched to just combine all of these to  create a truly automated artificial intelligence   virtual youtuber. However, we also know that all  of these are just several independent abstract   networks. The learning that these networks are  doing isn't based off of experience or intuition  

or arguably even any logical structure. It is  just a collection of shifting parameters and   mathematical evaluations that knows something  is correct because, in the case of supervised   learning, we informed it that it was correct.  A content creator about as sentient as the   trendline generator on Microsoft Excel. We know  this even without fully understanding sentience   because we know what sentience is not and what  it is not is a pattern recognition machine in a   computational void. The actual algorithms of human  intelligence may still be unknown but because of   the way that machine learning was developed,  artificial intelligence isn't intelligent in   the same way that humans are but it can learn the  features of human intelligence and reproduce it to   an incredible degree. It's no longer uncommon  for humans to be deceived by AI generated   conversations or art or music, though perhaps  given the nature of parasocial relationships   and corporate media, we simply wouldn't mind.  Ultimately, whether a content creator who is not  

sentient - who we know is not sentient - but can  imitate sentience perfectly will be in or against   our best interests will be up to us as living  breathing humans. Until then though, there are   at least a few more Hololive generations before we  have to make such a decision so we might as well   enjoy what A.I. has to offer and look forward  to the future of this still young VTuber era   without having to really worry about any  unforeseeable threats of other intelligences. "Yahoo"

2022-08-08 01:01

Show Video

Other news