AI Emotion Detector Using Mediapipe Facemesh Technology
developers have three modes happy that's if the code run successfully and everything goes as expected the second mode is neutral that if the output is not what they expect or what they want and finally sad when there is an error on a line where there is no code at all today I'm going to be teaching you how to build your very own mood classifier using computer vision and machine learning at the end I'll be sharing my code with you so that you follow the same guideline and VI your own emotion or mood classifier ready to do it let's get right into [Music] it all right so before we start the code including every file will be using in this particular tutorial will be on my giop repository and the link will be in the description you can leverage this code and update it work on it and also use it as you want so I'm not going to be coding from scratch I'll just take you through all the codes so we'll be using three files and I'll take you through how everything works I've have done everything so that you can take it on from there because uh coding all this will take a lot of time which will make uh the tutorial boring in my opinion okay so we'll be working with three files before I even introduce this files let's go through uh this particular image so we are going to be using the face meesh data so face merge Google face merge which is done by the media pipe team we be using a raer library cive Zone which will help us to get this fish mesh data in a few line of code so this contain about 468 landmarks on the human face and uh based on the action performing by the human pH we collect this data as to sad or happy and then we'll put this in a CSV file where we read and perform some data cleaning and also train a model out of this uh data so let me show you how the CSV data will look like before we go into the coding so in the CSV data what you are going to create is that because you have 468 landmarks we have the first uh columns here or our headers here to be class and this particular rows to take in the classes we are going to be taking the data for then this will be X1 y1 all the way to 468 so you can see up to here 468 because you have 468 landmarks on the human face so that's what we are going to be doing and for the classes column you can see we have class for Happy and when I come down right you can see we have class for sad and you are not limited by the number of classes you can actually take many classes as you want you can do happy um neutral anything at all you can get from F expression you can add that class to it the possibilities are endless because of the flexibility of the code so I close this app and now uh I have three python script datag gen. Pi training. Pi and test.py so the data gen dop is responsible for generating the data for us so for that we are importing uh these libraries and all the libraries you need to install are here which will also be uh on the gab repository you need opcv python media p Panda CV Zone and nonp then we use cyc l to also train our model we use some of their STS to train our model okay so we are importing CV Zone in order to use the face MH detector and uh for that I'll take you through how to get a face M detector or I will even take the code line after one line after the other so this is the path to our video we have two video F here happy and sad and uh happy right here just contains s of people being happy or people living in their happy moment so you can see the facial expression this lady is very happy this man is happy and so on so I just took some couple of videos and compile them together the same thing applies to the S video just some video of people being in their sad moments which is weird though but um we have to do it so now that you are familiar with all the file let's start off by looking at the data gen. by close this off and um we imported our libraries once again then we gave the path to the video file so if you using your camera you make sure you put uh the index of your camera here that's one Z and so on so if it's zero that means it will take your webcam associated with your computer but if you have a USB cam you want to use then you should start from one two until you get your camera to work with so since I'm using the video side here all I have to do is to specify the name of the video and that's video PA because that's video path right here from there create an object for the face detector so face detector object so fmd which I'll be using which stands for face M detector is an instance here of the class face M detector and over here you can specify some couple of things such as the mass faces so the maximum number of faces you want to uh detect so if you put one here that means U it only take one person's face when multiple people are looking into your camera or your webcam so you can specify or you will leave it so that it picks everybody in that particular frame then the next thing is to Define your class name so since we are doing it for a sad. MP4 then the class name should be sad so that in our data. CSV it will be
stored there as sad all right let me uncomment these lines so what we are doing with this Lin to create our columns in the uh data. CSV or in our data file so with this first we specify our class so we create a particular column in our data call class if you can see that right here at the top let me scroll at the top here we have class so the First Column should be class Then followed by this X1 X2 which are the data points so that's what we are doing we create our first one which is class then for the X1 X2 also we look through them and create them using the noral python formatting so um we have 468 landm marks then plus our class name or the class column making one so we add one to it which is making it 469 in total so that's what we are doing creating the first header names right here then after that when we run the code we want to put that header name in The First Column of our data set so we do it once that's why it's not in the loop we do it once here so what we are doing that first time we run the code this header names right here this first line or first uh row of names we put it once because we only need it once or we need uh down here will be the data and they corresponding class l so that's what we are doing it here so that means when you run this code once it will create this header for you so after that you need to comment it off because you will not be creating header anytime you run the code so once You' run this ones on your side video or any video at all you need to comment this one off okay so after this we are doing while cap do is open to get our video feed from the camera so we are just getting getting it since it's a video file we want to get it in the loop we read it we resize it and then now feed it into our face M detector so we are calling it frame so we are feeding this Frame into face Med detor and this will return as our faces if any faces has been detected and there is a mesh on it to return return us and return return us our image to which is the same as our frame okay so right from here we check if we are getting some faces if you are getting some faces we take the faces and then we take the data so so this face mer data will come in a list so we are converting it to a list and also we are making it a onedimensional data or vector that's why we are using flatten here which is the same as reshape into one long giant Vector array then we are inserting the class name into it so we are taking the phase data here and we are inserting the class name at the first index that's why in the CSV file at the first index here you can see the class so we are inserting the class name here then what follows will be our data so we take our phase data and then we insert the class name into it so that we can um select this class name if you want to preprocess this data and train a model out of it so that's what we are doing and after that um we are writing this to a CSV file and the format here pay attention we are appending it that's the mode the we using is we are appending so that when we take data for one class and we are coming to take data for another it will not overwrite that you append that I mean it will go down we'll be adding to it so we are just writing to a CSV file called data. CSV and our delimer we are using comma so comma separated values after that we are showing our frame so that we can see it and collect our data so that's all this data generation script is doing if you have any to do with it you can uh Advance it to do your particular tax in hand so let's run this and take data for the S class so before you run make sure the class name is sad and the video you are going to do is also pertaining to sad so right here you can see we have our video feed we are collecting data for the S class and you can see we have the data. CSV file being generated right here so this will take some time to generate data for the S Class so just like this you are done taking data for the S class and that's how we take data so the next time you run this script for another class you need to comment this to this line of code so that you don't overwrite the column names again because we don't need that again so if I want to take data for happy I can just now change here to happy here to happy so let me do that so happy. MP4 if
I could spell that correctly and then here should also be happy so happy. MP4 that's the name of the video right here and the class name is happy so that's all that's all I'm doing I'll run this once again and it will start loading data for the happy class so this is like a whole data collection uh process and just like that we are done taking uh dat or collecting data for the happy class in case I want to take data of myself or from my webcam all I'll do is to replace it with one so I'm going to collect data of myself being happy and let's see how it goes so guys this is me right here being happy so just as I'm talking right here I'm collecting data of myself being happy you can see it right so I'll do some for sub to as well add data of myself you can miss the data up it doesn't matter we just use use the class name to select them from our CSV file so that it right here I'm collecting more data and I think it's enough I will do that for sad to as well so to collect for sad I just have to change the class name to sad as well and collect data of myself so you can add your own data to it if you want or you can even use your own data from scratch that means you are not going to touch the video path variable here all you will do is to change then uh the names here and then take uh data for that specific class so you can add your own class whether you are smiling you are angry you are do this or that so now let's do it for sad let me run it so this this is me being sad so um I think it's enough I think it doesn't look good so um now that it so that's what this script is for for generating data and we've generated the data and here is our data being saved here so we have ton of data we have many rows so it's about 4,000 plus so I think that's enough so the next script we'll be working with will be our training. PI so this is it so over here we've imported pandas and pandas is for data manipulation cleaning your data structure your data rename columns check info about data and so on so we use pandas to uh structure our data to clean our data and then use skn which we are importing so many stuff from here to train our model split our model into train and test set and train our model then finally we use the pickle model from python is inbu to uh save our model so that we can use it later on okay so over here we are reading our data data. CSV so pd. read CSV and we are assigning the data to DF which is uh data frame then after reading this data there are some uh stuffs we can perform on this data data do head this will give you the first five rules of your data data do columns to also print all the columns when I run the code you see all of these STS data. info is going to give you some information about your data your memory usage and U some additional information then if you want to perform some statistical analysis on your data we use the DF do describe and this is going to give you uh the main standard deviation the first quarter second quarter okay since we have a lot of data points we are using DF drop now here which will uh clean off all other columns that we have no data so if there are uh columns which have null values we are going to clean them off and use um The Columns that we only have data for then the next thing is to go ahead and separate this data into our features and our labels so the labels are our targets we want to use these features to predict our targets so um our labels here is our class names and in this case we have only two class name happy and sad so that's what we are selecting here we are taking the data frame in the index of this particular class and for our features we are just dropping the class so whatever is left will be our features when you drop the class then from X1 up to 4 all the 46 eight landmarks will be our features which we'll be using to train our model over we just printing them to show the next thing I do on this line is to split this data into train and testing partition so we have our X train y train X test and Y test and we are using the escal TR test split function to do that okay so after splitting this data we create a model pipeline so we are using standard scaler to normalize our data and also we are choing using the model to logistic regression so uh with this particular pipeline it's very easy to switch between different models all you need is to type the name of the model here so let's say you need a random Forest classifier you just you can just type the name right there to change this particular model so you can test other models to see their performance on your data and then you use the model that best fits this data for you then uh we use the pipeline fit in order to train our model then after training the model we are running test on our test data so model. predict then uh the next thing is to evaluate this model to on our test data so we are evaluating it and we are getting the classification report this classification report will give you information such as the F1 score you also get a precision and the recall you'll get the accuracy and so much more so uh using the classification report from skn is very helpful then after that um we've done training the model so we have to save this model and that's what we are doing here we are using PCO to save the model so p. Dum and then we are writing it so write binary and then we are
damping it the name will be model. pkl so after running this code we expect our model name model. PK should be uh save in our directory we are working in so I'll just go ahead and run this code we just this few steps you can see we now have our model. pkl here that means this our saved model which we can use to run inference so let's go through some of the things it has printed on the terminal do head which gives us the first five rows and then also uh we have the info so you can see over here we have some informations about data set the memory consumption of this particular data set and some other informations for the columns we have all the column names here so the terminal cannot display all that's why you have the dot dot here but it gives you from starting to the end okay so here is where we are doing the data do describe you see you have your mean for each and every column so you can see X1 you have your mean you have your standard deviation after which we also uh printed the features and the labels as well so we have the labels features then over here we are testing our model on a test set so these are the predictions from the model from on the test set then we come here this is our classification report so you can see we have the Precision recall F1 score and the accuracy for this particular model is 100% wow that means our model is very accurate on um our classes so you can see the happy for the happy class both Precision recall and F1 score 100 100 set the same thing that means the distinction between our model on our classes is very great so let's see whether this model can generalize in real life where we are going to run it on our webcam to take our webcam feeld in order to classify whether we are happy or we are sad so for that I have the test. pi here so with the test. we are just loading a feed from our camera just like the data generator so field from our camera then over here we are loading our model then we process this field just as we did in the data generator we also flatten it to be a long list then we are feeding that phe data into our model to give us results which will display on the screen so that's all we are doing for the test set I think is s explanatory when you look through you can see similar to the data generator but instead of saving the data pH data we are sending it into our model for prediction so I'll run it and this will run and open my webcam so I I set my webcam here one I have a USB webcam attached so I'm using one if you using your actual computer webcam then this should be zero so let me run this and let's see how this model performs in real world so guys here am I you can see I'm in a happy moood so it's the tting that I'm happy um let's try start so I'm sad I'm happy and you can see so I think this works this classifier works you guys can go ahead and add additional class to it to see how it perform and also I'll be sharing the code and every other file you see here in this particular tutorial or my GitHub repository so just check the description of this video and you can have access to it guys this is all for today you can either be happy but don't be sad be happy and make things work out thanks for watching and um if today is your first time make sure you subscribe you share with your friends and up until then I will see you in the next tutoral
2024-03-10 14:36
