All. Right hey everyone welcome back. This is people who gave me okay. Alright, so if, as usually you can take a second to, your su. ID so, we know who is here um. So, today's. Lecture will. Be a choose-your-own-adventure. Lecture. So. I think you, know by now you've learned a lot about the, technical. Aspects, of building learning, algorithms, and then, in the third course in. The third set of modules you saw some, of the principles, for debugging learning algorithms, and how that you use these tools in. Order to be efficient, in how you build a machine, learning application. What. I want to do today is step. Through, with you a, moderately. Complicated, machine. Learning application. And. Throughout. All of today's, lecture I'm gonna you, know step you through a scenario and, then. Ask you to kind of choose your own adventure because. If you're working on this project what. Are you gonna do right and, so you give you all that practice, in the Nix what. Are in a bit that we have on. Thinking, through machine learning, strategy. And. You. Know I've seen in so many projects. There. They're sometimes, things, that a less, strategically. Sophisticated, team will, take a year to do. But. If you're actually very strategic, and very sophisticated in, deciding, what you will do Nix right how they drive a project, forward I've, seen many times that what, a different, team will take to do maybe, you could do it in a month or two and. You. Know if you are trying to I, don't know where. The research paper or build, a business or the product the the the ability to drive. A machine learning project, quickly gives you a huge advantage and, just you know you're making much more efficient use your life as well right, um, so in, for. Today I'd like to. I'm. Gonna host a scenario, post a machine learning application, and say alright, I mean you are the CEO of this project, what are you gonna do next so but I'd like to have, today's meeting, be quite interactive as well so can I get people to sit in groups of two, in ideally, three or so maybe plus minus one and, tried. Sit next to someone that you don't work with all the time so, if, it's simply sitting makes your best friend I'm glad your best friend is in the class with you but, go sit with someone else because I think I'm I've. Done this multiple times in the discussions, that he Richard you talk, to someone that you don't know super, well. So. Actually take a second introduce yourself and, just. Be, cool neighbor i yes, so. The example I want to go through today is actually a continuation, of, the. Example. I described briefly in. The last lecture I taught building, a speech, recognition system. Right so remember. Briefly. Motivated. This trigger. Word awake word or trigger word detection, system last time where. You. Know. Right. Iiiiii, have, both an Amazon echo and a Google, home but. You, know is this a lot of work to configure, these things to turn on and off your lightbulbs, and, so, if you can build a chip, to. Sell to say a lamp maker to. Recognize. Phrases. Like you. Know let's say we call the lamp robin. Then. You can recognize phrases. Like Robert turned on right. Robert. Turn off and. You have a little switch to give this thing different names you're called robbers or later. Or, elders or something as you can also linear. Turn on later turn off and give to give your lamp a name and just say hey Robert, turn on right so, rather than detecting, different, names and turn on and turn off I'm just gonna focus. On, just for the technical, discussion, I'm just going to focus on, the phrase Robert. Turn on but. This is kind of the same problem you need to solve like four times to give it two names or to turn on and turn off so I'm gonna abbreviate Robert. Turn on this RTO, right, if you want to call, your name Robert and. Tell. Your lamp to turn on I think. I was inspired by well Isaac Asimov, wrote these own robotic. Novel, C's and and all his robots names started, with our so, maybe our s robot turned on. And so. Let's. See so, let's say that you. Are the, new, CEO of a small start-up. With, you, know three persons and your. Goal is the building is, to build a circuit, or actually your goal is to build a learning algorithm, that, can, recognize this, race Robert, turned on so. That when someone either, buys this lamp and they say Robert turned on the, lamp can turn, on right and just focusing. On the task of building a linen you know to be, CEO to start to be to do a lot of things where I need to figure out how to do the embedded circuitry you figure out whether that Mika's sales general so, there's all that stuff but for today let's just focus on, the machine learning aspect.
Of It and. So my first question to you it's very open-ended, is but. And this is the life of a CEO right you wake up one day and you just got to decide what to do, but. So my first question to you is the open-ended question is you're, the CEO oh, you're. Gonna show up at work you. Know tomorrow in your startup office and you. Want to build a learning algorithm, to detect. The phrase Robert, turn on for, this application right. So um, so my question, is what, you're gonna do right so take a take a minute to answer, that by yourself first no, don't don't discuss your neighbor yeah but then you're gonna show up in your office and and, then you're gonna start working on this engineering problem, to build a neural network to do this so and. And do this as yourself right don't don't don't pretend, that you have this hypothetical. Whatever. A. Start-up. CEO with ten months ten billion dollars to spend or whatever just do this is let's say yeah I but, I don't think this is a terrible startup idea is, not the best idea but I think this could work so you're actually welcome to do this but. Let's say you decide to do this and you go into your office tomorrow like, what do you do right want you to take um, why. Should take let's. Say two, minutes to enter an answer then we can you can discuss I. Think. Yeah. Yes, one, thing I really like about answer, is, the meat, is extinguished. Share a pot right, in fact, when you start a new project. And. I think. When. You start a new project like that assuming, you've not worked my trigger words detection before you, know reading research papers or reading code on github or reading blog posts on this problem is a very good way to quickly level, up your knowledge, and. I think that you. Know it, turns out that um. In. Terms of your. Exploration. Strategy, right. Um I want to describe to you how I, read research, papers. Which. Is so. This is a not. A good way to review. The literature which, is if the. X-axis, is time and, the, vertical axis is research, papers what, some people will do is find the first research, paper and, meet, that until. It's done and then. Go, and find a second research paper and beat that until it's done and then you, can find the third research paper in this very, sequential way, of, meeting. Research papers and I find it the, more strategic way, to go, through these resources everything, ranging from blog, posts, lots.
Of Good medium articles that explain things write. Research, papers write. Github. Is. If you use, a parallel. Exploration. Process where, this is actually what it feels like when I'm doing research on the way I'm trying to learn about the new field I'm not that experienced, right so I've she's done a lot of work on trigger word detection but if I hadn't worked on this before then, I would probably find, you, know three papers, so again x-axis, is time and Mercury axis is different papers and, you. Know read a few papers kind. Of in parallel at a surface level and skim them and based on that you might decide to read that one in greater detail and then. To add other papers, that you start skimming and maybe. Find another one do you want to read in great detail and then, to gradually, add new, papers, it's, you're reading this and. Be some to completion and some not to completion. You. Know I was actually chatting with um. One. Of my friends at Peter Bua former, students, at Berkeley who mentioned that he was wanting to learn about a new topic and he, he was he told me is compiling a reading list of 200, research papers they, want to read that sounds like a lot you rarely read 200 papers but I think if, you read 10, papers you have a basic understanding if. You read 50 you're a pretty decent understanding if, you read like, 100, I think you're a very good understanding of. A. Few but often. This is time. Well spent I guess oh and. Some. Other tips again this is something I'm really thinking if you really are CEO, of this startup and this is what you want to do what, advice would I give you. When. You're reading papers, other. Things to realize one, is that some. Papers don't make sense right and this fine you. Know you might be some papers I just go nope I don't think that makes sense so and and it's. Not that uncommon for, us to find. Papers from a decade ago then. And we learned that half of it was straight and the other half of it you, know was, really talking about things that were not that important, right so it's okay. Authors. You know usually papers, are technically accurate but, often, what, they thought was important, like maybe an author thought that using, Vacheron was really important than this problem but it just turns out not to be the case that that happens, a lot but, that happens sometimes and. I think the other tactic, that I see Stanford. Students sometimes not use enough is, talking. To experts including. Contacting, the authors so when, I read the paper. I. Don't, I don't bother the authors unless I've actually like tried to figure it out myself, right but if you actually spend. Some time trying to understand the paper and if it really doesn't make sense to you. It's. Okay to email, the authors and see if they respond, and and people, are busy maybe there's a 50% chance they respond and that's okay because it, takes you five minutes to write an email and there's a 50%, chance to get back to you that, could be time pretty well spent. But, but don't don't don't bother people on this you try to do your homework I should get a lot of emails from you know high school students, that do not feel like they've done their own work and. Right, and, then so. So just don't don't don't bother people unless you've actually tried to do. Cool. So. After. Looking. At the literature. And. Having a base maybe downloading, a open-source, implementation. Or getting a sense I've been averaging when I try oh and, it turns out the trig reward detection literature. Is actually one literature where there isn't consensus. On this is a good algorithm is a bad algorithm right despite all the trigger.
Word A way core detection system is that you know some of you may use already there. There there isn't actually consensus. In the, research. For me today unlike this is the best ever and to try. But. So let's say that um you read some papers download, some open source of invitations and, now you want to start training your, first system, right a lot of time we talked about this we talked a little bit about how, much time you will spend to collect data and then you know we said you spend, a small amount of time spent like a day or maybe two days at most so, collect your first data set to start training up a model. But. My next question to you is what. Data would. You collect right. Um, in, particular what. Train. Death. Test, data. Would. You collect so you've decided on an initial neural network architecture and, you, want to train. Something to recognize this face robert turned on i think this. Probably. I don't think it's possible to download the data set i don't think anyone has collected the data set with, the worst rapper turned on and post it down the internet so you have to collect your own data for this particular trigger for a so you want to use but. Some you, know sco, of, this sort of trying to build a new, net to detect the phrase robert turned on. What. Data do you collect right so once you take once you again take, let's, say three minutes to, write. An answer, to. This yeah. I think this is an interesting. Robber. Turned on over, and over and then. Data augmentation. Data. Augmentation. Is one of those techniques that. Is. A way to reduce. Variance. In your learning algorithm, because you're generating more data and. Having. Worked on this problem, I happen, to know data augmentation, works you. Know it's very useful for this problem but, if you didn't already know that fact this is one of the things I would probably not do right away because. I would train a quick and dirty system. Validate. That you really have a high variance problem before, investing, of the effort into data augmentation. So they don't know as well those techniques that so, you know like it never hurts they're all very very hurt so usually helps but I don't, bother to make that investment unless you have collected. The evidence that you actually have a high variance problem and, that this is actually a good use of your time. Yeah. I. Think. This is just one that she just. Actually nice so. Require, gonna start to st. Robert turn a hundred times so doing really nice thing about that you can get done really quickly. When, I'm working with teams I actually. Think, in terms of hours, in terms of how long it take us to do something so this, one you could probably do in like, 30 minutes right, so you get your dataset collected 30 minutes ain't get going or. If, you run around Stanford they just ask you no friends. Or strangers to, speak. Into your laptop. Microphone you, didn't spend a few hours together much bigger data set then, possible to start up so I priority do that I probably actually couldn't collect data and several. Hours for app didn't only spend 30 minutes but this is actually pretty interesting as well because unless you get it done pretty quickly access, right so. Um. Yeah. So, let me actually uh share. Some more concrete, advice, right, and I think actually some point some time back to, prepare, a homework, problem that, you see later in this course Ken, and Eunice and I we're actually you, know building the system partially to create, the homework right then, you see later in, this course this is a good this. Trigger word thing is a nice running example that we're using a few, points throughout this course um, so. Here's, one thing you can do and, this is actually what, what. We did right which is a collect. Oh. Well. Simplifying, a little bit. Collect. A hundred examples. Of. Ten-second, audio clips. Right. And so, it. Turns out once, you grab a hold of someone and. Ask them to speak into your microphone you. Know you can keep them for. Three. Seconds which is how long it takes to say Robert turn on or you can keep them for ten seconds which they're actually very willing, to spend that extra seven, seconds, you right but. So if this is ten seconds of audio data you. Know so this is ten seconds of audio and, and, audio is just patterns, of little. Changes in air pressure right so if you plot audio the reason looks like this waveform it's, just, the. Way you're hearing my voice this, yo my voice or the speakers are creating very rapid changes in the air pressure and your, ear measures those very rapid changes in the air pressure interprets, in sound and so a microphone.
Is. A sensitive. Device for recording, these very very high frequency changes, in the air pressure and this plots, that you see an audio is just what is the air pressure at different moments in time right. But, so given, a. 10. Second clip, like, this if. This. Is the 3 second. Section. Where. They said Robert, turned on then. What you would like to do is to build a nest. Lam say they, can sit here and the lamp is turned off turn it off turn. It off turn off, turn, off turn, off and at. The moment they finished saying Robert turn on you. Know you turn it on so this is a open, table Y really right. And then. And then it's not detecting, afraid right so so what you want to do for, the trick word system, is at. You, know pretty much the moment, they finish saying Robert, turn on. You. Want your learning algorithm to output, a 1 that's. Your target label, why I saying yep I just heard this trigger word and. For, all other times you want it to output 0 right, because cuz and, then the 1 is when you decide to turn, on the lamp at that moment in time right, so, to. Collect the data set here's. Something you, can do which, is, collect. 100. Audio clips. Of. 10, seconds, each and. You. Know when, I'm prioritizing my, work home my teams work I would really you, know look at these numbers and think okay let's, say let's say you actually if you are doing it let's say you are running around Stan's and you. Want to collect a hundred audio clips. Maybe. 10, people 10 clips per person or maybe a hundred different people, I would. Actually estimate. You, know if you go to Stanford cafeteria. How. Long does it take to get one person, right and you can pray get one person every minute or two if you go to a busy place them on like, a Stanford cafeteria, so, you can pray to get this done in like a 100. To 200, minutes late or three hours right it's not that bad so you get this done quite quickly, and. So. And lets you collect 100, audio, clips and actually, for them for the purposes of today. Let's say, you. Collect hundred, or do clips to use for training. 25. For. Your dev set. And. Zero. For the test set right it's actually not that uncommon, if you're building a new product they just not have a test set because you go is to build, something that you're convinced is you know just early, prototyping. Phases of the project sometimes I don't bother with my test set if you if it goes the function, paper then of course you need a rigorously, corrected test that but if you're just building a product and you don't need a rigorous, evaluation sometimes, you can just get started without doing with a test set right those pretty little get started, um. And. Then. All. Right so. Taking, that audio, clip from above. One. Thing you can do to, turn this into supervised, learning problem, is. To. Take so, you know the phrase, robber turn-on can be said in less, than three seconds so let's say you take three seconds as the duration. Of audio right so what you can do is a clip. Out so. Let's say here was when Robbie turn on my set so, what you can do is, the. Type of. What. You can do is then clip, out different, audio clips of three seconds so here's, one audio clip you. Can take that audio clip, this. Is X and the, target label is. Zero because, Robert, turn-on was not set, and. You. Can take I know. This audio clip a different, random nature clipped, with, three second clip and that clip. Also as. The toggle a buzz rope. And. You. Know for this one right. Which is a three. Second clip there. Come that that that ends, at the reel on the, last part of the on sound you would have a target, label of, one right.
So, And when. You learn about sequence, models there are and ends you learn a better method than this, explicit, clipping but for now let's say you take. These on audio, clips and turn it into please. So take a ten second clip and by, clipping around different windows. You can take. Your, let's. Say 100. Clips. And. Because. For each 10, second clip you can take different windows, you, could turn this into let's, say, 3000. Training. Examples, right so here I took a 10-second clip and -. And, show you, know totally, three different. Three-second. Windows but if you take thirty three second windows then. Each 10-second, audio clip becomes thirty examples, and now you've. Turned the problem into a binary classification, problem where you need to train a neural network that inputs. A three second clip and they, build it as, either 0. 1 right those, mean this and so this, is an example of. The. The more, complex, pipelines. You might have if you're, building a learning algorithm to take. A continuous. You, know audio, detection problem to turn into the binary classification problem which, you've learned how, to build barriers near networks for right, and again we learn about our own ends you learn about other ways to process sequence data or temporal data okay. So. Um, go. Ahead. Oh. Is. This manual even yes. I I would yeah actually if you have a hundred examples. It's. Not that hard to just listen to it you know on your laptop with some audio playing, software. To figure out when, when. They finish saying or Robert, turn on and then at that moment to. Put a 1 in the target label because this is really when you want the lamb to turn on right. So. Any. Other questions actually few fideos clarifying, questions yeah go ahead oh I, wonder if this is gonna cost a problem that ones. Are tools bars, oh sure. Let me get back to that. And. Things. If. There are specific reason, we only train them with three seconds. I. See. Yeah why, do we do three seconds there for five seconds there's a yeah, because there's another hyper prompt you can test so. I can go oh no, oh. Yeah. You, have to stand really slowly to. Take. Three. Seconds is this right. Reabsorb, it turn. On. Right. So okay this is a design choice yeah. Um. Alright so, so, um, let's. Say you do this feed. It to supervised. Learning algorithm. Train a neural network um, and. Let's, say that when, you classify, this when. You run this algorithm you end up with. 99.5%. Accuracy. Right. But. You find that the algorithm, has zero. Detection. Right. And. You know and what I mean is that whether. Audio, you give it it.
Just Outputs zero all the time so the algorithm, just says nope I never heard the phrase Rob I turn on you know so. So so. So. And, so. My question to you is you, know and by the way the. Reason I'm going through these scenarios is um I found that a. Good. Way to gain. Good intuitions. And and, to become good at making these decisions is these are the decisions that project, leader right, a tech leader our CEO needs to make these are actually like pretty much exactly the decisions, you need to make and I, find that um one. Of the ways to gain this type of experience if you you, know find a job with a good AI team and work with them for five years right and then you said you lived through this and you see what they do but. Instead of needing you to go and spend five years to see ten examples of this I'm trying to step. You through maybe one example in. In one hour or so so instead of yo. Gaining, this experience, to work, experience. Which is great but takes many many he has how, many many months hoping. To you. Know let's just put you in the position of making these decisions you can learn from that much faster right. But. So. And. All the examples I'm giving or actually completely, realistic right there you either, exactly or very similar to things, I have seen in actual, you, know very real project, so question. Is uh your. Learning algorithm gives sisters out 95% ac0. Dissections. What, do you do let. Me mention some, of some dances are really light I. Think. That. You. Know I when, I think of building learning algorithms. The. Process, is often specify. A deaf set and all test set that, measure what you care about and, then. You. Don't always have to do it but it's good hygiene, and it just is it is a. Shoppings. Clarity, of your thinking, right if you have a very clear specification, a problem and I think when, inside all of this is that if your death set is really out of whack right, because it's so unbalanced, that accuracy your deficit doesn't. Translate to what you actually care about because you, know presumably this is 99.5. Percent accurate, on the deficit as well but. This performance is terrible, so it's doing great on the death set on your accuracy nice shake we're giving terrible performance, so, I think of it as good hygiene, you. Know this is kind of good sound, practice. To. Just, specify, make, sure you at least have a Jeff said an evaluation metric, that corresponds, more closely to what you care about, so, making the deaf set more balanced, equal. Numbers of positive and negative which would, be a good step toward that. And. Then I think. What. You could also, there. Are a few people that talked, about, give. The higher weights to the positive, examples, right so you, know, one. Way to do this is the resample, your, training, and your Jeff says to make them more, proportionate, in terms, of maybe closer to balanced, racial positive negative examples, that'd be okay the, other way to not do Reiss we just give the positive examples, it great to wait right.
I Would, probably sample. Another. Thing you could do. You, know India in the interests of. Speed. Even. If it's not the Matt Matthew most good most, sound thing to do is to change. The target labels to be a bunch of ones after. That. And. This is a hack this is not formally, rigorous, but, if you've implemented, the rest of this code already this. Might be a reasonable you know a little bit hacky thing to do but this is just this is this might work well enough right I would, I might. Not I don't, know if I would want to try to you. Know write an academic research, paper with this method maybe and get away with it but this little thing that I think if you try to publish your paper with this academic. Review is my raising eyebrows and say maybe, now, maybe it is okay but I think if you're one, something quick and dirty that just works I think, leaving. The ones changing. Back to labels to be one, so that say, a clip, here. Right. Oh that. And it's just a little bit off their proper turn on the still label one that'd be pretty reasonable it doesn't be saying that. For. Anywhere, within. Maybe, a 0.5, second, period after, Robert turn on finish it's okay to turn on the light any, time within that period then you kind of want to be, turning on the light turning, on the lamp you know say within half a second right, after Robert. Turn on this has been said, anyway. And this. Would be a not, this. Would be a way to just get more labels, of ones, in there, right. How. Does that translate to like when you deploy this you're not gonna see Robert. Turn on as much right like, one out of 1,000, might be reflective, of what you expect to see, yeah. Yeah. Right so I think that uh. By the button. So. If you actually yes, so well I, this. Is of a deaf set and evaluation, actually kind of question right so one. Of the couple of the metrics that people often use when. Actually working on this is when. Someone says Robert turned on what. Does the chance that she wakes up or the lamp, turns on and then the second is if no, one is saying anything to the lamp you know how often does it randomly, turn on by yourself without, you having said anything so, those are the two metrics people. Actually use and and sometimes. You could also try the combined in a single number evaluation metric or something, but. I think that you. Could tend to find the data set to measure both of these things and then and then hopefully find a way to combine them into single real number which. I think yeah and I think one of the ways you talked about in, the in the videos as well great, so make sense yeah. But I think I think the. Question, is really Oh. What. Is it this satisfies, a user need right and, Owen. Just want one thing about dumb the. Straightforward way of rebalancing is that if you don't do this then. Your whole dataset just as very few positive, examples, right. And. So, if you throw away all, the, negative examples. So that you cut down the number of negative examples until you have exactly, equal numbers of positive and negatives you've, actually thrown away a lot of negative examples this, make sense and so one, one one problem, with the straightforward way of rebalancing is, that you, know in. Your audio clip and your test ten-second. Clip they were collected by running around Stanford um, you, have one example of, Robert, turn on and, so. If, you want exactly. Perfectly. Balanced positive.
Negative It, means that you're, allowed to only clip, on one negative example, all of this so you can say that's. Negative and that's a positive, and you can't clip, out more negative examples, from this right, so so so if you use a if you insist on the perfectly balanced you're actually stirring away a lot, of negative examples that that could be helpful for the learning all right. So. Alright. So. You. Know a lot of the workflow of building. Learning algorithms, is. Building. Learning algorithms feels more like debugging right because what happens in a typical machine learning workflow is you, implement something and it doesn't work so you figure out what is the problem so you fix that, like. Rebalancing. Re-weighting we're adding more ones and so that fixes. The current problem and then, after, fixing the, current problem which which is the one we just solved say you, then come across a new problem and you have to solve that and you fix that problem you cook somewhere else another new problem so I find it the workflow. Of when. I'm working a machine learning project it often feels more like software, debugging than, software development. Right because you're often trying. To figure out what doesn't work and then trying to fix that and after you fix that problem then, another bug surfaces, and you squash that and you do that and another and you kind of keep doing that until the algorithm work so if I keep talking about you, know your algorithm doesn't work what do you do Nick right that's kind of theme of today's presentation. But, that that is what the, workflow, that is what your day-to-day work of developing, a learning algorithm is usually, like because I it doesn't work you fix it it still doesn't work as you fix that you still doesn't worry you fix it and you do that enough times until, it works right that is actually what often working. On the learning I don't work look looks like oh. All. Right so. Let's say you fix that problem, and. You. Conclude. Through. Doing error analysis, then, your algorithm is overfitting. All. Right so you, know you've, added a lot more ones so the data set is a little bit more balanced so let's, just add a bunch of ones like I did on that previous board where it was just out of all the ones here so, the data set isn't as unbalanced. And. Let's. See. Let's, say that's. Sorry. Two. Pages of notes here. Okay. Good so, let's say that um you, find out that the cheese now and. I mean assign accuracy. On. Training and, fifty. Percent accuracy, on. The dev set right so very large gap, between your. Training. And your, deaf. Set performance, and so clear a sign of overfitting, and so I think one their earlier question someone talked about data augmentation. And. So when you have this clear sign of overfitting. This. Is a good, time to consider, data. Augmentation. And, then and so let's say you go ahead and do JSOC, mentation, so for audio this is how you could do data augmentation. Which, is um collect, a bunch of background audio. You. Know so I guess if you're trying, to build a lamp, that might go into people's homes then. You could go into your friends homes and you. Know with their permission record. Right what the background sound in their home looks like you know maybe uh people talk in the background maybe the TV on the background, well, whatever goes on people's, homes and, then. It, turns out that if you take a, say. A one-second, clip. I've. Robin turn on of RTO, and, you. Add, that, to. A background, clip then. You can synthesize an, audio clip of what it sounds like and your friend's house if someone were to suddenly. Pop up and say Robert turn on against the background sound of your friend's house right. And. And it turns out that. If. You want to make the system robust. So actually for example I have a oh no I, actually. Know someone that lives unfortunately, closely, to a train station and so their, house that she has a lot of train station noise from the Caltrain and so what you can do to make your system more robust is also take. You. Know a clip of say train. Noise right, like cow train noise and if. You take that noise and take a in. This case let's say one second, one second or three second clip of someone saying robber, turn on and you, synthesize, that on top of the train in the background then. What you end up with is a ten second clip of someone. Saying robber turned on against. The noisy, you know train in the background type of pepper noise right. And so. In order. To do, data augmentation, or data synthesis, you, can take some one, second clip so people say robber turn on in the quiet background and then, take some one second clip but people saying random words right, let's say you, know caught it all all, right it's, a static, and synthesized, this against, the train noise background and then, you would have in this case you would have what, sounds like rain noise train, noise train on street noise Robert turn on Trina Jana's or the called in I was in a row right, and then, you, could generate the labels, now as.
Girls. There once. There and. Then zeros there, because. If this is what he sounded like you know and I uses home then, you. Want the lamp to turn on after, Robert, turn on but now after these random, words you can take different random words, great. Um. So. Let's. See. Right. So. What. I'd like you to do is, evaluate. Three. Different. Possible. Ways to. Collect. Noisy. Data right. Two to. Collect this type of background data right. And. So. What. I'd like you to do for the next question, is let's say you and your team you, know have. Brainstormed. Brainstormed. A few different ways to. Collect. This type of background noise data. And. Let's. Say you've decided that. You. Would like to collect ten. Hours of background, noise data right. Okay. So. I'm gonna present to you three options, one. Is um. You. Know run, around Stanford, and replace microphones, around Stanford or, your friends homes do this with consent. And don't don't you know California, you're. Not supposed to don't record people about their knowledge and consent right Oh. Second. Is uh. Downloads. Clips, online. It. Turns out you can go to youtube there are these like, ten. Hour long clips, of. You, know rain. Noise or cars driving around. So. You actually. Can, if you do that find something that's creative common sense of appropriately, license right. Um another thing you could do is, use. A mechanical, turk. Amazon. Mechanical Turk we. Can, have people all around the world be. Paid you know modest, amounts of money to, submit audio clips right, so, for the next exercise what I want you to do because, I want, you that this exercise, of this discipline, which is what, I want you to do is um I went, to the estimate, let's, see what. Time is it no okay. It's 12:30 p.m. right now what. I want you to do is write, down three numbers, in the next exercise, to. Estimate, if. You were to do this. You. Know let's say you were to go do this right now right by. What time will you have finished, it, we went to do option one what. Time would you finish you were to do option two what, time would you finish you were to do option three if, you go is to collect ten hours of data through. One of these mechanisms, that. Make sense so it's 12:30, p.m. now so. What I'd like you to do is just write down three numbers. First. Number is what, time is it what. Time would be, by. The time you collected, ten hours of data you. Know from, around Stanford what time would be right. And if you could do this in some. So if you think you'd do it by tonight then right 9:00 p.m. if you think I'll do if you think it'll take you one week then, write the date one week from now right whatever it is but. Just write down three numbers to these three activities, okay let's. Do this one relatively. Quickly can people do this in like a maybe, a minute and a half. All. Right cool, this is interesting. Yeah. What the people actually surprisingly. Large variability I mentioned. One thing that. Surprised. Me, I'll. Give you my own assessment I think that. You. Know what I'm leading startup, teams we tend to be very scrappy, right and so I think that if. It goes to collect $10 of data if. You have three friends with laptop, you can collect three hours of data per hour because you have three recordings, going in parallel so, if I were doing this with say two other friends you know I bet, I bet we could get this done by tonight great, because. If you need nine hours of data that's. Each person needs to collect three hours of data and. You run around stand for the cue 2 microphones running, I bet, I bet I could get this done by 6:00 p.m. right, maybe, maybe even earlier I don't know. Download. Clips online. There's. Actually no it's such an interesting one maybe a bit about the same time it. Turns out one tricky thing about downloading. Clips, online is that. I. Think, a lot of the you, know there are people that have. Trouble sleeping at night so, they listen to highway, noise or whatever and. So there are these you, know 20, hours of highway, Clips highway, noise on YouTube that you can find but, I I don't know how those clips were generated, and I suspect a lot of them loop right. Meaning is the same one hour play over and over so, I actually think is harder, than then, one might guess to get 10, hours of. Non-repetitive. Data and, there's, one of those things you know if I take an hour of how highway, sound and loop it you. Can't tell the difference because all highway sounds sounds the same I just can't tell one.
Would Have highway sound from another one but if. You have one hour of highway sound looped ten times the, learning algorithm where she performed much less well then if you have ten hours of fresh, highway. Sound so this, I would have a harder time doing I think I probably I would. Proud if I were doing this I because. Of these problems, I work very budget, until. Sometime tomorrow. Right. Maybe. Maybe 9:00 p.m. or something maybe that's doable I'm not sure um the, one surprise to me was some people thought they could do this by tonight, I again. I've used Amazon Mechanical Turk is actually a huge process, to set up and some accounts are get, people on board and. Especially to get them microphone, I don't. Know if you implant something on flash they can speak to their web browser all. And then factually, supportive, is actually so, it's actually not that easy to get a lot of Turkish to do this and the global. Supply of it's also unlimited, so I would if, I were doing this I would probably I. Know. Maybe a week or something right hard to say. But. So the. Specific opinion isn't that important, but I want you to go through this exercise because, this, is how. Efficient. Startup teams should you, know brainstorm, a list of things and then you all figure, out how long you think it'll take to do these things and I think, we, can have a debate about how high quality, the dangerous, I think you can get very high quality data, from this and from this I, I. Just didn't trust a lot of those online audio sources, but. This is really fast and you, can get pretty high quality data I would probably do this to collect the background sound to. Get going right, but I think that part, of the workflow I see of you know fast, moving teams is, pretty. Much exactly what you did which is why I just have that exercise of brainstorming. The list of options and then really estimating, what, time can we get this done and then, use that to pick an option right um. And then. I wanna just mention one, last thing. Which. Is that. These differences. Matter. Right. Um. You. Know I've actually a Pulaski. System, for long machinery systems but, oh and. I think by the way if you do everything we just described and you see this later, in a problem set you. Can actually with, this set of ideas pretty, much the set of ideas that we just went through today you can actually probably build and build a pretty decent, trigger where detection, system or way question or detection system and in fact well, I'll see that you're pretty much this in the later homework, exercise, but now you know. When you get to that power exercise, when you do are an ends up you, know how you could come, up with this sort of process yourself if you didn't already know how, to make these types of choices. Affect. My, results, by, the way the beginning I think, that it's, not super sense which like many micro, moments or lights is the same as the one that's used when. I run the run standpoint that's what I don't keep with my best but my data, points. You, have to yeah. So my advice ed what does, your microphone affect your results right yeah my, advice it would be to get. Something going quickens, are here and then. Develop. A dev set right. With the actual types, of data you think you get on your real, microphone and then see if it is a problem and it may be in different microphones, do have different characteristics and if it is a problem then, go back and think about how you collect, data there's more representative. Upon you yes. I want. To mention one more quick thing there were kind of hot surveys, I wanted to do something real quick which is um I want, to tell you why these things really matter which is um if, this, is a performance. Right. Unless. They actually missing error. And, this. Is tiny right. And if, this is today, and. The other see all this data remember yes that's what we're doing just Lessing and, this is six months from now and, this is 12 months but now.
You. Know maybe of a competitor, actually maybe maybe I don't, know maybe. A because we talked about this so much in this class it may be two of you in this posture gonna build this sauce up but I said it may be a competitor, um. But over. Time mostly. Machine learning teams. You. Know the error actually goes down over time as you work on problems right I mean this is what I see in tons of practical. Projects you know you work on the project improve. The system and the error actually, goes down over time as you work on this over, the next 12 months a right if you really see over started doing this and, it turns out that it's a start up to have the discipline, to constantly, be the most efficient. Don't. Do something that takes you two days if you can get a similar result in one day, the, difference is not that you're one day slower the, difference is that you're 2x, faster right, and then then having that mindset if. We take this whole chart and compress, it on the horizontal, axis um. Then. You. Want. To be the startup that you, know makes the same amount of progress in six months instead of 12 months variant, because. If. You're able to do this then your startup, will actually, perform much, better in, the marketplace assuming. Your accuracy is important, which it seems to be for wake where it and so, don't think of this as saving you a day here and there think of this as making your team twice as fast and that's, the difference between this level of performance, and downloadable so, that's why when, I'm you, know building teams executing, these projects, I tend to be pretty obsessive about making. Sure we're very efficient, in exploring, the options and don't, wait till tomorrow to collect data of dubious quality when you have a better idea of, collecting. Data by today because the difference is not that you wasted 12, hours the difference is you had twice as slow as a company, right, so I think so. Hopefully through this example on your ongoing, experiences. Throughout this course I can help you continue, to get, better at, this right, um. Last thing we want to do was. We're about halfway through the course I'll go ahead um we, want to hand out a survey, of an, anonymous survey, to. Get some feedback, from you about this class and whenever, we get these surveys, we. End up, thanks. To the previous generations, of students feedback we've already been gradually, making cause better so. I think Ken, and I actually, read all of these questions ourselves and, try to find. Ways to take your feedback to improve the class so I can take you know five minutes to felt. The survey and you can hand it in just drop it up anonymously up here in front but, very grateful, for your, suggestions. Okay so. Um I. Think. If you haven't entered, your. ID yet, you, can still do so but that's, it for today so please start the survey and anonymously. Drop that back, in front then we wrap okay thank you.
2019-04-09