Hello everyone and welcome back to Art&&Code Homemade: digital tools and crafty approaches. I'm thrilled to welcome you now to the session with Irene Alvarado who is a designer and developer at Google creative lab where she explores how emerging technologies will shape new creative tools. She's also a lecturer at NYU's interactive telecommunications program. Irene Alvarado.
Hi everyone very excited to be here. I'm going to talk today about a behind-the-scenes story of a particular tool called Teachable Machine. So as Golan said I work at a really small team inside of Google. I mean we're less than you know 100 people which is small for Google size. And some of us in the lab, the work that we do includes creating what we call experiments to showcase and make more accessible some of the machine-learning research that's going on at Google. And a lot of this work shows up in a site called Experiments With Google.
And time and time away, we're sort of blown by what happens when you lower the barrier for experimentation and access to information and tools. And so today I want to tell you about one particular project and especially the approach that we took to creating it. So why am I speaking at a homemade event when I work at a huge corporation. I think you'll see that my team tends to operate in a way that's pretty scrappy, experimental, collaborative. And this particular project happens to be a tool that other people have used to create a lot of homemade projects. So to begin with let me just talk about what it is.
It's a no code tool to create a machine learning models. So, you know, that's a mouthful. So I think I'm just gonna give you a demo. So I believe you can see my screen. This is sort of the home page for the tool. It's called Teachable Machine. And I can create different types of what we call models in machine learning.
Basically a type of program that the computer has learned to do something. And there's different types. I can choose to create an image one, an audio one, a pose one. I'm just gonna go for image, it's the easiest one.
And I'm just gonna give you a demo so you see how it works. I have this section here where I can create some categories. So I'm gonna create three categories. I'm gonna call this
like neutral or I could call this just like face or something like that. And I'm gonna give the computer some examples of my face. So I'm going to do something like this, give the computer some examples of my face. Then I'm going to give the computer some examples of my left hand. And then yeah that should be enough. And then I'll give the computer some examples of a book, Happens to be a book that I like.
And then I'm gonna go to this middle section to sort of train a model. And right now, it's taking a while because all this is happening in my browser. So the tool is very very private. All of the data is staying in my browser through a library called tensorflow.js So one of the main points here was to showcase that you can create these models in a really private way. And so now the model's trained. It's pretty fast, and now I can try it out.
I can see that you know it can detect my face. Let's see, it can detect a book, and it can detect my hand. And, you know, some interesting stuff happens, you know, when I'm half in half out, you see that the model is trying to figure out okay is it my face, is it my left hand, you know. You can sort of learn things about these models like the fact that they're probabilistic.
And, you know, so far maybe not so special. I think what really unlocked the tool for a lot of people is that you can export your model. So I can upload this to the web and then if you know how to code, you can sort of take this code and then essentially take the model outside of our tool and put it anywhere you want. And then you can build whatever you want with it, a game, another app, whatever you want. So let me go back to my slides here. We also have sort of other conversions, right. So you can create a model and not just
have it on the web. You can put it on mobile. You can put it, you know, in Unity, in Processing, in other formats, and other types of platforms. And just a little note here of course this is not just myself. I worked on this project with a lot of colleagues, a lot of very talented colleagues of mine. Some of them are here. And I'm going to showcase a lot of work from other people in this talk. So as much as possible I'm going to give credit to them, and then at the end I'll
share a little document with all the links in case you don't capture them. So okay I want to emphasize that a lot of our projects have really humble origins, you know. It's just one person prototyping or sort of hacking away at an idea, even though it might seem like we're a really big company, or a really big team. And just to show you how that's true, you know, the origin of this project was actually a really small experiment that looked like this. The interface sort of looks the same, but it was really simplified.
Like you have this sort of three panel view. And you couldn't really add too much data. It's a really really simple project. But more so than that, even though it was technically very sophisticated, I think our ideas at the time were very- we were kind of exploring really fun use cases.
And just to show you how much that's true I'll show you a project that one of the originators of this idea Alex Chen tried out with his kid. So let's see [organ sound effect] [bird chirping sound effect] [quiet organ sound effect] [bird chirping sound effect] So you can see he's basically creating these paper mache, you know, figurines with his kid. And so he's training a model that then triggers a bunch of sounds.
So it's really kind of fun at the time, just trying a lot of different things out. And then we started hearing from a lot of teachers all over the world who are using this as a tool to talk about machine learning in the classroom, or talk about sort of the basics of data to kids. And then we finally heard even from some folks in policy. So Hal Abelson is a CS professor at MIT, and he was using the tool to conduct sort of hands-on workshops with policymakers. So we had a hunch that maybe the silly experiment could become something more.
But really really didn't know how to transform this into an actual tool. And we also didn't know necessarily what the best use cases would be. And this is where the project took a really really interesting turn, because essentially we met the perfect collaborator to help us and to push us into making it a tool.
And that person his name is Steve Salling. He was introduced to us by another team at Google who had been working with him. And Steve is this amazing person. He used to be a landscape architect and he got ALS which is Lou Gehrig's disease. And he sort of set out to completely reimagine how people with his condition get care. And he created this space that everything is API-fied.
So he's able to order the elevator with his computer or turn on the TV with his computer. It's really amazing. And so he actually found the original Teachable Machine, and someone else sort of was using it with him. And, you know, we basically got introduced to him, and the question was well
you know, can we figure out if this could be useful to him, and in what way? And, you know, how do we just get to know Steve, and what he might want? So a little pause here to say that, you know, folks like Steve they're not able to to move or you know in Steve's case he's not able to communicate. So he uses something called a gaze tracker or like a head mouse. And he essentially sort of looks at a point on the screen and then can type into- can sort of press click and type a word or a letter. So he's able to communicate but it's really really slow. And so the thought was: okay can we use a tool like Teachable Machine to train some, you know, train your computer to detect some facial expressions and then trigger something. And this of course is not new.
The thing that was sort of new was not for me to train a model for him, but for him to be able to load the tool on his computer and train it himself. Like sort of put that control on Steve. And specifically you know, we basically went down to Boston and worked with him quite a lot. He became sort of the first big user of the tool. And we made a lot of decisions by working with Steve, and sort of like following his advice. And one of the things that the tool sort of allowed us to explore was this idea of immediacy. So what were some cases where Steve wanted sort of a really quick reaction, and how could the tool help for this.
And one use case was he really wanted to watch a basketball game. And he really wanted to be able to cheer or to boo depending on what was happening in the game. And that was something that he was not able to do with his usual tools, because he had to be like really fast at cheering or booing when something happened. And so he trained a simple model that basically, he could sort of open his mouth and you know trigger an air horn. So that was one example.
So you know we kind of immersed ourselves in Steve's worlds. And by getting to know him, we got to know that, you know, maybe other ALS users could find something like this useful. So we started exploring audio. Like could we have another input modality to the audio to potentially help with people who sort of had limited speech. And that led us to incorporate audio into the tool. So I actually have a little example here that I also want to show just so you guys see how this works.
I'm loading a project that I had created beforehand from Google Drive. So let's see if it doesn't, you know, this is- this is some data that I had collected beforehand, some audio data. So there's three classes, there's background noise, there's a whistle, and there's a snap. And let's see if it works. [whistle noise] As you can see the whistle works.
[snap sound] You can see the snap works. So you know, same thing here. I can kind of export the model to other places. So you know, but the interesting thing here is that the audio model itself actually came from this engineer named Shanqing Cai. And he created that audio model for people like Mark Abraham who also have ALS through exploring with him the same idea, like how can I create models for people like that so that they can trigger things on the computer. So the technology itself, you know, also came from this exploration of working with users who have ALS.
And, you know, you can't see what's happening here, but essentially Dr. Abraham has sort of emitted a puff of air and with that puff of air, he's been able to turn on the light. So you know, we decided that okay this could be useful to other people who have ALS. And we decided to essentially open up a beta program for other people to tell us if they had other similar uses, and trying to think about maybe other analogous communities that could find you know, some interest in Teachable Machine. And that's how we met the amazing Matt Santamaria and his mom Kathy.
And Matt had come to us and told us that, you know, he was playing with the tool and he wanted to try to use it for his mom. So he actually created a little photo album that would sort of, you know, change photos for his mom, and he could sort of load them remotely. But his mom because she didn't have really too many motor skills, because she had a stroke, she wasn't able to control the photo album. So we actually worked with Matt. And, you know, you can't see it here because it's just an image, but we actually worked with Matt to create a little prototype of, you know, his mom being able to say previous or next, and then training an audio model that then was able to sort of like change the picture that was being shown on the slideshow. And that was just sort of like a one-day hack exploration that we did with Matt. And then ultimately he sort of kept exploring potential uses of Teachable Machine with his mom.
And he created this tool called Bring Your Own Teachable Machine, which he open-sourced and has shared online. And what it allows you to do is to put any type of audio model, and then link that to basically a trigger that sends texts- that sends a text to any phone number. So pretty cool to see what he did here. And then finally, you know, just seeing that the tool sort of ended up being useful in a lot of analogous communities once we launched it publicly. We saw a lot of uses outside of accessibility. So I wanted to show you a few of my favorite ones today.
This is a project by researcher Blakeley Paine She used to study at MIT Media Lab, and she was really interested in exploring how to teach AI ethics to kids. So she open-sourced this curriculum. You can find it online and it just has a bunch of exercises, really interesting exercises that she takes kids through. So this one for example
explains to them the different parts of a machine learning pipeline. So in this case, you know, collecting data, training your model, and then running the model. She sort of gives them different data sets. So you can see here in the picture, it's a little blurry, but you can see the kids got 42 samples of cats and then seven samples of dogs. And so the idea is for them to train a model, and then see okay maybe the model's working really well in some cases maybe it's working really poorly in some cases. Why is that? And have a conversation with them about AI ethics and bias and sort of how training a model is related to the type of data that you have.
Here's another example of her workshop. She sort of asks kids to sort of label their confidence scores. And you'd be surprised, you know. We were invited to join one of the workshops and these kids sort of established a pretty good kind of insight into the connection between how the model performs and the type of data that it was given.
It's not just Blakeley. There's other organizations that have created lesson plans. This one is called readyAI.org and, you know, again they kind of use these simple training samples in a lot of schools. You can't use your webcam, so a lot of them sort of have to upload picture files or photo files in order to to use the tool.
And that was a use case that we found out through education that we had to sort of enable, you know, not having a webcam. And then more so in the realm of hardware, there's a project called Teachable Sorter created by these amazing designer technologists Gautam Bose and Lucas Ochoa. And what it is is it's a very sort of powerful sorter. So it uses this- oops. It uses this accelerated
hardware board it's kind of like a Raspberry Pi. And they essentially train a model in the browser and then they export it to this hardware. And they both were super super helpful in sort of creating that conversion between web and, you know, this little hardware device. Now this is a very complex project. So they made a simple version that's open-source and you can find the instructions online. And it's a tiny sorter, it's a sorter that you can put on your webcam.
And so again you can train a model with Teachable Machine in the browser, and then export it to this little Arduino, and then sort of attach this to your webcam and sort things. In a different vein this is a project called Pomodoro Work and Stretch Chrome extension. And what it is is it's a chrome extension that reminds you to stretch. And the way it works is that this person trained a pose model that basically detects when people are stretching, right. There's a team in Japan that integrated Teachable Machine with Scratch. So scratch is a tool for kids to learn how to code.
It's a tool in a language and environment for children to learn how to code. And unfortunately a lot of the tutorials are in Japanese. But the app itself sort of works for everybody. So you can take any model and run it there. And then a lot of other just really creative projects like this one is called Move The Dude.
I can make this probably a little bigger. So it's a Unity game that you control with your hand. And your hand essentially is what's moving the little character around. So again because these models work on the web, you can kind of make a Unity WebGL game and export it. And then just one last example. A lot of designers and tinkers started using the tool to just play around with funky designs.
So Lou Verdun created these little sort of explorations. In this one he's doing different poses and matching those poses to a Keith Haring painting. And then in this one it's sort of like a cookie store, and then you can draw a cookie and get the closest looking cookie. And then this GitHub user SashiDo created a really awesome list of a ton of very inspiring Teachable Machine projects in case you want to see more of them. So just to sort of go back to the process of how we made this tool. We took a bit of time to think about this way of working and informally started calling it Start with One amongst ourselves, amongst my colleagues.
And it's not a new idea, you know. This is inclusive design, I'm not inventing anything new. Just the word Start with One sort of was a way to remind ourselves that you know, the tactical nature of it. Like you can just choose one community, one person, and sort of start there. And we're really just trying to celebrate this way of working this ethos.
Start with One was just an easy way to remember that. So just coming back to this chart for a second. This idea of like a tool that we created for one person ended up being useful for a lot more people.
But I want to clarify that the goal is not necessarily to get to the right of this chart. A lot of the projects that we make, they just end up being useful for one person or for one community and that's totally okay. And it's not necessarily the traditional way of working at Google but it's okay for my team. And, you know, this idea of sort of starting with maybe the impact or the collaboration first rather than the scale, it doesn't mean it's the only way of working. It doesn't mean it's the best way of working. It's just a way of working that has worked really well for- for my very small team. So there are a lot of other projects that sort of fit into this bill.
And if you're curious about them, you can see some of them in this page google.com/co/startwithone. It's also a page where you can submit projects, right. So if any of you have a project that fits into this ethos, you're welcome to submit there. And, you know, right now that times are really hard for people I think it's easy to be crippled by what's going on in the world.
I've certainly felt that way, maybe even a little powerless at times. And for me when I think of Start with One I remember that, you know, even small ideas can have a big impact if you apply in the right places. And I don't just mean pragmatic, right. Like human need is also about joy and curiosity and entertainment and comedy and love, you know.
So it's not just a practical view of this. So just a little reminder for all of you I suppose to to look around you, to collaborate with people who are different than you, or even people you know really well: your neighbors, your family. I think the idea is to offer your craft and collaborate with a One and solve something for them. You know, even if you're starting small, you'd be really surprised by how far you can get.
That's it. There's this link, a tiny url digital machine community. I'll paste in the discord. And you can find all the other links in my talk through that link in case you didn't have time to copy it down. Thank you very much. Thank you so much Irene. Yeah I'll keep this up for a little bit just yeah that's great. Yeah I'll keep this up for a little bit just in case- Yeah that's great. Thank you so much. It's beautiful to see all these different ways in which a diverse public has found ways to use Teachable Machine in ways that are very personal and often you know, single scale kind, of scale to what a single person is curious about.
Like a, you know, father and a child or, you know, people with different abilities who can use this in different ways to make easements for themselves. It's really amazing. I've got a couple questions coming from the chat. So you mentioned this Start With One point of view. Is this a philosophy that's just your sub-team within Google Creative Lab or does Google Creative Lab have a manifesto or set of principles that guide its work in general? And if so, what sort of guides the work there, and how do you fit into that? Yeah great question. No I wouldn't say it's like a general manifesto or even of the lab itself.
I would say it's a way of thinking within like my sub-team of the lab. And again like I do want to say like it's not like I'm talking about anything new. Like inclusive design and co-design been talking about this for ages. It's really just like a short word keyword for us to refer to these types of products, but I would say like the Creative Lab does pride itself in basically embarking on close collaborations. So we tend to see that projects where we collaborate very closely like not making for people, but making with people end up being better. So not all projects can be done in that way necessarily.
But I would say like a good amount are, yeah. And so I know that there's a there's a sort of coronavirus themed project that uses the Teachable Machine by I think it's Isaac Blankensmith which is the ANTI-FACE-TOUCHING-MACHINE. Where he makes a machine that whenever he touches his face it says 'no' in a computer synthesized voice. It's super homemade and it sort of like trains him to stop touching his face. But I'm curious how has the pandemic changed or impacted the work that your team is doing or that people are working with Teachable Machine, or that the creative lab is doing? How have things shifted in light of this big shift around them? Yeah I mean that's a good question. I don't know if I have an extremely unique answer to that except to say that we've been impacted like anybody else.
I mean luckily I am in an industry where I can work from home. So I feel incredibly lucky and privileged to be able to be at home safe and not be a front line worker. It's changed the fact that certainly collaborations are harder to do. Like you saw in the pictures with Steve, like we like to go to
where people are, and like sit next to someone and actually talk to them and not be on our computers. And that has certainly gotten harder. But we're trying to make do with things like Zoom and just like everybody else.
I'd say the hardest thing is not collaborating with people in person. And it just takes like a shift, because there's so much Zoom fatigue and you don't get to know people outside of your colleagues. Like when I'm trying to know a collaborator or someone outside of Google like anybody else, you know, you benefit from going to dinner or grabbing a coffee or just like being a normal person with them, having a laugh. And that doesn't really exist anymore. Like Steve actually the person that I was talking about who has ALS, he's so funny. Like he says so many jokes. He's so restricted with the words that
he can say because it takes him a long time to type every sentence, you know, it takes two minutes to type a sentence, but he's so funny, like he has such a great personality and humor. And I think that would be very hard to come off through Zoom starting with the fact that it's very hard for him to use Zoom because someone else has to sort of trigger it for him. So certainly that type of collaboration would have been very hard to do this year.
Thank you so much for sharing your work and this amazing creative tool. I like to think that there are several different gifts that each of our speakers are giving to the audience. And one of them is the kind of just the gift- of permission to be a hybrid, right.
Like, you know, you are a designer and a software developer and, you know, an artist and a creator and an anthropologist and all these other kinds of, you know, things that bridge the technical and the cultural, the personal, and the design-oriented, and all this together. Another gift is just the gift of the tool that you're able to provide to all of us, you know, me, my students, and, you know, kids everywhere and other adults. Thank you so much.
Yeah I mean final words is that it's it's a feedback loop, right. Like I was inspired by your work Golan. Like you give a lot of people in this community permission to be hybrids. And I didn't know that that was possible. And then everyone making things with Teachable Machine inspires us to do other things, or to take the project in another direction.
So I think we have the privilege and the honor of working in an era where the internet just allows you to have this two-way communication. And I think it makes so many things better. So thank you Golan, and thank everyone for organizing the conference like Leah, Madeline, Clara, Bill, Linda. You guys all make hybrids around the world possible.
2021-05-03