hello everybody um my name is Aaron Kessler I'm the assistant director for learning sciences and teaching for MIT open learning the Residential Group thank you for joining us today for our October X talk using digital Technologies to refresh on-campus courses over time I'm thrilled to be here on behalf of Jovi nazareno uh and uh sort of the person behind making all of this work and bringing this all together we welcome you and we're thrilled to have you here for the conversation today today we're discussing ways to engage in continuous Improvement of on-campus learning experiences using digital Technologies it's a real pleasure for me to to Welcome to the panel Dr Michael Short Dr schwart is associate professor of nuclear science and engineering and the 2022 winner of the MIT Bose award for excellence in teaching he's joined today by Dr Sean Robertson a lecturer and associate director of the Helena Foundation Junior laboratory which hosts the experimental physics Advanced laboratory sequences 8 13 8 14. and finally uh Dr Monica avello 701 introductory biology instructor and a PhD Alum a post-baccalaureate former push Baccalaureate student and most notably in my connection with her postdoctoral associate in mitx biology digital learning lab um so a quick note on process our participants have received the prompt that I'm about to put up ahead of time so they can sort of prepare their thoughts so they'll share those individually and then we'll come together and have a discussion at the end um so uh so let's begin by putting up the prompt um and then I'll have Dr Short come up so what we asked is you know can you tell us a little bit about the context of your course help us understand what what students you're you're supporting where they are in their sort of learning trajectory within MIT and and their sort of goals so are they majors are they just in the class because it's a gir that type of thing um and you know how have you prioritized what changes to implement in next week's class next year's class maybe some future version of the class how do you make those priorities and decisions and then how have you rolled out different uses of technology and what does it look like to think about that Improvement sort of over time you know what are the sort of key decisions that you sort of thought about as you approach that work so welcome Dr Short to start us off today I'm Mike short I'm a lifer I started coming to MIT in 1998 as part of the ESP your educational studies program as a high school student and despite many times trying to leave never did because it's too much fun and we do things pretty awesome here also had the distinct pleasure or not of teaching a course that I took as an undergraduate so I teach 2201 intro to Nuclear Physics or ionizing radiation or as I like to call it radiation for everyone I'll preface what I'm going to show you with a couple of things which is that a lot of these Tech Innovations I'm going to show you are not specific to nuclear almost all are generalizable you'll see the couple that are pretty nuclear specific but of course there's going to be some domain specificity to what we do and while they look impressive they're not hard to implement and that's why I have to give a shout out not just to the folks who implemented and got canvas going and there's a whole team from programmers to student helpers that helped us make the transition to Canvas but also to those who came before and did learning modules in Stellar because that is the single biggest thing that if you take the time to use a system regardless of bugs it might have this is what helps you do both short and long range planning and pivoting that's the basis of some of what I'm going to show you today so my course is an evolution from the way that we used to teach things in the 1960s this is what a typical classroom looked like this is definitely what a traditional Nuclear Physics classroom looked like and what you don't see because you're seeing the backs of students faces is very few eyes are open and this is a major problem in our field and required us to innovate because people chose not to do nuclear because it was boring or hard or inaccessible or there are all sorts of other issues and so in 20201 we take things a little differently the on the non-tech side we'll do things like Source honey from Chernobyl and make measurements or have students burn thousands of pounds of bananas to measure the radioactivity but the most important change we made was no prereqs it's an introductory course anyone should be able to take Nuclear Physics that's easier said than done but a lot of the tech surrounding educational Innovation makes it possible and there are a few maxims I live by that help me pivot the course in the short and the long term so the goal is to make the course Dynamic respond to both students needs and rapidly evolving field to make it personal because nothing gets set something in someone's mind and creates a scaffold to close a knowledge Gap like personal experience in something as dangerous as nuclear should be empathetic to know that the professor actually cares about the students as people and is not simply there to grade them as if they were cuts of beef but is interested in making whole students Tech actually helps that in a few ways that I'll discuss and to make things Equitable this is one of the best and least utilized things on canvas that I want to highlight to everybody and in long term it helps keep the course contextual the field evolves once every 20 years a nuclear reactor seems to blow up we get to talk about that end user that we're in the news all the time right so it helps us keep the course contextual it helps keep the course transparent so people know what's going on behind the scenes and why from projecting learning objectives to adapting in real time it keeps things accessible to different types of Learners as a as a former MIT student myself I did not learn well by being talked at and shown things on a board and said go read the book and take the test and a couple times I was told or if you can't read the book and take the test you're not good enough and you should drop the course before you fail and that's not a growth mindset so in this class simple Tech helps break that pattern and finally to make it archival because in the end you want to leave something for more people than just the students who teach that day and boy does Tech help in that in a lot of ways in terms of keeping things Dynamic rethinking the idea of a syllabus it's no longer a thing that you hand to students at the first day of class having something like canvas means that as you change things there's one place to find everything that the students have to do for those who wake up at 2AM and say gosh what do I have to read or whatever do today it just gives you a to-do list of do this by this time and you'll be okay or catch up tomorrow and you'll also be okay in terms of making things personal this is the probably nuclear one that I'll mention is I might ask you a question about how to calculate something or other about a bunch of isotopes or I may ask the students for some toenail clippings which would then stick in our nuclear reactor and get gamma Spectra from them but the fact that everyone has a device now means I can give them this software and instead of saying here is a spectrum that someone collected in a paper I can say analyze your toenail Spectrum on your machine and tell me how much arsenic you have and that makes it really personal luckily no one's had a reportable level so I can just definitely say that right away it helps us be more empathetic and you might not think of tech and empathetic as in the same sentence but what it does is allows students time to see distributions but it also opens up opportunities to affirm values for example that the people in our class are not distributions and so every time now that you have grades you can instantly see where you fall on some normal or whatever not normal curve it provides the instant opportunity to remind everyone that no grade is final it's not an assessment of your intelligence we already kind of did that in admissions there's no point in doing it again and we're here to help you grow not to grade you on a curve by the way grading on a curve is illegal at MIT for anyone who's listening and finally it helps us make things Equitable and this is one of my favorite features so I will take a minute to explain this I started rethinking assessment what's the point of a quiz it's not really for me to grade students it's for students to have a full Stakes personal but low stakes grade total assessment of what have I actually learned confront one's weaknesses and have an opportunity to strengthen them and when you rethink the idea of how should a quiz be done that's made me realize the way we do things in Academia was mainly for the convenience of the instructor it's a lot easier for us to give an exam in one room when we're right there and we can monitor everything and we can write questions when you have to get a number and monitor that there's cheating or isn't so I've got a five-year experiment granted by the uh by the registrar to let all quizzes be taken at home and that allows me to release quizzes to different students at different times so that I don't want to know what you know at a certain point in space-time I want to know what you know when you're at your best like in the real world you'll if you're asked to do something it's rarely going to be life or death that you have to do it that second it's more I want to know what you've learned and that helps me identify knowledge gaps to build the student and let them do a do-over of the quiz if necessary because my goal is to assess that they've learned the material by the end of the course not by a specific instance in space-time let's say when the all the other midterms are happening the long-term things helps us keep it contextual for example when the Fukushima disaster happened that provided well it was a bad accident and it was a good learning opportunity so we can pull things I would say MIT libraries is one of our best and most underutilized pieces of tech along with the Librarians who are not pieces of tech they're fantastic but it lets us ask questions like how much more radioactive is the ocean due to Fukushima send the students to the paper which they all have access to on MIT campus and so they can make all of their assessments very real answer questions they're going to get at the Thanksgiving dinner table by their parents so you're Nuclear So didn't Fukushima make it so we can't eat fish anymore well they know how to answer that question it helps us make things transparent 30 lines of code is all it takes to lower the barriers to zero so that students feel comfortable telling me what I'm doing wrong I wrote a 30 line PHP thing that just sends me an email with what's in the comment box and that way as soon as students see that I address these comments instantly in class you get instant feedback doesn't mean I'm going to do what the comments ask me to do but it means I'll address them and that way we project that we're listening and we're adapting and there are some things that are asked to be done and if we don't do them I have to say why right it puts me on the spot Tech enables that it makes it fast and instant it makes things more accessible and this is to Learners of different types so for example before every class I have a before class reading most of our classes have that but I also have a before class viewing that I used Zoom during the pandemic I did synchronously teach my classes but I also recorded them to the cloud so that students can then watch the derivations and things later it's a lot nicer when you don't have to rely on your notes which you might have missed something later on I remembered to wear a black shirt so they could see the math in front of my chest it's archival where something as simple as using your cell phone to take pictures of the blackboards it's not just a matter of getting the notes onto paper it's the arrangement of the visual elements to bring back those cues and student experiences so every year we get pictures of the blackboards students miss something in the notes they've got it right there it takes six seconds and that simple little bit of tech makes an enormous difference when it comes to studying for quizzes I know that I know from canvas engagement they get downloaded by our students and saved for later I hope they'll be useful for later with that I'll seed the floor and welcome our next panelist um right so I'm I'm Sean Robinson from the physics department uh where I run the the junior lab as was introduced earlier um do we have to brag about how long we've been at MIT is that the I've been here I've been here three years longer than than Mike has apparently um um also very fun um I'm from I am from the uh the associate director of the of the physics Junior lab but as of six days ago with the physics department now has a physics education Group which which I'm the head of so um so that's nice new developments which is all the academic instructional staff in in the physics department gives us a sort of administrative home so unrelated to the topic of today's things but it's nice that uh uh the 14 of us or so in the physics department are now are now actually a group um um also I suppose before I get started I was pointing out on the way here I discovered this typo on my slide um so I don't know if this is Freudian uh if this is just thinking about these this topic today mentally put us all back to to that fall semester um regardless it's certainly relevant uh today so not not on purpose but um but there we are um so uh so so on the subject of of uh continuous Improvement using using Tech um 100 goal is that you know the continuous Improvement is the goal right and the technology is a tool to get there um so the real question is is how to do those sort of things most efficiently how to use the tool best hopefully I guess not use the Tool destructively uh I won't comment on that but of course with something to keep in mind um so um so so I'm going to talk about uh junior lab in in course 8 as the uh as an example of of something we've been doing so first I should just for context um talk about the the class itself because I'm sure not everybody has the physics Department's curriculum memorized um so so uh junior lab it as the name implies is of course usually taken by Third Year physics majors it's a two semester sequence uh course 8 13 and 8 14. um most uh physics Majors given the the way our requirements are structured for graduation most students are required to take the first semester 813 most are not required to take the second semester 8 14. then to be a little more of a self-selected group of people who want to do experimental physics um the overall learning goal here is is really the professional development of of the student as a scientist people who teach this course whether it's our course at MIT or the way this course slots in at physics departments around the United States all eventually come to to this this conclusion what these courses are for it's about the professional growth of the student it just so happens that laboratory physics is is a particularly good medium to achieve that goal whether the specific student is a you know a string theorist or an astrophysicist somebody who may never touch a piece of apparatus once they're professional or whether they go on into you know quantitative finance and are just a physicist who does physicist things in a non-physic setting it turns out there are many such people it turns out that the experience of what happens in a laboratory the sort of uncontrolled just barely chaotic uh settings of of having to encounter are real devices where you don't necessarily know not even what the answer is going to be but even what counts as a good answer uh like like a research problem um that turns out to be a particularly good medium for for teaching things like um um just the sort of the community of practice type things about attitudes language how professionals talk to each other uh judgment about when to decide the problem is could be better but I've got to publish I've got I've got to call it a day and go work on my other problem sets or whatever it is knowing when to stop because at this point in history uh the Notions of you know Albert Einstein working alone in his study as a lone genius that's just not how the profession works at all um thank you um people who within the physics whether it's MIT or the broader physics education Community around the country um who have not taught this type of class they probably took it at some point in their history they think of it as maybe um physics theory is very difficult so it's nice to have a Hands-On complement to it um where you see um how it's done in practice uh it turns out these courses are not very good at that uh there there is some data to support that um and it's also not about specific uh bench skills which really contrasts with most of our colleagues in Psy biology or chemistry where you're going to go to that laboratory class learn some and learn how to titrations or some specific thing you do and then you're going to graduate and go get a job doing that skill in Industry physics as a profession just doesn't work that way it's more about the modes of thought is what we really care about it's also apparently why people hire our graduates um so um so that's a little bit about about the the course um itself and its goals uh one of the questions how we got to those goals so so I've been in charge of this course for uh about about 13 years um we've been um working really continuously on improving the course every every semester we do something new pretty much um for about since about 2010 or so um and one of the first pieces of this is just figuring out what those goals were what what was the the thing on the previous slide what should it be um um so there's a question about how did we get there and this is again not necessarily a technology question I don't know shortcuts I I think you actually have to do the leg work we talked to a lot of alumni we talked to a lot of current students current instructors former instructors people who teach similar types of material at other institutions uh um employers who hire people what do they care about what we do and and so um and and uh it was really useful to eventually articulate those goals that this is a course where a lot of other courses people learn how to do a lot of in our case physics I might say science in general but then there's this particular course in the curriculum where people learn how to be a physicist and that's that's our goal here um um so yeah just talking to lots of people that was not you know a canvas thing that was not a that was not a um uh a technology uh thing um so ultimately we did get there um but then we had to design a course that executed those things um what do we ultimately landed on uh is is a course where in the first semester where a lot of these people have not even seen uh Hands-On work before um you have uh a sequence of exercises on this sort of multi-dimensional learning curve learning surface um many many in-class exercises on basic skills and Concepts and then they go into these longer experiments that are the main meat of the course um uh one of the things they have to do is make the most of that in class time when they're with um uh with the the instructors in the lab or my own time one minute yes thank you um it's all made possible by a lot of online Preparatory work um uh and then in the the second semester also so people have to know what they're doing before time to get into class so for us in Circa 2011-12 wherever we were um this was this was mitx for us being able to deliver that content so students come to class already knowing what they don't understand and we have very very efficient uh discussions um that something we did for a long time keep on tweaking the the details of that uh but eventually um like everybody else everything got upended in the past couple of years we had to make some changes and we learned a ton of things um so this is kind of where we're going now um so we all learned Zoom was great for giving classes uh and Junior lab students have to give presentations and of course those were all uh done by Zoom like every scientific conference was done uh for the past uh uh for that for that stretch of years um um now we're back to being in person and people are giving in person presentations but we want to record all those presentations we had a clunky in-house system um because students have to review those uh those presentations as part of their cim requirements um and so now we we've figured out well actually we all know how to use zoom now it's a just turn around the laptop and click the button um it's not as good as what our colleagues from AV will give us but uh but it gets the job done we started using Slack for in-class communication we use canvas and the messaging uh announcements is great uh this class runs like a big research group uh The Faculty use slack as their thing so lots of channels channels for each of the experiments within slack people can report bugs um there's a typo in the lab guide we find out right away we don't find out three weeks later when everybody got it wrong um uh each lab partnership the students work and partners we can see their chat with each other and we're included in those private channels they can they can pop Us in um a post in the general channel on a slack workspace is a lot more active back and forth than making an announcement in the in like on the canvas or Stellar uh front page it's a conversation instead of an announcement that's actually really cool um so we've started uh we used to have like this big clunky reservation system a calendar for when you'd Reserve different piece of apparatus now someone just goes in the experiment Channel and says I would like you know the NMR device Thursday of two to five pm thank you um finally is the the biggest thing it's the thing in progress uh we are experimenting with MIT Exxon canvas um one thing we did have with all these things was like it's at some point seven different websites the students had to create identities on in logins there was a big administrative overhead and we heard about it every single year in the in the course evaluations and the faculty hated it everybody hated it um we've Consolidated almost all of it into canvas um the one last thing that wasn't was all that Preparatory material which we had already written in mitx uh the problem types didn't exist in in canvas so it was very hard but a lot of people perhaps some people here I'm not sure um have have worked on figuring a way to embed mitx canvas as in a site sorry mitx content as assignments in canvas um it's not really obvious how to make this work well but it is working right now I believe we're the only group at MIT that's that's doing this it is um it's fascinating it is working it's actually working uh but there's definitely uh there's definitely stuff to learn from that so uh but I'm definitely out of time uh so that's that's where we're at things are keep on going hello so if you forgive me I'm going to be a little old school my name is monia Bello um I am the 701 which is introductory biology instructor a little bit about what that means for some of you who might not be familiar so 701 is introductory biology it is a general Institute requirement or a gir which means every undergraduate at MIT must take our class or pass out of it in order to graduate which means that we are serving a wide range of folks a lot of first years but also a lot of second year third years fourth years there's a real mix now in who we are serving in our classes and these classes are obviously Undeclared folks to course majors and majors and we get the whole gamut they can range anywhere from 60 students just 60 to 425. um in my short time being an instructor which is since January 2021.
um and this every class there's four flavors of these classes there's two of them per semester and every single 701 has two different professors who have different desires and expertise and things they're very excited about and teaching teams that can range anywhere from three Tas modest to 20. and then me and then however many students there are and so I think something that I just want to give you so the appreciation of is we have a very broad audience the broadest possible the entire MIT undergrad population and I as an instructor I am actually balancing and trying to consider what works the best for both the professors and what they are Desiring for their vision of this class they set the syllabus they delivered the lectures Aya system and basically everything they do a teaching team who is helping facilitate the learning and on the ground with the students and then obviously our students who we want to all have a working understanding of biology because we think it's important for being informed citizens in the world whether they also believe that or not and so um I think what I want to say is I have a very unique position in that I started when we were completely remote if you might recall in 2021 we weren't back yet and so it means that when I did this I guess I'm a lifer too I was here in 2010 as a post-bac student took a bunch of courses as a student was a graduate student he ate a bunch of courses and I was a postdoc built a bunch of courses and now I'm an instructor and I coordinate a lot of courses so I have in fact T8 some of these courses that I'm in and oh I lost my train of thought but I think I was probably saying something about the interesting challenge of we are negotiating very interesting interests everything we do as a design has to be based on the principle of really two things like how are we going to maximize our students experience we want things to be efficient we want things to be consistent accessible and we want them to find biology to be fun especially if you're like no please not biology we want them to be like no but it's really cool like we want you to actually understand this this is important you are living um and we want the teaching team to be effective and we want them to enjoy themselves like we want the professors to feel like they're having a good time get to connect with the with the students we want the teaching teams to actually teach our Tas teach students in recitations which means they get their own individual learning experience as teachers we want them to have that and we want the students to learn we want to give them feedback we don't want grading to take up a lot of time and so any decision we make we have to first ask what's the goal like why are we using the technology when I came into this position I had let's see I have everything and plus some folks have mentioned we had canvas as the LMS it's great great scope or mitx for assessments Zoom as a way of conversing um remotely Piazza for students to talk to us and to each other pull everywhere for polling whether we did it asynchronously or during classes and slack which we primarily use for the teaching team we've experimented a little bit with students but maybe not to the extent some of my other co-panelists have described and so we've made a bunch of decisions and I think I want to actually highlight a little bit of like the process of how we've done it as more generalizable principle and then if I can with enough time I'll describe one example and so I think the process is very simple one get clear on the goal like what are you trying what are you trying to do if you're going to use technology it can't just because it's bright and shiny um it should connect to something you care about that you've set for either your teaching for your students or perhaps for both and then you know implementation really matters no matter how great the tool is if you botch something you traumatized everybody involved YouTube students you traumatize yourself you traumatize your professors your teaching assistants and so um either spending a lot of time in the implementation I know a co-panelist mentioned interviewing or talking to a lot of people this is very valuable you have not done this in a vacuum other people have expertise go seek them ask their advice implementation matters and the other is do a safe to try experiment and that starts small like don't commit to completely flipping your entire classroom semester around maybe start with a lecture have a taste and then obviously you want to gather feedback the best feedback is some Metric that you think is a measure of success are your students actually showing up are they engaged do you actually ask them for their feedback and they tell you this is working dear teaching assistants like what you're doing and then iterate and so you can every round or maybe perhaps even in a lecture whatever grain size your is your safe to try experiment a semester a recitation session can you improve can you gather what's good can you try to get rid of what's bad and can you keep going and can you grow and as I think someone said beautifully the goal is obviously continuous growth and so hopefully you never stop and you're never static um now I've been here very shortly so I've only been serving as an instructor for a little over a year and so I want to give you like one example of this just as a quick Evolution um exams we all take exams our students take exams we grade exams we Implement them when I started remotely um in let's say 7012 our largest class 425 students easy um all our exams were on grade scope they were remote they were 24 hour because who knew where our students were in the world Then Fall 2021 hit and we were back on campus for the first time and we were all not very sure of what was going to happen in in the semester were we all going to go remote again like what's even happening and so we again did remote 24 or perhaps a 12-hour one we did a couple iterations in different classes and it worked but my professors had concerns about academic integrity and we had some instances where it did look and some students did in fact unfortunately confess that they had cheated because they could and that was really disappointing and so coming back to this semester um there were all these great things about great scope which is the technology we used on exams it's from a grading perspective really wonderful it's really cool from a student perspective you can write psets where you ask people to upload videos like demonstrating some principle and you can grade it really easily you can ask all sorts of questions that are interesting and yet close-ended and graded so that Tas are really focusing their time grading free responses where students are actually articulating biology in their own words so they're like really getting to see students actually try to articulate biology which is really cool and they get to focus on that it's great in terms of consistency because we can actually see how everyone's grading and we can assign people to questions and so that students are being graded evenly there's a bunch of great things about it and so I asked my professor could we do this again the kui asked the students to come in in person and then I feel like that meets your interest of preventing any um perhaps uh the dark siren Call of looking up things on Google if you're taking your exam at 2 A.M in the morning and feeling desperate um or let's see in my house but still be able to give students a consistent experience their peace sets are on great scope we don't want to suddenly switch to paper also I don't want to scan 425 exams that's not a good use of my time when I'm one instructor split across two courses at first per semester and my professor very reasonably said I'm willing to do this that sounds good for the teaching team which is important consideration and she said the only way we're going to do this if it's good for the students I don't want my students to come into a class have to take a computer frazzledly finish something in 50 minutes rush off to the next class disrupting everybody what if the technology fails is the internet connectivity good enough all of these were really valid questions and so I said if I can answer every single one of your concerns and convince you that the implementation of this will be Flawless can we commit to this design decision tldr she said yes spoiler alert we've already done this we had 362 students take computer exams across three different lecture halls with a bunch of teaching assistants and Proctors and no Tech failures except for somebody not bringing their computer or their computer crashing and we had planned for it because we spent hours talking about the implementation of how we were going to do this we talked to isnt I interviewed course 6 instructors who already do this for their very large classes and got a lot of advice in order to make this a success and so I think my time is probably up but I just um want to say that in summary I think technology there's a lot of really cool technology especially after the pandemic there's a lot there and I think what makes the most sense is to always use your own goals for yourselves as the instructors and the teaching teams as well as your students balance obviously you want to maximize everyone's experience to the best possible and ask how can I implement this successfully how can I do this in a modest way how can I expand it if it's really really cool how can I keep getting feedback and iterating in essence you're being a learning engineer but how can you do that and that's what I would recommend and what I hope to continue doing in the 701s as an instructor supporting my professors and my teaching teams and students thank you thank you you just want to stay up there and we'll we'll join wonderful have a discussion thank you so much I uh there's there's two as everyone comes up there's two pieces that struck me and I'll sit down here as well um the the first one just to build off your last piece which was sort of that idea of learning engineering something we talk a lot about in our group and sort of have been thinking about what does it mean to sort of cycle through these iterations what does it mean to to collect evidence and and understand how that impacts the decisions about design and I think it's a really good frame for thinking about some of the the thoughts and questions that'll sort of come back to I was struck by a couple of pieces from the group as a whole which was the first one you know everybody came back to goals everybody came back to like what what do we actually hold valuable for this course so whether that be from an individual course perspective and an individual instructor whether it be from a sort of the the group and the sort of profession whether it be from a community of of people who are teaching these courses sort of down the line one of the things that I think is really important is understanding the sort of needs of the community and so how do you know which ideas and goals to pursue like I how have you made the decision of like okay you know there are seven things that I could potentially address this next iterative cycle but I'm going to choose these three or these two I'm just curious if anybody's had some experience in That explicit sort of like decision process and what went into it yeah go ahead sure yeah I could say in our case um we're almost always operating in a resource strapped uh version so so it's it's almost always the decision is not sort of what's the most educationally useful thing it's usually what's the lowest hanging fruit it's not the the you know most noble of approaches but it's what we what we can get done we could go for the thing that's clearly the best choice but we if we see this we're probably just not going to be able to get it with the with the uh with the available hours of Staff time with with whatever budget is available if we just can't do it then we will try something else and we sort of put it on a list for the future of of other things we want to try you know when it gets to some critical moment we really have to do the big thing so it's usually it's usually as as pedestrian as that for us that makes sense I think you know context matters right where were the decisions that we're making are situated within the environment that we're trying to do that so uh Dr shortiev yeah I guess I would try to balance things like repeatability of experiments versus immediate feedback and I have one story to relay so in 2020 I taught lightboard synchronous remote recorded of lectures in 2021 we were back I flipped the classroom with the students some somewhat permission it was a great hit so I said let's do it again and I got so many Anonymous rants of this just isn't cutting it for me that I said okay let's have a poll and then the day after I got those uh comments we did a poll and I said look if it's 50 50 split we're gonna have to keep with flip because it's in the catalog it was a violent vomiting objection to the flipped classroom there was a couple people who just said no I am sick of online video I want to come in and be talked to cool switch the next day left the videos up there for everyone who still wanted them so I guess balancing repeating these experiments see if they work with the knowledge that everyone every year you have new people with different experiences and different desires I think that last bit is really important in understanding that um repeatability in the that the those Learners are changing their experience what they've come to expect or want out of out of different time from different times in different parts of their learning may change as well yeah I wanted a plus one on the the low hanging fruit or feasibility part that Sean mentioned because it's stuff sometimes something is easy or like low hanging fruit why not right like it's not hard to implement that and then sometimes it's because you're like strapped for time and you're just like this is this is what's happening right now um I think if I were to add anything else I will say in my perspective because I am often responding to folks desires I will say sometimes it's as simple as my my professor telling me it's really important to me that students engage in class and that like we really know if they're getting the material like in real time lump sum and they say we need to make polling happen like it needs to happen and I'm just like yup all right I'm Gonna Learn pull everywhere because you've told me this is a priority and I will throw my life into figuring it out and so sometimes it can also be as simple as there's one thing we really want to accomplish can you please make this work and then just pick one thing and try it and you know um from last time I tried this um and we're doing it again this semester I have learned a lot about how to make this happen and I think I have gotten one step closer to my professor's vision of what in-class polling looks like and it is much better this semester compared to the first semester I tried it and so I think the you know I think it made sense the goal was we want to engage students in class and we want to make sure they understand and we said that was a high priority and so we picked one thing and I said this is new for me I didn't do it during the zoom times let's make this happen and I've picked one thing and this semester we're doing it again and we've made it more intense we do polling competitions we're like hitting recitations against each other and then like we like buy them snacks and like people are designing t-shirts and like it's getting close to the vision of when my professor had once done in a former class that she also taught and she's very pleased and I'm very pleased I think it's really cool and the students seem to be really engaged in coming to class I I think we'll uh there's a really one piece I want to pull out of that specifically is a it's a great example and it's an example of that it's not necessarily going to work perfectly that first round right that part of the iterative process is understanding you know where were the failure points and are they addressable you know where where was it that things went really well and where was it that things maybe didn't go as well as we wanted and how do we address those pieces rather than just saying well the whole thing didn't work right and sort of understanding the value proposition of of that trade-off which is sort of inevitable which which brings me to sort of the next piece which is one of the things that I think I heard across all of the the conversation was feedback loops the sort of idea of you know there's sort of this collection of evidence there's this collection of ideas and and um you know in your example that sort of the polling and in these pieces and I'm curious what other evidence you utilize that maybe is a straightforward is sort of something directly from the feedback but maybe is is more open other people might not think of it as as evidence maybe it's your perceptions of preparedness for class or maybe it's other parts of your experience in facilitating you know communities within the the group so it's it's how many people are actually posting on that on the slack or on the discussion board I'm just curious you know where you all sort of consider those those points of potential data collection well I have a couple ones that that folks may not have often thought of I took our department through abet accreditation and one of the things that are one of the best pieces of feedback we got was from our students employers saying what our students were really good at and what their student what they wish our students were really good at another one was I had assumed you know naturally that people would do the reading for courses and I noticed a distinct linear drop in lack of preparedness over time starting about 2014 when I first pulled folks it was 80 percent by 2020 it was 20 so naturally now should be zero so that's what led to the impetus to start flipping things which happened started happening a little before the pandemic but of course that kicked it into overdrive and it made it actually happen and now we go back students are like I would rather do the reading than watch another video but the point is to ask and adapt and show that you're open to that feedback great yeah I don't know I think like I'm thinking of only the obvious things like when students actually show up to a thing is like a really obvious metric and we try to collect all those things oh non-obvious I suppose you know I think someone mentioned like canvas clicks or like things I guess like for those who don't know there are like analytics on canvas and so you can act like there's some really cool technology data about like you know I see the like exponential curve of how many pcset submissions are started like right before the deadline so you get a sense of like when students are actually starting their submissions or like as simple as how often our students on canvas or like what had they actually gone and looked at the solutions key you posted and so there are ways to actually see like where students are going within your digital sort of resources and just taking a glance at that can sometimes be really helpful because you might notice like wow like everyone's like clicking on this like one thing and like I wonder why and then you can like send your Tas out into the field to ask their students and report back to you um and so I haven't done that in Earnest but I do definitely use it when I'm trying to see if an unengaged student isn't like really anywhere or are they active at all but I think it can also yield a lot of insights if you're like even if you just look at it bird's eye view and anyone who's a coder you can do some really cool things with it so highly recommend yeah I know when we first started putting our Preparatory work these again these are basically uh quiz reading quiz type questions you know did you read the lab manual before you showed up to the first day of the experiment that's it's those type of materials when we first started using that on mitx there was this you know qualitative perception amongst the teaching staff that oh my gosh everyone knows what they're doing all of a sudden and but but you know there's no real metrics for that we all saw the transition uh but but how do we back that up uh eventually uh Aaron and I actually did dig into a lot of mitx Click data and timing and uh see was it really true that people were um actually engaging that material um there's a whole nother talk there's a whole book chapter on on what we did but uh but but the conclusion was yes people actually were engaging material before coming to class I rather you know simple conclusion um but at least you know it gives you something something quantitatively that we could back up the very very qualitative very soft uh observations of the sort of The Craft of the expert instructor in the room so there are there are those things um I don't know as the as someone who's uh you know just a teacher not a data scientist you know that's so inaccessible uh to us um I haven't actually tried it in canvas at all but certainly mitx stuff the data was there but very you know very difficult to get at without expert help yeah so I I'll take this as a quick plug so thank you for that which is to say um our group in our office is always happy to collaborate with whomever has these types of questions right so if you're curious about these types of of pieces and you want to ask some more detailed questions or you want to have conversations about hey we're thinking about doing this but would really like to understand what's been going on maybe with some of the data before we're happy to to help facilitate that conversation and exploration and and do some of the work collaboratively so happy to do that thanks for the plug um I did that was not prompted I promise um I I will I will close out with with asking this this sort of general question which is you know all three of you having engage in forms of iterative improvement in your specific courses if you had maybe some advice to give others who might be just starting to really think about this for whatever reasons maybe it's they've come back from the pandemic and they really want to transform what they've been doing or maybe it's just they're seeing some changes in the students they want to address them curious you know generally speaking what advice might you give others who are starting off in this journey of sort of iterative improvement it's a tough one right now yeah I mean I will throw my answer out into the void um I would say um I would probably say this human um talk to a lot of other people so just like survey the scene there's gonna be my answer you've got to talk to some other people um and I think so like one just like build your networks MIT um is a wonderful collection of many kingdoms which unfortunately means we often get siled and so um finding people who are doing things that are really exciting and just collecting things and then feasibility like don't change everything pick one thing that you're like I could do this within your context are you one professor and you call all the shots are you in a huge team you gotta negotiate like what are you capable of um and then start small just try it just try it and then collect feedback see how you feel about it and move from there and it can be as modest as that but if you have the appetite for it and you like get this process there's a lot that you can do and so that's what I would say to a human who was like baby yeah I've I've got something else I'd add too is now that there are so many tools out there I would say the same thing that we ask our students to do which is to do your prep work and start as soon as you can my fall classes are all fully prepared before the fall semester starts I started doing that because when I didn't do that it was a constant scramble and things got dropped but what that also does is it forces let's say funneling you to organize your course through canvas also funnels your brain to organize your course in your mind and then you're going to come up with ways of using new things that you've read about heard synthesize when you're on a run when you're sleeping when you're supposed to be paying attention at a faculty meeting or whenever the opportunity strikes you and to give your subconscious time to create those ideas it's non-deterministic I don't think anyone's ever found a way to be like here's how you become more creative the only thing I could say is give yourself the time and it'll pay back in spades when you teach the course yeah and I would say if I I'd love to say this is a research and development process you follow a scientific method but uh for me it's really you have you have a gut instinct that comes from your observations of everything you've been an expert in this environment so a lot of times what we're doing is craft uh we should do the legwork to understand the science and understand the tools and understand the engineering uh but but uh I I found you know letting letting your own sort of intuition guide you for a while but use those tools as a constraint that keeps you on the straight and narrow sometimes your intuition is wrong a lot of people's intuition about how Learning Happens is just absolutely wrong I think probably if we're in this room probably agrees with that and it takes a while to learn how to be a good teacher uh so um you know but but you sort of you know follow the path that seems right in your trust your own expertise but also uh kind of lean on others because sometimes you just really are doing something that's dumb and sometimes other people can help bring you in so working as a team and and trusting yourself but also trusting people around you and valuing their opinions is huge well thank you all for sharing your ideas your Journeys your advice uh it's been tremendous and awesome and I know it's going to be a wonderful resource that we can share with many Beyond just those that are in the room so please join me and thank you everyone and um [Applause] I do want to take a quick second for those that are in the room if there are any questions where we can sort of take a few here at the end and before everybody has to run now I can jump in on that um more and more I feel like I'm in good company one of the most important things that's happened and we have talked about is that mit's incentive structure for faculty is changing when I started here I was told by some not everybody to be an okay teacher don't spend your time in a great being a great one because your research will suffer and now excellence in teaching is explicitly valued in tenure in promotion that's probably the biggest single thing that could happen because it tells everyone starting out be a great teacher and it will help you it won't just be a marginal Improvement it'll be really noticed and we talk about this kind of stuff a lot we're a small department and nuclear but we have to be ahead of the curve because our field is so contentious politically motivated personal it affects everyone and small yeah for me I know it's a little bit of all the above for the most part yeah the physics Department's actually a really good Community where we're teaching good teaching is valued uh you know as a as a staff member we're in principle the research faculty could you know certain things flow downhill it could be the worst job in the world uh if if it was like nobody has to care about teaching but no it's actually people respect what we do and and value it most of the The Faculty care very much about doing their teaching well and and want the the research and development and the classroom support um that we can provide and uh some people are just like okay the teaching is good enough you know I I saw good teachers when I was a student I just imitate them and that gets you okay it gets to the good enough bar of University teaching which is not necessarily awesome but it's it's good enough and then there's people who really want to become as part of their professional identity as a my case as a physicist they also want to be a professional educator they want to go that extra that extra level and um it's certainly not every member of the physics department but there's a nice core of people who who want to do that so yeah it's not it's not so bad yeah also in biology I'm very fortunate having come from I'm still part of the mitx biogroup um which is part of the digital learning lab and that also obviously does not just edx and massive open online courses and like cool outward facing stuff but also helps with a lot of residential education and so I think having come from being a postdoc in that position and moved over to an instructor I'm like still very adjacent and obviously working with everybody still as colleagues and so my experience but I think this might be new um I'm not sure if my predecessors necessarily felt the same way but in my current position as an instructor in the biology Department I feel like I have 10 different professors who care very deeply about doing a good job and engaging the students and I have a bunch of colleagues who I trust who are very skilled and knowledgeable and that I am sort of a conduit between the two of them and so I think and that we have a bunch of Junior faculty and they all seem to be very exciting and there are members of our group who are engaging them in pedagogy and and some of them are coming into the 701s and so I think um from my view but very you know historically not old it seems like we have a lot of interest that is just like growing in our view apartment yeah I want to thank everyone again for coming and sharing and being part of the conversation if you've got any follow-up happy to stick around for a few moments but uh I want to be cognizant of the time and let everybody go so thank you everyone [Applause] thank you
2022-11-10