Engaging technology to support effective feedback for learners

Show video

Hello everyone and welcome to another  episode of Integrity Matters by Turnitin.   My name is Chuks, and today, with me in  the house, is our Chief Product Officer   Annie Chechitelli, who is going to be exploring  and having a conversation with me with regards to   engaging technology to support effective feedback  for learners. Hello Annie. Hey, happy to be here.   Thank you. Just like we do for all our  guests we have on Integrity Matters by Turnitin  we would like to get a little bit about your academic and professional background.   Can you tell us a little bit about that and what  excites you about the work you do at Turnitin?   Okay, so my career has always been - at least  for more than 20 years - in education technology   software, primarily in product management and  product marketing, so I would say my career has   been dedicated to building tools really, in  the scope of online learning, digital learning   or ways of using technology to improve the  process, as well as the outcomes for teachers   and students. I started out at a startup called Wimba, it did virtual classrooms   which was purchased by Blackboard in 2010, and  then I stayed with Blackboard for a few years   and then I wanted to see what it was like at a bigger  company, to figure out what it means to actually   scale technology. So, I spent about five and a half  years at Amazon in their Kindle for school work  

in government group, and also spent some time in  AWS, so that's my work. My undergraduate degree   is at Columbia in New York City, in engineering, so  I've always been more on the tech side, but really   liked also the business and user aspect of it.  For me, I really wanted to understand the problems   I was solving for customers. The technology was  great, but I really gravitated towards   roles and opportunities where I could just  understand the impact, and the education   technology space at the time where I started  my career was just incredible luck, right?   The internet was just exploding and so I thought it  would be a great opportunity to do something new   and it's really surpassed my  expectations in terms of passion and excitement   and I'd say my favourite part of it is working  with educators. The commitment that   educators have to their crafts, and I call it  a craft because there's a lot of   science that goes into it, but it's also thinking  about their curriculum and thinking about their   objectives and thinking about assessment, and how  that all comes together in this kind of alchemy to   create this course for their students - it's  been incredibly inspiring to see their work.  

I remember also in doing product management,  early in the days, one of my favourite stories is   we had some prototypes but they weren't clickable, they were just really screenshots   and we would send them to our customer  advisory board. The best part is, and you don't   get this in other industries, but in education,  I had some of the educators   would red pen, so they would just take a  red pen and basically grade our mock-ups   and they took such care, and then they  would like scan them and send them back   and if you work in corporate and there's a user of   an expense report, or an ERP solution, it  doesn't matter to them in the same way - they're not going   to take out a red pen and tell you all the ways  that they need it changed to reach students   so I find that interaction an impact on faculty  really, what keeps me going every day.   Nice! I'm very very excited to hear about  how you work with educators. My first question  

is going to be around how you stay up-to-date with  what's happening in the educational sector. We know   that things are fast evolving and for anyone  who is managing a product, especially something   around assessment or even research - you need to  be up-to-date with what's happening   so you can meet those demands, so how do you  stay up to date-up-to-date with what's happening   in the industry? Well, number one, I read a lot. I think generally  you know, my wise words to anyone - always take the time.  I always read all the journals that are coming out   from a publication perspective, every morning I leave  about half an hour to read everything that's come   out overnight, and some of it's education, right?  But I also look at other technology trends that   are affecting other industries. You can learn  a lot about the applications, for instance, of AI   by kind of looking at other industries and then  thinking about how it applies or how it could   apply to educators and education, so I make sure I  can stay up to date on those aspects in terms of   lots and lots of reading. I'm also a fan, in  education it's very much - at least pre-Covid   very much an event like trade shows, conferences  kind of organisation, and so I built a lot of my   career just going to events. I went to every event  I could and I'm obviously - well you can't tell obviously 

but I'm not shy, and so I usually  when there's like a lunch or people are sitting around I would just sit down at those round tables  at a conference and introduce myself  and I'd ask questions, and everyone's very  open in those environments, so I try to learn   really kind of what they're grappling with  and some of the things they are there for  why are you here, what are you interested  in learning here? So a lot of that, a lot of conferences  I think is helpful - both education and  technology. I went to 'AWS Re-invent' a few   years ago which is really a lot about kind of new technology around AI and ML (machine learning), and so that was   an opportunity for me to kind of dive in and learn  about that. I would say staying in touch with users   and balancing that with reading - if you  just read everything, just like in life, you can   be a student but you're not really a good student  until you reach out and you try to practise.   And so I think you have to have a combination  of those two things in order to kind of progress   and keep current, and also challenge your own  assumptions. Quite interesting and exciting how  

you stay up to date, especially with your very  busy schedule, but I'm glad to hear that you're   doing a lot of reading. With this level of reading,  exposure to conferences and roundtables   what would you say are some feedback and assessment  trends that are sort of informing product design?   So, obviously we've had a lot of change with Covid  in terms of the trends that are happening and   quite honestly, what happened in a matter of six  months was probably more than we'd   had in 20 years. I always say, you know, I built  back in 2000, I had built web conferencing for   virtual classrooms for learning, and back in  those days, I had to teach somebody like here's   how you use a microphone - this is before it was all  built into to the computer - and people didn't   really adopt it. There were certain folks who  would adopt some of these tools here and there   but you know, as a result of Covid, everyone  started and it went really quickly and so   there was a lot more experience using online tools  and figuring out how to use assessment   in an online environment and what that meant,  but experience doesn't always result in comfort.   Sometimes it does, sometimes the more you do it  you're more comfortable, but sometimes you're still   just not great at it and it doesn't always  result in best practices, so there's definitely   more comfort - I mean there's more experience and  a little bit more comfort, but we're still kind   of at the point of trying to figure out how to  use it best, and I think that there's an   understanding now, like "oh we don't have this all  figured out", right? We definitely have some steps   and we know what direction, what's directionally  correct, but even though Covid   changed a lot, some things are still the same, right?  You're going to have users, you're going to have teachers   who are going to push the boundaries; they embrace the technology and they think about   their pedagogy as it pertains to how can I advance  that with technology and they really want to   experiment and try new things, and they're always  pushing us as a company to give them more   and to make the tools more powerful or more  transparent and powerful in reaching the student.  

So, we're going to always have those and so  they're challenging us, and on the other end of the spectrum   you're always going to have new teachers  who haven't used the the software or who aren't   comfortable with doing assessment online, and it's  a challenge, the challenge for us is   how do you reduce the complexity for  that first time user or even just the occasional user?   We know how it is, you think  you know software, but then you go away for   four months or even two weeks, and you come back  are you're like: 'how did I do that?', so how do you reduce the   complexity but still maintain depth? And that's  really kind of the art of what we try to do from a   design perspective, of making sure we're supporting  the advanced use cases and the advanced teachers   but making it comfortable and designing it in such a way that some of the power   reveals itself as the instructor uses it,  and so it's more of a guide for them   as opposed to every feature in their face on  the user experience, and how do we help do that?  So like I said, it's a challenge, but it's  a challenge that we enjoy and this is why we   have to work directly with users, because we're  not - the one thing I know and I always tell my team   the one thing I know for sure is that we're gonna  get it wrong, and what I mean by that   it sounds bad, but bear with me, the reason  I say that is because we're not the teachers   and so what we need to do is have hypothesis  and recommendations and put together things that   a teacher can react to, to give us  that feedback. It's really important   not to just get their verbal feedback - I like this or I like that - but it's really important to also   watch what they're trying to do and  especially around assessment, to ask them   'what are you trying to achieve'? We can  talk about where the buttons are and how that   clicks through, but what goal or problem are  you trying to solve and thinking about that   that's kind of how we think about design in terms  of assessment and how we kind of adapt to these   trends as they're happening. The other  thing to note is as a result of Covid   I think student expectations have  changed and so students - I mean before   students were used to a consumer-grade technology world where I want it now   I mean, I can say this because I worked at  Amazon, where it's: 'I want it now, I want it delivered   tomorrow, why is it taking so long, I don't  want two passwords, why is this difficult'?   They have very little tolerance for kind of a  not-streamlined experience, but now they're looking for   their education to be a lot more flexible,  that they expect hybrid. Even if they're  

on ground or they're in a k-12 institution  and they're in grade school or in higher education   they definitely want to make sure that they have  access to things online, even if it's just to   reinforce, like 'hey I want to rewatch that lecture,  and they kind of expect that, even if it's   an in-person class, and so we're trying to think  about some of that in our tools. For instance   I think Gradescope is a really good example  of this, of trying to bridge that   physical world and taking a physical test  that's written out and bringing that into   digital in a way that we can help the instructors  be more efficient, we can also help them kind   of understand, work through, grade and  provide that feedback quicker. Those are   some of my thoughts on that. Great, so when thinking  about the importance of feedback for learners   how is this taken into consideration in the design of  technology? So, the relationship between a teacher   and a student is so powerful. That's really the  most - that relationship and how the student can   grow from that feedback or from that mentorship is  incredibly important, and so what we know for   sure is we don't want to impede that relationship.  We don't want to get in the middle of that  

relationship in a way that will make  it difficult or that won't allow that to shine   that kind of feedback back and forth, so  our job is to make that relationship stronger   by providing different mechanisms to give feedback  in a timely manner so that students can absorb it.   There's an element that if you get feedback  you need to absorb it, learn from it   and then incorporate it into their future  activities in that course and beyond. So, we're   always striving to create that path that changes  grading into more of a conversation as opposed to   a thing in time that you do, right?  Yes, it's something that we all have to do but it's   not about 'it's done'; it's a constant  evolution in that conversation between the teacher   and the student, instructor-student, faculty-student that we're always trying to reinforce.   I would say that students don't always review their feedback, right? We're all   very busy and our brains are trying to always  create shortcuts, that's just how our brains work   and so a lot of times students will  just go right to the grade   and they won't look at the feedback, and that's  unfortunate, so we try to offer a wider range   for teachers to better support these students.  Whether that's to require the student  

to review and write a reflection on the  feedback before they can see their grade   so that's an option. Or maybe peer review, in which the student actually has to   have that critical eye on somebody else's work,  which will help them then judge their own work   when they're going to be developing those  critical judgment skills and that evaluation they   can apply to themselves; that's a good option, as well as requiring that this student   do a self-assessment to the rubric, so  tell me how you would grade this?   Here's the rubric. And it kind of makes them think through  the assignment and how they're growing   as opposed to just focusing on the grade.  I quite like how you put it there. It's about   the relationship between the student and teacher  and using technology to maintain that path.   I also want to call out a very good point  raised in terms of the conversation. Part of  

what we've been talking about over the course  of other episodes, is the need the move towards   dialogic feedback where we're having a dialogue as  opposed to just giving a blanket statement which   goes to all the classes, and 'how does this affect  me' is what the student will be saying, so thinking   more critically about how you personalise feedback  and how you involve that level of conversation   even if you're not doing it in the face-to-face  delivery. So, that's a really good point to call out there.   The norm for us, for educators out there, is to think 'pedagogy before technology'.   Now, most times, edtech companies are being called  out for thinking technology before pedagogy   so taking that into consideration, what  are some suggestions for educators engaging with   technology for feedback and assessments? So in  terms of pedagogy before technology, so my answer   to that, is yes and no.  I think that the right approach is always always   for educators to determine the best  way to assess based on the curriculum and the   objectives. Absolutely you should not  just jump into technology and just start cramming  

content and curriculum into it, thinking it's  going to magically create an engaging   course and outcome. With that said though,  sometimes technology can help an instructor   think differently about pedagogy. Maybe something  they hadn't thought was possible before   so I think about it more recursively  than as a linear cycle, just because I think   pedagogy is constantly evolving and so  is technology, so sometimes it's good   one of the principles I also like to abide by with  how I work and how my team works, is a little   bit of wandering is good, right? Sometimes people  think 'hey I'm going to be super efficient   this is what we're going to do, go build it',  and I think technology and pedagogy is one of   those things sometimes you have to wander a little  bit to see how it comes together a little bit more organically.   Obviously don't do it just because the  technology is there, but you know, I think the   technology has offered new opportunities to assess  that didn't exist before, and then has   really opened the instructor's creativity  to think about and maybe challenge some   of their methods before, and offer them new tools.  I do like the whole the recursive approach and  

it did actually get me wondering, because we're  talking about the need for wonder, so I've actually   in the course of my career as a learning  designer, we've been more focused on   understanding the pedagogy than trying to align  the technology, but you do raise a good point in   terms of thinking 'Ah, this is technology, what can  we use it for?' and can we also inform or facilitate   the pedagogy as well, so that's a really good point  you raised there. For educators out there   there is the need to make assessment work for learning,  so what are better strategies, or how would you   engage educators in the process of informing  best practices in feedback design; especially   knowing that we need to make assessment work for  learning? At Turnitin, we're always trying to   work within the educator workflows.  I don't want to create more work, I don't want to   take them out of something that they  already do and know and are comfortable with   so it's really important that we think  about feedback that is designed as   part of that everyday workflow so it's really easy. Because the goal is  

more timely and frequent feedback leads to more  engagement, and leads to better student outcomes   in general, so how do we make it just easy  and part of what they naturally do as teachers?   So, if we put it in those workflows, some  of those best practices just emerge organically   and so it's our job to design the solutions  alongside educators, to kind of create those   moments to see how that evolves organically.  You know, we'll learn a lot more   sitting side by side, about how the educator  thinks, about when, where and how much is helpful   versus how much does it burden them.  We don't want the teacher to say   'oh, feedback...'. That's not what we want,  so when we've gotten that far we've gone   there and you can kind of tell when we're  working with educators and we've done that   to kind of rethink that. And as I said before, it's also really important to  

to leave space for us with educators to help  us understand their goals at a higher level.   Many times, and this is just the nature of  the beast when you're building technology   like a conversation can get really granular, really  fast, like "I want this button here and I don't want   two clicks there", and those are  all good conversations, don't get me wrong, but they   can't be it, right? And so sometimes we  get lost in that and we don't kind of say   "well what are you trying to do? Maybe moving  the button is not - maybe we just rethink this whole thing".   And so it's important for us as we  work with educators to understand kind of the   tactical but also strategic, and understand  and kind of have that empathy towards what are   they trying to accomplish, in general. Maybe it's  not even feedback, maybe it's not even Turnitin.   What does their day look like?  The job of an educator is so   complicated and difficult; all we can do is  just try to empathise as we learn more about that   and at the same time, kind of get detail to see how  we can improve and make things more efficient.   This is a slightly - I'll call it slightly touchy subject  right now in terms of how big data and AI has been   introduced in education. When we think about, to  your point, with regards to supporting educators  

in terms of other efficiencies or moving  towards better practices in your feedback design   what role does big data and AI have to play in  effective feedback design and how might educators   engage with AI in practice? Yeah, and this one isn't  that complicated. I think that people expect   this really fancy answer, but it's not  for us. For our products, the role of AI   is to save instructors time and to allow  them to focus on the art and science that   is teaching, and by revealing hidden  patterns that they might not be able to see   or by automating tedious tasks such as grading  the same type of answer again and again   by kind of taking some of those repetitive - I  call it undifferentiated heavy lifting   it's not really unique it's just  stuff they have to do, but it's not really   adding to their relationship with the student - if we can take that some of that off and make it   easier and make it faster, we give them more time  back to think about the feedback they want to give   and to really absorb how the student  is interacting with that feedback   so that's generally how we think about kind of AI at a high level.  It's also a teaching tool, so when we were  designing our formative tool Draft Coach   we made the conscious decision that we were not  going to solve all the students' problems, so we're   not just going to say: 'here's your draft, fixed'.  We got feedback from the instructors  

who said: "no, no, we don't want you to do our job,  we want to do that job. You can guide them   you can give them some information to help  them continue to get better", but essentially   we're not replacing that ultimate relationship  between the teacher and the student and that's   really important for us. We're passionate  about helping them do their job. I think you sort   of responded to my next segue in terms of  how we position AI, so we're looking at how   best to position the power of AI in assisting  with feedback as opposed to replacing the human   approach there. Have you got more to discuss  in terms of how we would best position the power of AI?   Yeah maybe just kind of sum it up like AI is about extending   and quickening than human reach and insight. It's just getting there faster and easier, and I just   think of it as like a teacher superpower; we're giving a teacher   superpowers that they can then use to do their  job more effectively. Just for a proper wrap-up  

what are some informed strategies to support  educators in engaging technology to support   effective feedback for their learners? It's really timely, relevant, personalised feedback   That will allow spending the time on that as an instructor   so that that will allow the student to  kind of take an active participation role   in their growth. We all know  that education is a marathon, it's not a sprint   and by allowing students to take that active  part in their own growth, it gives them agency   they feel they have a role and they are in  control of the outcome and their learning   and that's what keeps a student engaged.  A student isn't engaged when they kind of shrug   their shoulders like "well I don't really know  how to impact this, I don't know how to get   better or I don't know why I got that or I  don't know where I stand or how I'm growing".  

You know, every time they can see the results  put to work, it continually motivates them in that cycle.  Thank you so much Annie for shedding  light on this and also your expertise and insights on   how we can best engage with technology to  support learners with effective feedback.   Always a pleasure talking to you and thank you  again for your time today. Thanks for having me!

2022-09-10

Show video