Hello everyone and welcome to another episode of Integrity Matters by Turnitin. My name is Chuks, and today, with me in the house, is our Chief Product Officer Annie Chechitelli, who is going to be exploring and having a conversation with me with regards to engaging technology to support effective feedback for learners. Hello Annie. Hey, happy to be here. Thank you. Just like we do for all our guests we have on Integrity Matters by Turnitin we would like to get a little bit about your academic and professional background. Can you tell us a little bit about that and what excites you about the work you do at Turnitin? Okay, so my career has always been - at least for more than 20 years - in education technology software, primarily in product management and product marketing, so I would say my career has been dedicated to building tools really, in the scope of online learning, digital learning or ways of using technology to improve the process, as well as the outcomes for teachers and students. I started out at a startup called Wimba, it did virtual classrooms which was purchased by Blackboard in 2010, and then I stayed with Blackboard for a few years and then I wanted to see what it was like at a bigger company, to figure out what it means to actually scale technology. So, I spent about five and a half years at Amazon in their Kindle for school work
in government group, and also spent some time in AWS, so that's my work. My undergraduate degree is at Columbia in New York City, in engineering, so I've always been more on the tech side, but really liked also the business and user aspect of it. For me, I really wanted to understand the problems I was solving for customers. The technology was great, but I really gravitated towards roles and opportunities where I could just understand the impact, and the education technology space at the time where I started my career was just incredible luck, right? The internet was just exploding and so I thought it would be a great opportunity to do something new and it's really surpassed my expectations in terms of passion and excitement and I'd say my favourite part of it is working with educators. The commitment that educators have to their crafts, and I call it a craft because there's a lot of science that goes into it, but it's also thinking about their curriculum and thinking about their objectives and thinking about assessment, and how that all comes together in this kind of alchemy to create this course for their students - it's been incredibly inspiring to see their work.
I remember also in doing product management, early in the days, one of my favourite stories is we had some prototypes but they weren't clickable, they were just really screenshots and we would send them to our customer advisory board. The best part is, and you don't get this in other industries, but in education, I had some of the educators would red pen, so they would just take a red pen and basically grade our mock-ups and they took such care, and then they would like scan them and send them back and if you work in corporate and there's a user of an expense report, or an ERP solution, it doesn't matter to them in the same way - they're not going to take out a red pen and tell you all the ways that they need it changed to reach students so I find that interaction an impact on faculty really, what keeps me going every day. Nice! I'm very very excited to hear about how you work with educators. My first question
is going to be around how you stay up-to-date with what's happening in the educational sector. We know that things are fast evolving and for anyone who is managing a product, especially something around assessment or even research - you need to be up-to-date with what's happening so you can meet those demands, so how do you stay up to date-up-to-date with what's happening in the industry? Well, number one, I read a lot. I think generally you know, my wise words to anyone - always take the time. I always read all the journals that are coming out from a publication perspective, every morning I leave about half an hour to read everything that's come out overnight, and some of it's education, right? But I also look at other technology trends that are affecting other industries. You can learn a lot about the applications, for instance, of AI by kind of looking at other industries and then thinking about how it applies or how it could apply to educators and education, so I make sure I can stay up to date on those aspects in terms of lots and lots of reading. I'm also a fan, in education it's very much - at least pre-Covid very much an event like trade shows, conferences kind of organisation, and so I built a lot of my career just going to events. I went to every event I could and I'm obviously - well you can't tell obviously
but I'm not shy, and so I usually when there's like a lunch or people are sitting around I would just sit down at those round tables at a conference and introduce myself and I'd ask questions, and everyone's very open in those environments, so I try to learn really kind of what they're grappling with and some of the things they are there for why are you here, what are you interested in learning here? So a lot of that, a lot of conferences I think is helpful - both education and technology. I went to 'AWS Re-invent' a few years ago which is really a lot about kind of new technology around AI and ML (machine learning), and so that was an opportunity for me to kind of dive in and learn about that. I would say staying in touch with users and balancing that with reading - if you just read everything, just like in life, you can be a student but you're not really a good student until you reach out and you try to practise. And so I think you have to have a combination of those two things in order to kind of progress and keep current, and also challenge your own assumptions. Quite interesting and exciting how
you stay up to date, especially with your very busy schedule, but I'm glad to hear that you're doing a lot of reading. With this level of reading, exposure to conferences and roundtables what would you say are some feedback and assessment trends that are sort of informing product design? So, obviously we've had a lot of change with Covid in terms of the trends that are happening and quite honestly, what happened in a matter of six months was probably more than we'd had in 20 years. I always say, you know, I built back in 2000, I had built web conferencing for virtual classrooms for learning, and back in those days, I had to teach somebody like here's how you use a microphone - this is before it was all built into to the computer - and people didn't really adopt it. There were certain folks who would adopt some of these tools here and there but you know, as a result of Covid, everyone started and it went really quickly and so there was a lot more experience using online tools and figuring out how to use assessment in an online environment and what that meant, but experience doesn't always result in comfort. Sometimes it does, sometimes the more you do it you're more comfortable, but sometimes you're still just not great at it and it doesn't always result in best practices, so there's definitely more comfort - I mean there's more experience and a little bit more comfort, but we're still kind of at the point of trying to figure out how to use it best, and I think that there's an understanding now, like "oh we don't have this all figured out", right? We definitely have some steps and we know what direction, what's directionally correct, but even though Covid changed a lot, some things are still the same, right? You're going to have users, you're going to have teachers who are going to push the boundaries; they embrace the technology and they think about their pedagogy as it pertains to how can I advance that with technology and they really want to experiment and try new things, and they're always pushing us as a company to give them more and to make the tools more powerful or more transparent and powerful in reaching the student.
So, we're going to always have those and so they're challenging us, and on the other end of the spectrum you're always going to have new teachers who haven't used the the software or who aren't comfortable with doing assessment online, and it's a challenge, the challenge for us is how do you reduce the complexity for that first time user or even just the occasional user? We know how it is, you think you know software, but then you go away for four months or even two weeks, and you come back are you're like: 'how did I do that?', so how do you reduce the complexity but still maintain depth? And that's really kind of the art of what we try to do from a design perspective, of making sure we're supporting the advanced use cases and the advanced teachers but making it comfortable and designing it in such a way that some of the power reveals itself as the instructor uses it, and so it's more of a guide for them as opposed to every feature in their face on the user experience, and how do we help do that? So like I said, it's a challenge, but it's a challenge that we enjoy and this is why we have to work directly with users, because we're not - the one thing I know and I always tell my team the one thing I know for sure is that we're gonna get it wrong, and what I mean by that it sounds bad, but bear with me, the reason I say that is because we're not the teachers and so what we need to do is have hypothesis and recommendations and put together things that a teacher can react to, to give us that feedback. It's really important not to just get their verbal feedback - I like this or I like that - but it's really important to also watch what they're trying to do and especially around assessment, to ask them 'what are you trying to achieve'? We can talk about where the buttons are and how that clicks through, but what goal or problem are you trying to solve and thinking about that that's kind of how we think about design in terms of assessment and how we kind of adapt to these trends as they're happening. The other thing to note is as a result of Covid I think student expectations have changed and so students - I mean before students were used to a consumer-grade technology world where I want it now I mean, I can say this because I worked at Amazon, where it's: 'I want it now, I want it delivered tomorrow, why is it taking so long, I don't want two passwords, why is this difficult'? They have very little tolerance for kind of a not-streamlined experience, but now they're looking for their education to be a lot more flexible, that they expect hybrid. Even if they're
on ground or they're in a k-12 institution and they're in grade school or in higher education they definitely want to make sure that they have access to things online, even if it's just to reinforce, like 'hey I want to rewatch that lecture, and they kind of expect that, even if it's an in-person class, and so we're trying to think about some of that in our tools. For instance I think Gradescope is a really good example of this, of trying to bridge that physical world and taking a physical test that's written out and bringing that into digital in a way that we can help the instructors be more efficient, we can also help them kind of understand, work through, grade and provide that feedback quicker. Those are some of my thoughts on that. Great, so when thinking about the importance of feedback for learners how is this taken into consideration in the design of technology? So, the relationship between a teacher and a student is so powerful. That's really the most - that relationship and how the student can grow from that feedback or from that mentorship is incredibly important, and so what we know for sure is we don't want to impede that relationship. We don't want to get in the middle of that
relationship in a way that will make it difficult or that won't allow that to shine that kind of feedback back and forth, so our job is to make that relationship stronger by providing different mechanisms to give feedback in a timely manner so that students can absorb it. There's an element that if you get feedback you need to absorb it, learn from it and then incorporate it into their future activities in that course and beyond. So, we're always striving to create that path that changes grading into more of a conversation as opposed to a thing in time that you do, right? Yes, it's something that we all have to do but it's not about 'it's done'; it's a constant evolution in that conversation between the teacher and the student, instructor-student, faculty-student that we're always trying to reinforce. I would say that students don't always review their feedback, right? We're all very busy and our brains are trying to always create shortcuts, that's just how our brains work and so a lot of times students will just go right to the grade and they won't look at the feedback, and that's unfortunate, so we try to offer a wider range for teachers to better support these students. Whether that's to require the student
to review and write a reflection on the feedback before they can see their grade so that's an option. Or maybe peer review, in which the student actually has to have that critical eye on somebody else's work, which will help them then judge their own work when they're going to be developing those critical judgment skills and that evaluation they can apply to themselves; that's a good option, as well as requiring that this student do a self-assessment to the rubric, so tell me how you would grade this? Here's the rubric. And it kind of makes them think through the assignment and how they're growing as opposed to just focusing on the grade. I quite like how you put it there. It's about the relationship between the student and teacher and using technology to maintain that path. I also want to call out a very good point raised in terms of the conversation. Part of
what we've been talking about over the course of other episodes, is the need the move towards dialogic feedback where we're having a dialogue as opposed to just giving a blanket statement which goes to all the classes, and 'how does this affect me' is what the student will be saying, so thinking more critically about how you personalise feedback and how you involve that level of conversation even if you're not doing it in the face-to-face delivery. So, that's a really good point to call out there. The norm for us, for educators out there, is to think 'pedagogy before technology'. Now, most times, edtech companies are being called out for thinking technology before pedagogy so taking that into consideration, what are some suggestions for educators engaging with technology for feedback and assessments? So in terms of pedagogy before technology, so my answer to that, is yes and no. I think that the right approach is always always for educators to determine the best way to assess based on the curriculum and the objectives. Absolutely you should not just jump into technology and just start cramming
content and curriculum into it, thinking it's going to magically create an engaging course and outcome. With that said though, sometimes technology can help an instructor think differently about pedagogy. Maybe something they hadn't thought was possible before so I think about it more recursively than as a linear cycle, just because I think pedagogy is constantly evolving and so is technology, so sometimes it's good one of the principles I also like to abide by with how I work and how my team works, is a little bit of wandering is good, right? Sometimes people think 'hey I'm going to be super efficient this is what we're going to do, go build it', and I think technology and pedagogy is one of those things sometimes you have to wander a little bit to see how it comes together a little bit more organically. Obviously don't do it just because the technology is there, but you know, I think the technology has offered new opportunities to assess that didn't exist before, and then has really opened the instructor's creativity to think about and maybe challenge some of their methods before, and offer them new tools. I do like the whole the recursive approach and
it did actually get me wondering, because we're talking about the need for wonder, so I've actually in the course of my career as a learning designer, we've been more focused on understanding the pedagogy than trying to align the technology, but you do raise a good point in terms of thinking 'Ah, this is technology, what can we use it for?' and can we also inform or facilitate the pedagogy as well, so that's a really good point you raised there. For educators out there there is the need to make assessment work for learning, so what are better strategies, or how would you engage educators in the process of informing best practices in feedback design; especially knowing that we need to make assessment work for learning? At Turnitin, we're always trying to work within the educator workflows. I don't want to create more work, I don't want to take them out of something that they already do and know and are comfortable with so it's really important that we think about feedback that is designed as part of that everyday workflow so it's really easy. Because the goal is
more timely and frequent feedback leads to more engagement, and leads to better student outcomes in general, so how do we make it just easy and part of what they naturally do as teachers? So, if we put it in those workflows, some of those best practices just emerge organically and so it's our job to design the solutions alongside educators, to kind of create those moments to see how that evolves organically. You know, we'll learn a lot more sitting side by side, about how the educator thinks, about when, where and how much is helpful versus how much does it burden them. We don't want the teacher to say 'oh, feedback...'. That's not what we want, so when we've gotten that far we've gone there and you can kind of tell when we're working with educators and we've done that to kind of rethink that. And as I said before, it's also really important to
to leave space for us with educators to help us understand their goals at a higher level. Many times, and this is just the nature of the beast when you're building technology like a conversation can get really granular, really fast, like "I want this button here and I don't want two clicks there", and those are all good conversations, don't get me wrong, but they can't be it, right? And so sometimes we get lost in that and we don't kind of say "well what are you trying to do? Maybe moving the button is not - maybe we just rethink this whole thing". And so it's important for us as we work with educators to understand kind of the tactical but also strategic, and understand and kind of have that empathy towards what are they trying to accomplish, in general. Maybe it's not even feedback, maybe it's not even Turnitin. What does their day look like? The job of an educator is so complicated and difficult; all we can do is just try to empathise as we learn more about that and at the same time, kind of get detail to see how we can improve and make things more efficient. This is a slightly - I'll call it slightly touchy subject right now in terms of how big data and AI has been introduced in education. When we think about, to your point, with regards to supporting educators
in terms of other efficiencies or moving towards better practices in your feedback design what role does big data and AI have to play in effective feedback design and how might educators engage with AI in practice? Yeah, and this one isn't that complicated. I think that people expect this really fancy answer, but it's not for us. For our products, the role of AI is to save instructors time and to allow them to focus on the art and science that is teaching, and by revealing hidden patterns that they might not be able to see or by automating tedious tasks such as grading the same type of answer again and again by kind of taking some of those repetitive - I call it undifferentiated heavy lifting it's not really unique it's just stuff they have to do, but it's not really adding to their relationship with the student - if we can take that some of that off and make it easier and make it faster, we give them more time back to think about the feedback they want to give and to really absorb how the student is interacting with that feedback so that's generally how we think about kind of AI at a high level. It's also a teaching tool, so when we were designing our formative tool Draft Coach we made the conscious decision that we were not going to solve all the students' problems, so we're not just going to say: 'here's your draft, fixed'. We got feedback from the instructors
who said: "no, no, we don't want you to do our job, we want to do that job. You can guide them you can give them some information to help them continue to get better", but essentially we're not replacing that ultimate relationship between the teacher and the student and that's really important for us. We're passionate about helping them do their job. I think you sort of responded to my next segue in terms of how we position AI, so we're looking at how best to position the power of AI in assisting with feedback as opposed to replacing the human approach there. Have you got more to discuss in terms of how we would best position the power of AI? Yeah maybe just kind of sum it up like AI is about extending and quickening than human reach and insight. It's just getting there faster and easier, and I just think of it as like a teacher superpower; we're giving a teacher superpowers that they can then use to do their job more effectively. Just for a proper wrap-up
what are some informed strategies to support educators in engaging technology to support effective feedback for their learners? It's really timely, relevant, personalised feedback That will allow spending the time on that as an instructor so that that will allow the student to kind of take an active participation role in their growth. We all know that education is a marathon, it's not a sprint and by allowing students to take that active part in their own growth, it gives them agency they feel they have a role and they are in control of the outcome and their learning and that's what keeps a student engaged. A student isn't engaged when they kind of shrug their shoulders like "well I don't really know how to impact this, I don't know how to get better or I don't know why I got that or I don't know where I stand or how I'm growing".
You know, every time they can see the results put to work, it continually motivates them in that cycle. Thank you so much Annie for shedding light on this and also your expertise and insights on how we can best engage with technology to support learners with effective feedback. Always a pleasure talking to you and thank you again for your time today. Thanks for having me!
2022-09-10