How to Block AI Cheating in Online Exams | Exam Tips & Technology
[Music] welcome we are going to let more attendees fill in just for the next couple minutes and then we'll get started all right I think we can get ready to kick it off welcome everyone to our webcast today my name is Olivia and I'm going to be our moderator and we're so excited that you've joined us for this session on how to prevent the use of AI tools during online exams uh there will be poll questions throughout this so please participate we really like to see your feedback as well as questions are open so if you have a question please drop it in the chat or the questions section and we'll either answer it then or get to it at the end all right so I'm going to kick it off with our first poll question and then I will be handing it over to Jordan so first our for our first poll question we're going to ask how familiar are you with recent AI tools and how they work so you should see this launch and we will give you all a few moments to answer that all right I see we have a few more answers rolling in give that another moment all right so we are going to go ahead and close that poll and share these results so it looks like the majority are somewhat familiar Jordan that's a pretty good sweet spot so that's should be well suited for today's presentation nice thank you all for participating and keep an eye out for the other poll questions right now hand it over to you Jordan all right thanks so much so uh before we kick it right off and get into things I just do a quick brief introduction um to my for myself I'm the VP of product here at Honorlock I have my background teaching Elementary and Middle School spent about five years doing that before getting into edtech in the product World. My entire product career has been spent in assessment and more specifically really geared towards secure assessment, so now that ChatGPT is mainstream you know we're all trying to figure out how do we adapt and what does it mean for the world of academic integrity I don't think we at Honorlock or anyone has all the answers yet but I hope today's discussion can really teach everyone a little bit about AI chat Bots like ChatGPT and then also open up a dialogue around the options that you have to protect your assessment process so to make sure that we're speaking the same language there's sort of two different types of AI tools that we're going to include in today's presentation things like ChatGPT which are essentially chat Bots and some other similar tools to ChatGPT would be like Google Bard Jasper Bing AI the other side of things that we'll discuss for a bit are extensions that are geared towards cheating. An example of this would be the transcript extension other ones are courseology quizard quiz whiz these extensions allow a student to essentially overlay buttons into their exam window and when they see a question click a button to get an answer and it's going to tell them hey there's a 78 chance that the answer is a out of these multiple choices so I just wanted to kind of define those two groups or categories of AI tools so that we're understanding what we're talking about as the presentation progresses First let's kind of let's take a look at how is it that ChatGPT was built at least the basics and how does it learn so that we can understand maybe how it might evolve in the future and and what its present State uh really represents so to be clear I'm certainly not on the inside of the open AI team or ChatGPT but researching and looking at a variety of interviews that were done by the OpenAI team the people that specifically worked on ChatGPTwe can get a pretty good picture of the key milestones in their training process and how that all ties together so first there's a step called pre-training pre-training is really where the development team and the team working on the project are going to try to load up as many resources as possible that can be used to train the language model so you're looking at you know millions of resources and what you end up with at the end of this pre-training step is really a large language model that will really spit out anything because what it's trying to do is predict the likelihood of what the next word should be in a sentence the simple way that I like to think about this imagine you're texting on your phone and you have you know predictive text that's going to kind of suggest what that next word can be that's essentially what you end up with in this pre-training phase obviously take that and multiply it by a million in terms of how many resources are being put in to the training whereas you know your texting is trained really more on your own behavior but that's what pre-training is all about getting a model that can predict what the likelihood is of the next word phase two of training a model like ChatGPT is around fine-tuning this is where you attempt to tune it by teaching it what type of responses a human user would actually prefer um and what ChatGPT their team referred to this as supervised fine-tuning where so what would happen is they would create a prompt then a team of experts on that prompt 10 20 people would create a variety of responses so let's say each expert produces 10 possible responses to that prompt then those prompts or then those responses excuse me are fed back into the model so now you're actually using human produced work to teach the language model here's what a human would expect when you're presented with this prompt phase three is really all about evaluation and feedback the GPT team calls this the reward model this is where ChatGPT will now respond to the prompts they'll produce let's say 10 responses and now humans are going to rank those generated responses based on their expectations and which ones they feel most align so if there were 10 responses produced by ChatGPT the human team would then take them and rank them one to ten feed that back into ChatGPT so they understand hey this was your best this was your worst response make it more or less likely to produce a similar response like that in the future and then step four which is the phase that we're in today with chat GPT 3.5 which is the one that most everyone's utilizing this is where you deploy the language model and you iterate and now the they're receiving feedback from everyone right millions of users are utilizing chat GPT and after you've had it generate a response for you you can give it a thumbs up a thumbs down let it know how well it did in responding to the The Prompt that you provided and then that again is used to feedback into the model and continue to train it so that it evolves and improves so now that we understand a little bit about the basics of how chat GPT was put together and what it's capable of let's discuss how it works when we're talking about AI and plagiarism detection tools and in order to kick that conversation off we're going to pose another poll question for the group so we're asking true or false ChatGPT plagiarizes content from the internet we'll give this one a few minutes to have responses roll through as well so we can get a feel for the audience's opinion and how we feel on this one all right so everyone should be seeing the poll take a few minutes to a few seconds give some thought and response so ChatGPT plagiarizes content from the Internet true or false also while we're waiting for some responses to come through I do see some questions coming in and please keep them coming and I will attempt to answer some live uh if I'm able to to catch it otherwise we'll be sure to Circle back at the end of the presentation to cover those the first question that we had come through while we're waiting for these responses was if we use an iPad app for our exams are these tools a threat uh the short answer to that is yes ChatGPT is certainly accessible on an iPad and also you know a student could utilize a second device to potentially access it as well all right so we've got 58 saying false 42 percent saying true so those of you that responded false congratulations you are correct ChatGPT does not plagiarize and we will explain a little bit more as to what it really does do so that we're all aligned like I mentioned before kind of using that text message example a language models function is to find the probability of the next word in a sequence that's essentially it like in a nutshell, if you have 10 sources let's say all talking about the scientific method each one might explain the scientific method some somewhat differently right there's going to be different words different explanations different sentence structure all of those things are going to differ based on the source the model is going to take the aggregate of those 10 inputs and they're going to try to create the likeliest sequence of words on that topic so what that means is there's not a single one of those 10 sources that's being plagiarized It's a combination of all 10 that it is creating a algorithm a formula to try to predict what word should come next now take 10 sources and multiply that by hundreds thousands and you start to understand the likelihood of one specific Source being plagiarized becomes essentially zero what this also means because ChatGPT is not plagiarizing from the internet it means that there's there's this concept called hallucinations which is when ChatGPT or any AI chat bot really just makes up non-existing or wrong information and it will present it as if it's factual and the reason is because it's not actually looking at a specific Source an example here would be I once just kind of toying around with it ask ChatGPT about the history of Honorlock as a company right and It produced a story about how Honorlock started who was in charge all of this stuff and some of it was very accurate but there were two or three just absolute false so it's mixed in there because it wasn't citing one specific true Source it was trying to utilize bits and pieces from a variety of sources one I don't know asterisk or caveat to this is when I when I'm talking about what ChatGPT is doing we're talking about today's version which is 3.5 there is another version that exists not yet
for public use gpt4 gpt4 does have some element of internet access which means that it is going to be better at uh citing actual sources and so far the early results as far as I've seen from the openai team is that gpt4 is 40 more likely to produce factual responses compared to GPT 3.5 so keep that in mind that this is a an ever-changing Ever evolving world I also saw a question come through you know is what I just described still plagiarism not just direct plagiarism I certainly am not arguing that that's academically honest what I described right so we want to try to discuss how we can stop students from utilizing that to whatever you would categorize that version of aggregating different sources and putting it into different language whether you would consider that plagiarism or not isn't too relevant in the regards to what we'll continue to discuss because what we want to do is we want to figure out how that no longer matters how can we prevent students from doing that or modify our Assessments in a way that um you don't mind if they use ChatGPT and we'll discuss both of those in just a bit so now that we've defined you know the difference between um or we've defined that the fact that AI chat Bots are not plagiarizing directly from the internet depending on how you want to Define that let's talk a little bit about the difference between plagiarism detection and AI detection tools because there are tools out there that do both of these things but in different ways plagiarism detection is really about an application looking through their database and seeing what percentage of a student's submission matches what's in the database and that database could be sources from the internet it could also be sources from past student work and it's going to look for it's going to compare you know against through infinite number that are in the database and we're going to try to find the percentage of of a match AI detection applications they are doing something a little bit different they're trying to predict if that content has been human generated or AI generated so it's looking at things like the structure the length of the response the predictability the complexity and it's trying to then assign a likelihood that that thing was produced by AI so certainly two different purposes and two different um features within an application that would be trying to do these two things and you know we're familiar with dozens of companies that are doing both plagiarism detection and AI detection or one or the other so when we consider how these tools are going to perform when we try to feed it an AI response or any type of response that's coming from a student or ChatGPT what are how's it going to work and what are the tools going to look at to produce the percentage likelihood that it's was created by Ai and it really all comes down to the prompts if your students or are using basic prompts then the tools are very accurate you know this would be you see the example of the basic prompt up on the screen describe macroeconomics in one paragraph if the students are getting a little bit more advanced and they understand that they should use a semi-detailed prompt which you can see on the screen again that's going to decrease the likelihood of the tool being able to identify it as AI generated content and then if a student uses a very detailed prompt the likelihood decreases even further so we'll show you a few examples of uh these levels of prompt detail and what type of results we we got in our testing so beginning with the basic prompt we tested over a variety of about a dozen different tools and we took the average right so certainly not going to name any names about scores or anything like that because that's not what this is about the Dozen tools that we tested on responded with an average of 85 percent likelihood that this response was AI generated and again this is using the basic prompt as we progress through to the semi-detailed prompt the score drops significantly now the average across those 12 tools was only at 30 percent likelihood that these were AI produced and then when we moved on to our detailed response our detailed prompt excuse me that likelihood of AI generated content reduced down to two percent in addition to this there's you know there's other little tricks to the trade of how you can get around some of the AI detection tools some examples uh we had one where uh after ChatGPT produced a response all we simply did is responded to ChatGPT and said hey the response that you just gave me was flagged as AI generated please generate a new response and after giving it that additional prompt the likelihood of being AI generated the score or the percentage went from 99 to zero uh that was on a smaller batch of testing across three different AI detection tools so just that little tweak letting ChatGPT know it was flagged as AI generated and asking for a new response was a easy way around the the detection the other the other thing is you know you can ask Chad gbt to write a response in the tone of a college junior and include one typo uh that one produced 99 or went from 99 likely down to sub 30 and again on three different detection tools tested so the rule the key takeaway here right is not so much the specific prompts that can be used to game the system but the the key is the prompt and how a student or how anyone utilizes that prompt plays a big role in how effective or ineffective any of the AI detection tools can be so I want to pause briefly and I'll also take a look at some of the questions that have come in at the same time as we're responding to the poll so at what AI or plagiarism score percentage would you feel comfortable taking action so we know that this is going to be a percentage match if we're talking plagiarism or it's going to be the application's prediction of this is the likelihood that this thing was produced by AI so we've got a few different options there for you to select from and I'm definitely curious to see how universities and schools are utilizing these results today because I know a lot of us are using these tools so I see some questions around like specific companies that uh do they pick up ChatGPT right so just generally let me back up just for a second so when we're talking about AI detection tools yes there are a lot of companies that will do exactly what I just described that will give you a percentage of likelihood of AI Generation Um and that's what they're going to provide you right and I think there's uh there's no proof as to what tool was used or what AI generation generator produce that content but it's trying to give you its best guess as to was that content produced by an AI chatbot somewhere um and so that's what those tools are attempting to do all right so looks like we've got a good number of responses through very interesting spread relatively even across the board so greater than 90 we have 14 of the respondents 76 to 90 at 22 51 to 75 26 to 50. and I think this just really goes to show you that there's no I don't know what the right or wrong answer is here um I don't think anyone does probably why the responses are so spread but uh as you kind of take into account what I just showed of how those prompts can change those percentages so drastically it makes it difficult to set a standard and say every time above you know 75 we're going to do X as an action so hopefully um you know we're talking a lot about kind of the the problems but I think now we can get into what are some of the solutions now that we've laid the foundation of what we're up against and how can we now evolve to live in this new world and maybe in sometimes leverage Ai and also block it if we need to foreign have you redesigned an exam or assignment due to concerns about Learners using AI tools because as we start to talk about our options here and what we can do that one of those options is always change the assessment um we may not know how to yet and that's partially what I'm going to talk about here in just a moment but I'm definitely interested to see is this something that you and the audience have already began or are planning to do in the near future and this could mean you know modifying your question types it could mean totally changing the questions you present on an assessment or it could be changing the style of assessment that you're even planning to give your students to try to be ChatGPT proof for a lack of a better term I'll take a few questions while we're letting the responses come through I did see a question about what are the AI detection tools that are being used um I don't I don't know if they're if this is reference to what we did in our study I'm certainly not going to call out anyone by name to try to uh play that game but it's common AI detection tools right are turnitin copy leaks uh those are some of the big names in the game and those are also tools that can detect plagiarism and it's very common for apps that were originally designed for plagiarism detection you know years ago have also added in some element of AI detection so when I talk about those tools in general hopefully that gives you a good clear idea of which we're talking about and again not necessarily the ones represented in our percentages earlier in the presentation all right so let's see what we have here for our responses so 35 percent of you have already redesigned your exam for assignments due to some of these concerns we've got a good chunk you know nearly 60 that have not yet done it but certainly plan to and then a very small fraction that are not planning to so what I hope is that what we'll cover here in the next batch of slides for those of you that are thinking about modifying your assessment maybe it'll spark some ideas give you some tips on what you can do and then for those of you who maybe aren't planning to do anything I do think there are some pretty simple kind of low-hanging fruit options that maybe you would consider introducing to your assessment process if you haven't already so first let's talk um exam tips I've sort of painted the the problem of ChatGPT but I do also want to kind of play the flip side for a bit and you know AI is more than likely a tool that students are going to want and or need to utilize in their professional future no one knows what that future is going to look like how ChatGPT is going to evolve but um you know at Honorlock we use AI as part of our product so I certainly don't want to paint AI as the boogeyman that must be stopped at all costs uh I think there are ways that we can incorporate it into our assessment process that can be a a real learning experience for students and also for educators as well so the first example and this is something that I've talked with a few professors that have done this already with pretty positive responses you can assign your students to debate against ChatGPT on a specific topic and assign a set number of rounds for that debate so as an example um you could provide the students with the initial prompt it could sound or it could be something like this I want to debate you on the question do public libraries have a role in the future uh and then the student can choose their side of the argument they would say I will argue in favor of libraries role in the future and you will argue against it and the you being ChatGPT and then the student would say you know here's my opening argument and they would go ahead and type their opening remarks on that debate topic after they finishChatGPT is going to then respond with its rebuttal they can go back and forth for any number of rounds you know maybe you want to assign three or five back and forth between the ChatGPT and the student and then at the end the student submits that transcript as their assignment or their assessment and you can see the students responses you can see chat gpts and hopefully get a good feel for how well the student kind of understood that topic from the professors I've talked with that have done this they mentioned that the students loved it they even just the fact that they were opening up and including ChatGPT as part of the assessment process they appreciated and they kind of appreciated the open-minded approach and the professors also mentioned that they found it to be fun um and interesting the second option here would be requiring your students to use ChatGPT to generate whatever answer they're going to get to but instead of grading the ChatGPT response you're actually going to focus more on the prompts themselves we talked earlier about how important the prompt is related to you know AI detection but really The Prompt is also very important to you as an educator to evaluate the students understanding of the topic if the student is just giving a very generic prompt there is a good chance that they're not going to get a great output from ChatGPT and there's also a chance or a pretty strong likelihood that the student might not understand the topic as thoroughly if the student is giving a detailed prompt and then not only doing that but waiting for ChatGPT's response and then tweaking it with a further prompt and another prompt after that they're using and they're working with the AI to get to an outcome and that I think is where the value will lie you know in their professional lives in the future is how can AI augment the human and make them better how can the human work with AI to get to an acceptable outcome so in that scenario you'll be focusing more on the prompts than you would be on the the AI generated response now those are both fun uh but also time consuming right because as a professor instructor you might not have time to grade lengthy transcripts coming out of ChatGPT so we also need to think about other ways that we can prevent the use of this and protect our assessments if you have a large class of you know 200 300 students this is probably not a scalable way to assess them uh tip number two here is about setting clear expectations I know that sounds obvious and it's something that you know we all consider when it comes time for a syllabus we try to set expectations for the course but in this new world that's kind of been flung into students laps you know unexpectedly I think it's important even if you don't have institution-wide guidelines yet around how to deal with AI tools set the expectations for your course what do you want students to use AI for what do you not want them to use AI for and I think um we've seen a statistic within our honor lock data that I kind of find heartening we have a feature which I'll talk about a little bit later where we block specific extensions that could be used for cheating purposes a lot of these are ChatGPT or other AI chat bot you know extensions and when we introduce the feature you know students who show up for an exam we would tell them hey we're going to shut down these extensions because they're not allowed and they would be blocked interestingly enough when the students return for their next exam about 60 of them are already coming back with that thing disabled now I don't know exactly what that means right sometimes maybe they never re-enabled it in the first place but I think what can be maybe pulled out of that is when you make clear to students what is and is not allowed leading into a test they're coming back ready hey let me shut this thing off I know I'm not supposed to use it and let me get ready for my exam uh I'd like for that 60 to be even higher but I find some uh comfort in that in that number and knowing that students once they understand what's expected of them the majority of them are going to try to deliver on those expectations so what are some other things that you can do you can cite specific resources and case studies so thinking back to earlier in the presentation ChatGPT responses and AI chat bots in general they're simply trying to predict the most likely next word in a sequence so if you can get specific with the sources that you require then ChatGPT is not going to be able to reference those specific resources it's going to take the thousands of resources that was fed into it to try to produce that response not the specific fact or specific detail that you've got stored away in your uh primary source they won't be able to get to that that could change again in the future with gpt4 so let's all be aware that this isn't a permanent solution but in today's world where GPT 3.5 is
what's being used by most students that this is a way that you can increase the likelihood of that hallucination that I referenced and decrease the likelihood of a good GPT response getting returned I would also strongly recommend setting time limits there's um our internal data shows us that the longer a student continues on in an assessment the likelihood of some type of academic dishonesty does increase so by shortening down the available time you're essentially giving students less time to poke around and figure out a way to try to cheat or work around the rules um less time to navigate to gbt and work back and forth with it less time to navigate to any resource for that matter and see if they can find answers so those shortened time limits they help prevent those opportunities to use any outside Source including ChatGPT and then finally what does Honorlock specifically offer that can help you protect against ChatGPT and others so number one uh we offer browser guard which locks the student into their exam window we've seen that about 13 this is aggregate across all exam takers about 13 of students try to navigate away from their exam window during a test and this is blocked or flagged because browser guard is in place so that's something that is just prevented um and now you have you know 13 of students that I'm not saying they're trying to all navigate to ChatGPT but in theory they could be so you have 13 of those that are kind of nipped in the bud with browser guard in place honor locks recording your desktop assuming you turn you you elect to turn that feature on during your exam that means you have evidence of what's going on during the test so it's no longer you know you don't need to be reliant on any uh percentage score that an application is producing or any data point for that matter you have video evidence of what went on on the students testing screen during the exam that you can reference and build a case if you need to like I mentioned on the couple slides ago we also block AI extensions this includes things like ChatGPT uh and transcript as we mentioned on one of the early slides and we found that about three percent of students show up to an exam with one of those extensions enabled and ready to utilize during the test so we shut those extensions off and we don't allow the student to re-enable it during the exam so again that's three percent of possible cheating incidents that just get shut down right right before they even enter the gate honor lock can prevent copy and paste if you choose to turn that feature on so if a student is trying to navigate out somewhere else and maybe you have I don't know some elements of browser guard turned off or you're allowing certain uh access to certain resources you can prevent a student from copying and pasting from ChatGPT or anywhere else for that matter again three percent of students attempt to either copy or paste in an exam and that is blocked once uh with this feature turned on in browser guard in companion speech detection which includes kind of key uh keywords so there are some voice activated AI applications that are leveraging ChatGPT where you don't have to necessarily type to it you can just speak to it with our version of keyword detection we're going to be listening for some of those key phrases that activate those AI tools and then we're going to flag that and get our Proctor interview involved quickly so that we can get the student back on track we also will check for the room if you have check the room if you have room scan enabled so that'll help us make sure there are not secondary devices in the area notes or other resources because even though you know we're locking down the browser the other key component to that is we have to make sure that secondary devices are also protected against because nothing could stop a student could very easily just pull up ChatGPT on their phone get a response or an answer and then transcribe that you know via typing into their exam window so in addition to these features here we've also really focused in on the secondary device detection element of what we do and we have by far the most expansive uh secondary device detection you know we have Search and Destroy which is a great preventative tool to make sure that your content has not leaked online um what will happen is when you enable on unlock on your exam we're going to search the internet for all of your questions and see which ones are out there we're going to batch up those those results and we're going to let you know which of your questions are unique meaning we can't find them anywhere on the internet which of them are compromised those are the questions that we found on between one and five websites and you will be able to send a dmca take down to those sites with one push of a button if it's content that you actually own and then we'll also let you know which of your questions are just prevalent all over the Internet we found them on more than five sites and really what this is all about it's it's about you being proactive and having the tools to modify that exam if too much content has leaked interestingly enough over the past three years if we were to compare pre-pandemic to post the quantity of leaked questions online has increased from 12 to 36 percent so in about you know 2020 12 of your exam questions may be online in today's world it's a stronger likelihood that it's 36 percent so Search and Destroy is a vital tool in trying to get ahead of that we have Multi-Device detection which is um a patented feature that only honor lock has which helps us prevent students from Googling and finding questions online and the great thing about Multi-Device detection is we really want it to be a deterrent whereas you know we catch one student using it and then students understand oh that student got got caught with Multi-Device detection we want them to just avoid Google right and say it's not worth the risk because most students again they're really trying to take their assessment honestly and if you introduce any element of risk it's not going to be worth it to most students and that's a Multi-Device detection can be a great tool in that realm we've also recently added Apple handoff detection which is again unique to honor lock where we're able to determine the our own custom AI trained model does a student have another device with a Chrome browser active with Safari active with the text messages a whole variety of applications on another Apple device and we're able to identify that and again get our Proctors involved as quickly as possible and try to get that second device removed from the area and in in conclusion here kind of to wrap that up uh just to hammer home the importance of the secondary device over 70 percent of the human verified violations that honor lock discovers are tied to mobile phone use so it is by far the number one method that a student might look to utilize for any type of academic dishonesty so having Tools in place to prevent that is is really vital all right so I hope that that gives you some options and some solutions I'm going to skim back through some of the questions here I see there's a pretty good amount um and see if I can answer some in the last bit of time so we have and Olivia if there's a way for the the questions to be shared definitely feel free to go ahead otherwise I'll just click through and read them out loud as I as I find them yeah yeah that's we did have a lot roll in so if your question isn't answered it's just because we are trying to make sure we get um all of them kind of grouped so I've just been seeing them over like that in batches okay so I'll read them out loud so what AI detector or detectors do you prefer so I don't have a preference and really what our stance is on the honor lock side and my own personal one as well is I think that you are much better off trying to prevent versus detect after the fact uh all the experimenting I've done has shown me that trying to detect after the fact is not accurate enough I think to meet um kind of the stringent requirements of needing to take a student to like academic court or whatever that follow-up process might look like at your Institution so I don't have any thing positive or negative to say about anyone in particular in the AI detecting realm but what we're focused on is really preventing students from getting access to ChatGPT because if we can do that it eliminates a lot of headache for you at the institution you don't have to deal with building a case around maybe muddy uh evidence or dealing with all that hassle uh because it's just been stopped because browser guard was in place because you know blocking copy and paste was in place so that's really what we're focused on um and then someone put this in I'll kind of paraphrase the question they didn't understand the percentages that were being shown on the ChatGPT slides in terms of AI detection tools I'm actually going to cycle back to the slides just so that we can all be on the same page so what these percentages show us it's in its simplest form How likely does the AI detection application think that this was produced by AI right so if if a student put in that prompt describe macro macroeconomics in one paragraph and then ChatGPT produced this response you would feed this response in to any AI detection tool and they're going to give you a percentage and they're going to say hey we are 85 sure that this thing was produced by AI and then as we see the progression of the prompts if you were to take this ChatGPT response and feed it into the same tool now that AI detection tool is going to say we are now 30 confident that this was AI produced content and then if we moved on to the detailed prompt that same tool would say okay we are now only two percent confident that this was AI produced so that's what that score represents it's essentially a confidence score of How likely that content was to be produced by AI hopefully that clarifies that and then someone in the question kind of mentioned you know the problem that they're worried about around detection is that there could be false positives and there's also going to be Evolution over time right and I agree 100 which is why um we're focused more on the prevention versus detection um we have a question what are the average percentages of known non-ai generated essays I I don't know the answer to that in terms of the AI detection tools so that would be interesting thing to follow up on and do some more research on but don't have that sat for you today yeah and then we had a someone who kind of did exactly what I had described where uh they referenced a very specific case in their textbook and they were saying that the student the student's response just had totally false information in it and then you know kind of asking perhaps this student used AI I certainly don't want to accuse anyone of anything but the fact that you referenced that specific case in your textbook that was a great idea and certainly what I'd recommend doing moving forward where you can protect yourself right because now the student if they did use AI they produced a poor response which in turn you probably greeted them with a poor score so you know you've essentially achieved the outcome in preventing someone from from Gaining your assessment so there was also a question does this apply mostly to essay or open-ended type exams does it apply to multiple choice I definitely Focus primarily on the AI like ChatGPT side of things with the tips and tricks and things of that nature but if we think back to the uh beginning where we talked about you know these different versions of these tools umChatGPT or things like transcript so this slide on the right hand side these These are pure answer generating applications so if you're doing if you're using a multiple choice assessment transcript is going to tell the student the likelihood that a is the right answer right courseology quizzard quiz Wiz similar type tools and again this overlays the transcript and other interfaces right on the top of your assessment and the student can just one click of a button now obviously if you're using honor lock that's not the case because we're going to block these these extensions and so the student won't have access but I will also mention that of the extensions that we block transcript is Far and Away the most common that we see so that one seems to have caught or be trending the most on the student side and that's like a subscription service where the students pay I think it's five dollars a month to have access to it to be able to get those answers and it does work with every major LMS as far as I know um so certainly some things to be mindful of even if you're delivering mostly multiple choice exams all right skimming through some other questions so does browser guard excuse me does browser guard um already block AI tools like ChatGPT so with browser guard turned on assuming that you don't have any whitelisted websites we're going to prevent students from navigating out to other tabs or to other things in their browser so they're not going to be able to go to open AI site and access ChatGPT now you might if you are electing to use one of those tips around how do I leverage AI in my assessment then you can still turn browser card on but you can whitelist or you can allow the URL for ChatGPT and let the students go there and do their work and then come back to their test and that way you know that's the only tool that they're able to access so there's complete flexibility within browser guard but by default assuming you're not allowing any access to any other sites yes the student won't be able to navigate away and get to Chat GPT on their device and then we have those other safeguards in place for secondary devices if they should be trying there all right we still have about half a dozen or so other questions can we detect these extensions was a question that comes through so uh we we do have all of the extensions tracked and part of your semester semester Business Review With Honorlock is you're going to be have some of this data shared with you around what are the most common extensions at your institution that are being blocked and how are students utilizing them we also are working on a admin dashboard that would make that data available to you know ad hoc at your fingertips whenever you'd like to pull it up so the intention on our end is absolutely to expose some of that extension blocking data back to you at the institutions so that you can have some insights into it and then we had a couple questions that seem to be confused around the extensions and and just I will reiterate we are and I'll give you even the background on how our process works around extension blocking so that we all understand so we've elected to block uh a large batch of extensions that are all potentially cheating related these are AI tools these are things like transcript and so honor lock shuts those down and make sure the student can't use them also what we're doing simultaneously is behind the scenes we are consistently analyzing our data around which new extensions are popping up right because sure we're going to shut down transcript we're going to shut down courseology and quizzard and all these but there's no there's no stopping the next one from coming up in the future so as students start to install those other extensions we track that we report on it and we can see oh there's a new extension that seems to be gaining steam we've had 100 students pop into an exam with you know some new extension enabled then we have a review process to analyze what the purpose of that extension is and if it is something that should not be allowed during a test we will then immediately add it to our list of blocked extensions and that goes into effect the second that we kind of switch that toggle over so just to be clear yes we block all of these key extensions that we talked about today and we are staying ahead of the curve so that we can continue to stay ahead of what's going to come down the pipe foreign by default and we had some questions around what types of settings do you need to use to make sure that all this stuff works so when you enable Honorlock there are institution defaults that can be set up for your for your school so I don't want to speak for everyone right because your institution might have certain defaults in place that others don't have but I will say that the majority of our customers have browser guard enabled by default meaning when you turn on Honorlock the key function and browser guards the most important part of all the things that I talked about you will automatically have all of these benefits that I've explained right where extensions will be blocked access to other sites so we blocked copy and paste will be blocked so definitely confirm with you know your key admin whoever is kind of in charge of your your proctoring settings that that's the case at your institution but if you see browser guard turned on when you enable Honorlock then you're good to go on the majority of what I've referenced today also just to reiterate their cut if you turn off copy and paste that will block it entirely meaning also within the assessment itself we have a question that kind of came through where I believe this person may want to allow students to copy and paste in certain scenarios within the test so in that world you would want to disable that you disable or well you would allow copy and paste to clarify and um when with that allowed you're still going to reap the benefits of the student not being able to navigate to other tabs you're still going to reap the benefits of you know shutting down the extensions so a lot of the key components that would prohibit the AI use are still in play but you can allow them to copy and paste within that test if it's something that they need to do to take that exam effectively I will answer maybe three or four more there's a lot of questions and I'm sorry I'm not going to be able to get to all of them but just trying to skim and pick out the one that might be most beneficial to the whole group yeah and I I did want to say Jordan if there's any specific questions anyone has about um you know using honor lock or actually how you're using it in your LMS I'm going to share our knowledge base Link in the chat and that has a lot of how-to articles and tutorials as well as our customer success team so feel free to use those resources as well if we don't get to your question so we have one here that's a good one so just this this question says just curious if a proctor gets involved in quotations what does that entail you know what happens and I did kind of talk about our Proctors without going into detail on this so what I mean by that is when um a proctor gets alerted of a potential academic violation right which is triggered through a variety of our AI detection tools uh then the Proctor will essentially open up an investigation window where the student is not yet interrupted so should there be a flag that was incorrectly marked or anything like that the student does not know they don't get bothered because we don't want the student to get any added stress when it's not necessary the Proctor will openness evaluation window and they'll observe and make sure that all right is that uh what do I need to do with this flag the student got flagged for a second person being in the room but I open up this window and I can see that that second person is their child right or their son came in and said that they needed dinner the Proctor is not going to bother the student in that scenario there's no cheating risk there so that that's Step One is the Proctor is going to weed out anything that does not need intervention and that doesn't require us to to interrupt the student but assuming there's something legitimate going on and um maybe the student was flagged for Multi-Device detection and now the Proctors popped in and they can see oh yeah they're using their phone the Proctor is going to try to de-escalate the situation and get the student back on track right so goal number one for our Proctors is to try to stop the action as soon as possible so in this case maybe they would ask the student to power down their phone or to you know move it into another room and they would verify this is being done on camera and then the student would return they maybe ask them to do a room scan to confirm that that phone's out of the area and then the student can continue their test I will say the the concept of de-escalation is very important to us so when a proctor Pops in just under 90 of the time we are able to de-escalate the situation without creating a violation and that I think is an important distinction between how honor lock approaches things and perhaps other proctoring companies is we want to stop the cheating before it gets started essentially so that you don't have to deal with it as an institution so if we can get there before the student uses that phone or we can get there before the student does X Y or Z and de-escalate and get that thing out of the testing area that's a huge benefit the student you know maybe they won't receive a failing score by their instructor you as an instructor don't have to deal with academic court and all the headaches that come with that and you know we've saved you time and hopefully created a positive or at least a more positive experience for that student because now they understand all right I'm back on track let me take my test do the best I can so that's really our focus is that the escalation piece all right all right I'll pick one more here so can there was there was a variety of questions around Multi-Device detection and we've got about three minutes so I think it's a good topic to close on so I think there's three oh there's more than three but I'll focus on three there's really the three main components that I talked about around Multi-Device detection the first one is completely preventative and puts the tool in the instructor's hand that's Search and Destroy because using a secondary device doesn't matter as much if the content isn't leaked online I'm not going to find it right if I go to use my other device to search for a question if I can't find the question I might not be able to get to an answer that's certainly not foolproof but that is it's better if that's the outcome then you know the alternative of them easily Googling and finding the answer so Search and Destroy is going to allow you to make sure your content is not out there online Multi-Device detection itself right is going to be when a student Googles a question and lands on certain sites honor lock is aware of these sites and we are able to flag those incidents and in those worlds right you're going to end up with a screen recording of the student you're going to end up with an audio Watermark that confirms the student landed on one of these sites and tried to access the answer and they're never going to see an answer on these sites either so they're only going to find the question so the the benefit to you is you know your questions are not out there with it with answers and you're going to have hard evidence that a student was Googling trying to find this response because you're going to have video you're going to have audio Watermark and you should have a pretty close cut uh or a close-ended button-up case then the other element to that is what we're doing uh recently with apple handoff and Our intention is to expand this to other device detection device sharing applications as well if a student is on a second device an apple handoff is enabled and they're logged in using the same Apple ID it has nothing to do with like their test window or Honorlock being installed on another computer all right so just to be clear on that if I'm taking my test on my MacBook and then I have my iPhone over here I'm logged in with the same Apple ID on both we're going to be able to detect if the student goes on and uses their browser on their phone or if they open up their text messages and they're trying to you know cheat with a student using text message and a whole other variety of applications and that's going to expand Beyond this apple handoff and go to other applications that would you know work with Android and other IOS as well so hopefully that clarifies those key three points and then there's a whole other slew of other things too right if a student is looking off screen they may be looking at a second device and Honorlock's going to flag that and we'll jump in um the room scan helps prevent a second device from being in the area keyword detection allows you know if a student says hey Google hey Siri we're going to detect that and flag it so really the overall honor lock package so much of it is designed to prevent deter detect the use of a second device so I hope that clears that up a bit and I apologize I couldn't get to all the questions um I definitely would encourage you like Olivia suggested follow up with your uh csms with questions and we will be more than happy to try to give you detailed responses but uh appreciate a good chunk of you hanging around to go through the QA awesome thank you so much Jordan and thank you everyone for joining we will be sending out those links as well as the recording and thank you for your participation in the polls have a great rest of your day bye everybody [Music]
2023-07-12 21:21
