we have NSS staff who share with us who will share with us about the funding mechanisms at the National Science Foundation and details about the specific solicitation for this Workshop we will also hear from higher ed and K-12 researchers about their Innovations and technology that help to advance our missions as educators I will include I will include the agenda in the chat for your convenience and a shareable link to this webinar I would like to say thank you in advance to all of our speakers as well as our closed captioning and I.T support and I look forward to a lively discussion we will get started with Jessica and Rob from NFL great thank you Amber okay so I think we are good you can see my screen right Amber okay great all right so um my name is Jessica cybrook and I am a program officer in the division of research and learning at NSF and I'm joined today by my colleague Rob oxendorf who is also a program officer in the division of research on learning so we are really excited today to have the opportunity to talk about this your colleague letter um it was just released in September and we're looking forward to this kind of being a kickoff to continued conversations between us and folks in the field as they consider submitting um proposals oops okay so um for today we're gonna go over some of the basics of the DCL it's connection to the building capacity in stem education research program also known as vixor and I will try not to use too many acronyms because NSF has a lot so I will do my best to be sure to explain them before I use them uh then we're going to talk about some things to note when you're preparing a proposal talk about the overall NSF Merit review criteria that will be used to evaluate the proposals and finally we're going to provide you with a set of links to some useful resources so and just as a reminder we are putting these slides out there so therefore the links will also be available to you so um about this particular dear colleague letter so first off this is a result of a collaboration between NSX and the Bill and Melinda Gates Foundation Schmidt Futures and the Walton family foundation and really it is designed to support and develop investigators capacity to leverage emerging Technologies to generate foundational knowledge for Equitable stem education so it's really important to note that what this D is it's calling out a particular Focus within a program or a funding mechanism that already exists and that particular program or funding mechanism is the mixer program and so um for those who are interested in responding to this DCL just keep in mind that you're actually going to be submitting to the bixer competition and then you're going to identify that you are responding to the particular DCL in your title so that said what that means is that it's really critical to read and to understand the mixer solicitation so a few notes about this particular program one is that it is a cross division program within the directorate for stem education and I'm excited to call it the directorate for stem education so some of you may be like what is that I've been working with NSF for a long time and I'm not sure so this is formerly known as education and human resources directorate and it was newly renamed and announced this week as the directorate for stem education so proposals to fixer go across the divisions within this particular directorate and they are due February 24th of the upcoming year so lots of time to really get on top of things now so the key to the Big Sur program is that these are capacity building projects and so there's two specific types of projects that are funded under bixer and one is meant for individuals who are proposing to build their own capacity in a particular area and the second are Institute projects where this is a group of experts in a particular area who are proposing to build capacity for groups of researchers so they're going to host and um different types of institutes to train groups of researchers at one time so let's start with a little bit more um individual investigator development track so these are intended for individuals from academic and non-academic organizations who are seeking to build stem education research skills and it's really important to recognize that there's kind of two types Within These individual investigator Awards so there's one specific type that is for investigators who are new to stem education research and then there's a separate strand for investigators who are experienced in stem education research and this can be a little bit tricky to kind of wrap your head around what we're talking about here so basically the idea is that Awards to investigators who are new to stem education research are intended to support folks who are new to the field and to provide them with experiences that are going to build their capacity to make meaningful contributions to the stem education knowledge base and also to support them in establishing their careers within the stem education research community Awards to experienced stem education researchers are intended to enable researchers to expand their areas of expertise and acquire additional Knowledge and Skills to conduct rigorous fundamental stem education research and these are individual Awards with a maximum of up to 350 000 for a total of three years the second type of awards are the Institute that I mentioned a minute ago and these projects we were seeking to fund experts in a particular area to train groups of researchers through Institutes so for example um past Awards have included summer institutes training cohorts of researchers in how to conduct meta-analyzes these projects have also a maximum of three years and up to a million dollars in funds so the focus of this particular DCL is to encourage folks to submit capacity building projects so both individual and Institute projects specifically around emerging Technologies so for example um an individual project to build capacity to use artificial intelligence or machine learning or learning Analytics to improve stem education research so the list that I provide here this is certainly not a comprehensive list but it's just meant to give you a sense of some of the areas that we're looking at when we think about emerging Technologies you'll note that there's a focus on math education research and research in K-12 settings however all stem areas are welcome as well as post-secondary and informal education settings so I'm just going to briefly talk about a few things for preparing these proposals and I want to do that because they look different than what you might kind of think of as your typical research proposal so for the individual investigator proposals everyone has to include a self-assessment of experience and expertise for conducting stem education research and identify the capabilities and the skills that they're seeking to develop through these Awards in addition there must be a professional development plan to build the investigators knowledge and support their career trajectory um there's a few things that do distinguish the new and the experienced tracks so while both groups need to identify broad areas of stem education research interests the applicants to the new stem track are proposing a pilot study um whereas the applicants to the experienced stem track are proposing kind of more of what we would conceive of as a typical research study for the new applicants they or those into the new to centrac they also need to describe their Mentor structure and the roles in rationale for how they're preparing to work with their mentors The Institute again these look very different than typical research proposals and so you need to describe kind of how this institute is going to meet the bixer program goals um the significance and the scope of the mission of The Institute the third one I'll highlight is the details of the curriculum this this is important we want to see you know what is the training plan what are the activities and what are the expected outcomes as well as evidence that there is a need in the field for this type of training and then a description of why this team is particularly qualified to leave lead a particular Institute so this likely looks very familiar to most people so I just want to briefly mention that all proposals individual and Institute proposals will be evaluated based on the two Merit review criteria intellectual Merit and broader impacts that govern all and accept proposals and this and again you can take a look at this more closely when you get the slides just kind of highlight how the Merit cry view right criteria can be broken down and so it's all often really helpful to look at this and say hey how does my proposed project kind of meet these different questions so the last thing that that I want to leave you with here is a set of resources and the first two the dear colleague letter and the Victory solicitation I would just say please please please read these really carefully I skimmed kind of higher level stuff here there are a lot of really important details within each of those documents the third link I have on here is something I personally find to be really helpful when you're coming up with your proposals and that's to look at prior Awards so this link will take you to a page where you can search for prior projects and kind of see what we've funded over the last several years and the last two links are just general guidelines for preparing proposals and submitting things finally I will leave you with our contact information so if you have questions please do not hesitate to follow up with Rob and I we're happy to answer questions and we're really looking forward to hearing the ideas and the plans from folks in the field thank you I will turn it back to Amber thank you so much I learned a lot myself you really broke down all of those letters uh for me or for all of us so thank you um now we're going to hear from Rob and Perry and they are going to discuss some of their um higher ed education innovation I'm sorry we're going to hear from Bradley and Perry I'm sorry I think we need to unmute for us no problems can you hear me now thank you all right um well thank you um my name is Perry Sampson I'm from the University of Michigan I am an atmospheric scientist so I'm happy to say on this sunny day um and this is a project we're going to talk about two projects I and Bradley we're going to talk about two projects uh that we've been involved in the first of which is of um the role of in-class inquiry in uh shaping student identity and the second one is a kind of a wholly different project uh based on building a technology which helps students with helps help seeking this was funded by NSF and we are part of the seismic collaboration of multiple universities uh working with the Sloan Foundation to improve stem education now when we're thinking about building a collaboration I have to say that my model probably isn't going to be helpful for everyone because I I literally was on the beach it's behind me here uh one summer afternoon and ran in ran into my neighbor at Avi Kaplan who you may know and talk to him about the things I was doing in technology and he pointed out that holy crow you're at the University of Michigan you should go talk to Stuart carabini and I did I went back and talked to Stuart and Stuart my gosh it was just a an inspiration a joy to be with a man who really understood help seeking for students at all levels and Stewart unfortunately we we grew up unfortunately wrote proposals together we out two of two funded that we're going to talk about today but unfortunately he then passed away literally two months after we got the Grant and one month before we got the second Grant um and I floundered for a long time because he was he was both an excellent friend as well as a inspiration but I had the good grace through Avi again to to find Bradley burgie and Bradley has joined me in collaboration to help me think through the help seeking issues that we were trying to address so I'm going to turn it over here too badly hi everyone this is Bradley um so anyone who knows Stewart Gary Bennett's work knows that he was a great advocate of of help sticking a scholar in that area and helped establish the notion that um helping students seek the help that they need is a really critical goal of Education but we know that students don't always seek help when they need it academically and there's lots of reasons that go into that psychological factors like how they see themselves and how they see the people around them um and also contextual factors like what resources are available to them and what technologies are available and so the two projects that Perry and I are going to talk with you about today are each anchored in some kind of Technology that's aimed to support help seeking um the first that I'll talk about is the back Channel project we know that students actually if you can stay on that slide in one more second uh the back Channel project focuses on the fact that students often don't ask questions in class especially in high enrollment courses and so we're exploring what happens when you add a Anonymous back channel to those environments in terms of student engagement and stem identity and then also we know that students often struggle to manage the various resources that they have available to them and make contextual connections between them and so Perry will talk to you about the learning fluids project that helps is about a technology that's helping students identify and integrate information to help study if you could go to the next slide very we framed the back Channel project uh in terms of kind of the ongoing uh challenge to attract and retain uh racial and gender diversity in stem we know that high enrollment entry-level courses uh where students first have their their first higher ed experience with stem are is a critical inflection point and we lose quite a bit of diversity at that point and we also know that in those classes um there's very little opportunity for students to ask questions and it can be a very intimidating environment to do that in and those barriers might be particularly high for traditionally underrepresented um students in stem uh what I'm going to do is share with you a little bit of the pilot data that led to the study and then tell you about the the central research questions and the design that we're using to research them and then tell you where we're at in the data collection but let me just present a little bit of the data that led to the conceptualization of the grant um Perry had been piloting uh uh a back channel in his class for several semesters and was asking students uh whether they felt comfortable asking questions verbally in class and you can see from the graph there on the right that the majority of students do not feel comfortable asking questions in those contexts and there's a gender difference uh where female students feel less uh less comfortable and the question that arises is what if we take away the barriers to asking those questions where you can ask them anonymously in real time get an answer right away what happens to student engagement and their perception of themselves as stem students so if you could go to the next slide Perry this is going to show a a screenshot of echo 360 which is one software that allows a digital back Channel by digital back Channel I'm referring to some technology facilitated real-time Communication channel where students can initiate communication and in this case um questions and get answers in real time um and so this is what it might look like during a live lecture uh that students are in person but students can ask questions in real time and get answers right away and those questions might be logistical uh like do we lose points if we're not here or conceptual in terms of trying to understand something that's being taught um if you can go to the next slide Perry as um this was being piloted in uh in his classes he was looking to see at the the the questions that were being uh asked in the back Channel and what I'm showing you here is a table that just that shows you uh the frequency of questions in the back Channel and the right hand column um disaggregated by female and male students top and bottom in the table and also disaggregated by how comfortable they reported being at uh they reported being asking verbal questions in class and just two take-homes here is one that we see that students who reported being very uncomfortable asking questions in class we're asking more questions on average in the back Channel than those who reported greater Comfort asking verbal questions um and the other the other Trend uh that comes out of this is that female students uh greater percentages of female students were asking questions than were male students um this along with uh one more piece of evidence that I'll show you if you could go to the next slide when students were asked how valuable is it to have an anonymous uh back channel available to you in these classes there was a strong trend for students seeing value here uh and so let me tell you a little bit about the current investigation if you go to the next slide Perry foreign so the research questions that are driving the current investigation is do students in classes with an anonymous back channel uh compared to similar students in a control group and I'll tell you a little bit more about the research design in a moment two students in that back Channel have higher rates of student questioning and other forms of classroom engagement so we're interested in the questions that are coming through the back Channel but also how the back channel is impacting the in-person questions in classes and also housing in affecting other forms of Engagement in the class we're also interested in whether uh having a back Channel results in more adaptive motivational perceptions um so does it lead to higher sense of belonging in the class do students feel more self-efficacious that they can be successful in a class do they see increased value for both engaging in the class and the content of the class itself and do students feel more supported in terms of their autonomy as a student and do does having a back Channel lower the costs associated with participation and asking questions um that kind of wide range of motivational constructs have been associated with achievement and persistence in stem and so we're interested in how a back Channel might influence those motivational perceptions ultimately we're hoping that uh we're trying to understand whether the back Channel leads to uh students forming a greater stronger sense of identity as a stem student and a stronger commitment to persisting in a stem major a final set of questions that we're asking is is how uh back channels affect engagement and perceptions uh and persistence for different groups with an attention to traditionally underrepresented groups like women students of color and students from low socioeconomic backgrounds uh if you could uh advance to the next slide Ferry uh we're using a quasi-experimental research design uh where we keep the instructor the same and the course the same but we manipulate whether or not there's a back channel in the in the course so in the control semester the participating instructor would teach the classes they typically do they would help facilitate a beginning of term survey where we're assessing the motivational uh constructs that I just talked about we record the all the class meetings so that we can analyze them for the verbal questions that are coming out and also the instructional uh format that's being used and the strategies that are being used and then at the end of the the term there's a uh another survey where we're looking at change in perceptions um so that's all within the control semester in the intervention semester then we have the same teacher teach the same course but this time there is an anonymous back Channel and we have a teaching assistant who's there Fielding um the questions in real time and otherwise we're collecting the same kinds of data in that semester you can see across the bottom of the the slide different technologies that we're using to provide a back Channel we're working with instructors to find a technology that works for uh their needs from our perspective what what those Technologies need to have is that they can students can ask questions anonymously in real time uh they're Anonymous to their classmates but they're not Anonymous to the instructor and the TA and also the researchers and that on the back side we can have access to the data that we need to conduct the analyzes uh if you could advance to the last slide so right now we're in uh data collection uh we're collecting data across three different institutions um several courses at Michigan State University and also courses at University of Michigan and State University of New York Geneseo um and all of almost all of these uh courses are in the control semester and so we're about a year out before we have data on uh what What will What what will ultimately be able to answer our questions in terms of the effect of of a back channel in these environments uh but look forward to to sharing that information and answering those questions and if there's an opportunity to share back to this community we'd be happy to do that um at this time I'm going to uh turn it over to to Perry who will tell us um about the learning Clues project thank you Bradley and this this session here is about um building capacity and I've got to stress to you that despite what in addition to what we're showing you key here is finding a partner or partners with whom you can interact and and I I word it this way find somebody with a sense of humor and and and the patience to work because you know badly speaks a language that I don't understand I'm a climate scientist I gotta believe that I just don't speak anything uh clearly anyway so finding people with whom you can have the interaction is key and part of that building capacity is not only the skill set but if a lot has to do with simply the personality of finding someone who has the creativity and interest to expand what they're doing and this project is a wholly different type of project this project was funded by a group called The Partnerships for Innovation at NSF it's a grant where they discouraged us from actually publishing which is unusual for NSF that what they wanted us to do is build out a technology we had started at Michigan and so that other schools could could use it in the future and so it's myself and Bradley I'm bringing into this project as well and Kevin Collins Thompson who teaches as a head of Masters of Applied data science at Michigan in this project we focused again on help seeking but this time it's about we have a undergraduates we have adult Learners and and they're often challenged they're arriving at school and they don't necessarily know how to study and what are the how do we go about this what should I look at next and more importantly uh you probably have not had the same experience where you you put all sorts of excellent resources on into your LMS you supply PDFs to your class Etc and the students don't necessarily make the connection between what you're doing in class and these other resources so that led us to think about how could we basically help students find materials they need to study more efficiently uh which is going to help with their retirements and energies many students who have jobs they have maybe are parents and so how do we help them get the material they need more uh quickly an instructor is helping instructors know which of all these resources you make available which of them are actually useful to the students or most useful and institutions who have gobs and gobs of video laying around from classes how can they explore the videos they have available in the department in order to create maybe new courses based on a new need coming from some other other source so we've created two different Technologies as part of this the first of which is a search engine and what we do is we work with your institution and we then are able to access the videos from class with the instructors approval the videos from class and we mine those videos from class looking for what's being said in class and we even take the you know what was visually presented in class and extract The Words which were presented as we just build this metadata of what's being taught in each class this is live now in over 400 courses at the University of Michigan so with the with the search engine we mine the videos and we identify the key words we listen to what's going on we make some assumptions then about what's the discipline of the course we then make further assumptions about what are the important words in that discipline basically reverse engineering the learning objectives of the course store all that information in a database with links back to the you know the place you keep your video on campus so that students can we can then build things through an API and among the things we've built one of the things is a search engine here I'll show in my course for example in a search engine you put in a easterly if I asked for it it'll show you which lectures I mentioned that in and the specific moments in that lecture and takes you as quickly to that moment in the lecture uh that you might want to explore that topic so allows the students much more quickly to get access to information the second part of this is a building study guide now I'm a lazy man and the students forever asking me for a study guide and I don't do it so what I've done is we've made a system here where the students can come in and they can um click the study guide and then select a date range where I got an exam coming up I'm going to pick some period of days our system has listened to and extracted what were the important words talked about or during that period and then list the words for you can decide which ones you want to study and by clicking next then it automatically generates flash cards based on what was talked about in class and again we think this is going to be helpful to students who may not have the best past experience and what should they study in an exam what what things are important that we can help design these things and in addition to giving you things like like the definitions we also mine if we again with the instructor's approval uh any documents you have in your LMS or links you have in your LMS and store that metadata also so we can point to the students if the issue is thunderstorms I'm like oh there here's some articles in your in canvas might be your LMS here are articles which might be in your textbook working as a vital source and or or and or other other content so the whole idea here is to help the students by providing basically pointers to the other resources that you should know about in your learning ecosystem based on what was talked about in class and do that automatically and let me so the research questions that we're working with are here does the ability to search for a specific moment in class videos result in deeper learning or is it does at least result in a more efficient use of your available time um to to to achieve a certain level of learning um will students using the study guide experience a higher level of learning outcomes are they going to learn more are they is it going to be more efficient use of their time for the same learning outcomes to what degree do will instructors find Value in learning which resources the students find most useful for study and to what degree will institutions and instructional designers find Value in being able to take all the video we have available on campuses and use it for other educational purposes now part of the system that we've built in Orange here are the we we are sniffing what's being taught in the class by watching the videos again always with the instructors uh approval and also one of the resources are available in that course in the LMS or whatever the instructor says and storing this metadata which points back to the original Source at the same time we're building working on another system over here which will be able to access then well what was the characteristics of the learner from the student information system and what were their learning outcomes and all this would be available to us we go through a layer here of de-identification and then make expose an API so that now you could do research so this is exactly about building data sets that combine you know background combine and outcomes learning outcomes and what students are doing during class it provides information about which resources they went to look at which resources they found useful which helps in machine learning for this process to become more efficient so this is the kind of thing that I think would be how to move forward in this building capacity is work with the technologists who can deliver more types of data to you but we can't know enough about how that information should be used in education research it's going to take a community and I am confident that this call for uh proposals can can help get us in that direction I'm going to pass it back to you Amber yes thank you so much um I always enjoy learning um learning about what you do and I I really appreciate hearing about the help seeking and the back Channel because I was that student so um I'd love to see that there's research in that area and we're actually providing that that service to students at least um where they can access your technology all right we're going to hear from Kay can you help me pronounce your last name please yes so today we'll be hearing from Kay Richard puma and she will be talking about evidence-based research in case law environments thank you Amber uh as Amber introduced me I'm Kayla jakuma from Texas A M University uh and I've actually held many different played many different roles and educational research I actually started off as a computer scientist Building large-scale Solutions for corporations and got into cognitive science to figure out what moves of people to learn something and then utilize it in whatever activity they are engaging in and I I actually built a intelligent touring system for reading comprehension system and then uh took it to a school and then got elected to the local school board after that experience so I've seen schools from many different angles and many different roles and uh the more I work with schools the more engaged and passionate I become in the needs that are out there yet to be solved so I'm going to share a few things today with you based on the work that we've done our work has evolved and been refined through multiple projects that were funded by the U.S Department of Education and The Institute of Education Sciences and I want to call attention to the numbers of schools and large numbers of students and teachers that we have worked with throughout the implementation and design and development of our systems early on in the 2000s I developed this web-based intelligence tutoring system for reading comprehension through a development Grant and then we took it to an advocacy study with over 128 classrooms in approximately 32 schools in elementary and middle schools in Rural and Suburban settings and since then we've actually now taken it to high poverty settings as well uh all over the United States of America we've also always been open to requests from the schools that we work with and one of the most common requests was what do you have English language Learners and so we extended our web-based intelligent tutoring system to the strategy instruction on the web for English Learners that has a Spanish component to teach children how to read and comprehend in both languages another request that came by was what what do you have to help us teach writing and so again we've partnered with different schools and different universities to bring uh we write it's a self-regulated strategies development model of writing to the schools and most recently we've conducted a large-scale efficacy study or we write in 32 schools we've also extended our work into science comprehension and we've also repurposed our web-based intelligent tutoring system as a professional development mechanism for teachers and received a supporting effective educator development grant called speed and so we've trained over 4 000 teachers using this mechanism to implement our evidence-based strategies broadly across the world uh just uh in this January we received an expansion Grant to take our itss and the Kate project to scale with 60 schools across the country so um what I'm sharing with you today is the compilation of techniques we have used to successfully develop and deploy these web-based tutoring systems and feature-led interventions to schools I do want to call attention to the fact that we have published these things very broadly and widely and most importantly we have received the what works Clearinghouse evidence standards we've met the evidence standards without reservations which is the highest possible rating given by the Clearinghouse our research studies have established impact estimates about its as well that are statistically significant and positive favoring the classrooms and schools using them and the WWE CW also shows that our research was rigorous and met standards for randomization attrition and the use of valid and reliable proximal and distal measures for student outcomes so what all this means is that what I'm sharing with you has worked and has actually made positive impacts in schools across the country and I hope that you will benefit from some of the things I'm going to share so how do we bring these evidence-based practices to life in schools first we focus on problems of practice things that actually matter to schools and even if you have carved out a specific area of interest that you're studying it is important to put that in context of what's going on in the schools because you can't expect them to stop doing something and intervene with some new idea that you're bringing to them if that does not directly impact some broader goal that they are maintaining the next part that I want to cover is that you have to embrace complexity because schools are extremely complex systems and as I've said I've seen unit from a parent from a school board member from a teacher from a Community member and schools have decision making that is hierarchical non-hierarchical there's lots and lots of moving Parts different people making decisions sometimes contradicting each other sometimes in alignment with what's required also sometimes imposed by external entities sometimes brought through from internal entities another area that we focus on is investigating the wh questions which is why did our intervention show this impact on what under what conditions does it show this impact and to whom does this benefit because it's really important not just to present some number that says this is our impact factor and this is ifx ice and it was statistically significant but there will be groups of students who do not benefit from these interventions what do we do about those children because we cannot assume that just because we got some result that is actually impacted all of the students in some positive way I want to call attention to the fact that I was talking at a conference and a superintendent came up to me and said well I'm very moved by your passion about improving schools I'd like you to come into my schools just for our sub populations because we've met the state standards we get a 76 pass rate for our reading you know elementary schools we've met our state standards so I don't need you to come and help us with that I need you to come and help with the specific subpopulations that are failing at a much higher rate and I said to the superintendent so if you say even 80 pass rate that means 20 out of 100 children in your elementary schools have failed to master fourth grade reading what are those children going to do and his response was what do you mean and I said okay we cannot be hiding behind these numbers and statistics there are real children and there are real consequences if we do not make sure that every child is taught how to read and write effectively before they can move on so everything that we do should be focused on making sure that we at least know why it doesn't work for certain groups of children and possibly propose solutions that work for everyone I must say that in our most recent study with the high poverty schools we have achieved 100 pass rate with children in high poverty settings highly bilingual settings in Border towns across the United States where we are getting 100 passage for special education students and regular education students or the constructs that we're teaching so how do we work with schools we start with cultivating a deep understanding of the context and establishing trust and I think they all go hand in hand they're not individual separate constructs because if you understand what who you're working with they learn to trust you more and you have to focus not just on the student but also the practitioners the people that are actually bringing your intervention to their classrooms and implementing it with you or for you and when I say A practitioner I'm including the coaches the support staff and the administrators because these are the people that make the decisions on when the intervention will be included at what time how much time will be devoted and what happens when you have other competing priorities is this going to be a priority for them and most of all be humble one thing I have learned in my school-based research if you spend the time understanding the context they know so much more than we all do and we have to tap into their expertise on what they know about their students their schools their curriculum their standards everything that they know needs to come into play in our design and it's very easy to be humble once you realize that you are dependent on them to keep you informed about about what this would do in their content some of the specific techniques that we have used we've taken state level data National Data we have mined data to look at the patterns and performance of different groups of students in fact we actually before we go to meet with some of the schools we look at the particular performance of those schools over multiple years not as a gotcha not as finding out what they did wrong but just to keep ourselves informed about what our intervention and how our intervention fits into making life better for them what we are bringing to the table that helps them and meeting with the leadership is a very very powerful technique and I've always maintained that the pi the person in charge must make these efforts this is not something for your postdoc to do or one of your doctoral students to do I make the effort to go meet with these people because they are very important too us and the work that we do so here's an example of what we found when we mined the state level data we found specific constructs that were not being taught and that were not being Mastered by students that were actually affecting their overall comprehension in science and social studies and in every subject area so when we share this type of data with them and share their own data with them it's a very powerful and convincing argument for people to trust you with your intervention coming into their school we review their textbooks and they are curricula to see what their scope and sequences things that will help them to understand how your intervention fits into the greatest scheme of what they're trying to do we look at their high stakes assessments their benchmarks their progress monitoring systems and we lay in context our intervention so if you're teaching a specific concept how does that fit into the Benchmark is that going to be measured in the next Benchmark you can't convince the teacher to prioritize your intervention if that is not going to be something that's part of their benchmarks coming out and you have to understand who's delivering the intervention and when I'm talking about who I'm talking about the administrators the teachers the coaches the special education professionals and we want to build towards sustainability that involves all the factors that I've shared so far including the measuring of the Fidelity and I want to draw attention to one little story we recently in our seed project measured teacher content knowledge and pedagogical knowledge it was found that the teachers caught that knowledge was much much more influential in the student outcomes because they were able to customize responses to the student when they understood it deeply as pedagogical knowledge could be mostly very surface level implementation of procedures whereas contact knowledge provides deep knowledge that the teacher can engage with the students with so overall when you're talking about Fidelity it's not just a matter of staying true to what is happening in your intervention but also how that intervention is being implemented in the larger context of the school in our instructional systems now we have developed a 360 degree support view for school s which includes teacher and administrator professional development on how to implement our interventions in their classrooms using online tools as well as face-to-face tools we also promote and support with classroom coaching modeling and additional supports for professional learning communities we allocate Personnel to meet weekly or bi-weekly with school personnel to sit in on their planning times and provide support and feedback on how our intervention fits with their greater goals for Learning and we've built sustainability through monthly check-in and online online support for office hours for the teachers and the implementers we also review with the school Personnel their benchmarks and analyze help them analyze their quarterly improvements while our intervention has been played so that they actually learn to trust us and see the Improvement as it's happening without waiting for the final data analysis to arrive and be published which you know will take many years so overall this is our most complex system that we have developed and put in place for our most recent key project to acknowledge the complex systems that schools are and I want to encourage everyone to take into consideration that schools really really need your help but help in a way that is that is sustainable for them and make a change for the children and the teachers for the greater good thank you thank you so much okay I would like to just take a a moment of pause to thank all of our speakers um I really appreciate appreciate hearing about all of your Innovations and I hope that my other participants has written down your questions or have put them in the Q a so that we can answer them for you um so I do have a question to kick us off um someone chatted directly and they asked about um how your Innovations and this is for both groups uh for for K as well as uh Bradley and Perry how do your um Innovations explicitly um address or uh are related to some of the emerging Technologies in the dear colleague letter so I did hear a few of you uh um a few of you mentioned Data Mining and maybe some learning analytics but if you could speak more directly and explicitly I guess to the um the type of emerging technology that you all use in your project foreign as you point out yeah the data mining is an enormous field artificial intelligence requires expertise that is just just emerging to be honest and and getting deeper as we get into it uh that's going to take a a group of people who are up to date on on those Technologies and they probably don't have any experience necessarily in education research on the other hand education researchers are up to date on the Frameworks and and the expectations as we just heard about in the classroom depending on which classrooms you're dealing with and far more advanced on on what are what is known in the literature that's not something I'm going to have access to or read Because I my day is full dealing with enough climate issues so finding finding that colleague finding groups together who can work together and take ideas and and mix them you know all too often we get technologists uh just just we can build stuff and leave us to it we will build stuff and that stuff may or may not be useful to education I might think it's useful at education but until I get somebody like Bradley to help guy put guard rails on that road it's it's um is wild wild entertainment and not necessarily progress and from our perspective we have engaged in all the aspects that are out in that call as was noted we have done the meta-analyzes we have done the data mining we have actually uh developed and tested the web-based intelligent tutoring systems that are continuously refined through the work we have extended it to include science computational thinking quite a few other additional constructs being taught using our intelligent tutoring systems so all of these things can be done as Terry mentioned it does require a partnership it takes a village uh quite a few people who play different roles and you have to bring in people with the right expertise but also the right mindset because it's not just technology for the sake of Technology we have plenty of that going around we need real solutions these schools and the children and our society is hurting quite badly until we step up and solve some of our pressing problems and for that we have to put our best thoughts best groups together and our thought processes to bring the best solutions that are likely to work and prove or disprove that they work or do not work yeah that is excellent I'm an education researcher a PhD in education and I was a K-12 educator before and I know exactly where you're coming from as a teacher when you're coming in with new technologies in my classroom it has to be to the end benefit of my students so it's that collaboration that I think you and Perry are both speaking to uh technologists they can build and Engineers they will build um but we needed to to what in uh you're building this to what in uh nobody likes busy work not even our students Jessica and Rob you all want to speak a little bit more into the car yeah I just wanted to add a little bit to that Amber so um one of the things I was thinking about too when you're talking about these collaborations is for example like let's consider an Institute proposal so if a group of experts in um learning analytics for example come in and they don't have anyone on their team who does stem education research that's challenging right because we want these teams to prepare and build capacity for folks to use learning analytics to improve stem education research so I really see those collaborations from an Institute perspective in terms of like expertise in the emerging Technologies as well as you know some folks from stem education research um and I could imagine a similar thought with the individual investigator Awards so you know a key piece to that is having mentors and so you know you may want to be learning about how to use machine learning and so you're going to seek out mentors who are outside of maybe your regular area because that's a new skill set for you and so again those collaborations are going to be key in putting together strong proposals and plans I'm going to jump in here and argue that maybe NSF should build I hate to say this the equivalent sort of a dating app for this because you know my model was to meet somebody on the beach out here I don't think that's scalable um and more more to the issue here is we have to find ways to talk to people who aren't I wouldn't normally walk into the school of education how am I supposed to meet these people but for grace of God and good luck I I did but for many times and one time earlier in my life uh NSF actually said well you're working on technology here's somebody's working in education and slapped us together that model also works but there needs to be some thought here about how do you find those people and how do we make that work that's that's and by the way I just a plug right now both Bradley and I should have said that both of our projects we are looking for volunteers outside our project who who might want to participate in these projects because we do have opportunities there I think on the on the dating front I think that's Amber's realm and uh at least part of this effort across these two workshops is to try to make some connections for people and I think that's a lot of people are introducing themselves in the chat and I know Amber's collecting data if you fill out the survey I think that's part of it yeah Amber yeah exactly thank you so much and that we went right into that uh very well we we are hoping on session two to create these collaborations um I have to speak frankly you guys heard me mention before you know we don't do busy work here we are here to help you prepare to be successful in your submission of a proposal to this call that's what we want and we are bringing together uh groups of data scientists and education researchers and people who are passionate in the field have experience in this area to come together to create very strong um collaborative research projects so the forms that I put in the chat is for you all to fill out it's just a general interest form I don't believe anything on there is required but if it is it's probably just are you interested in being contacted for a collaboration if you have not already um registered for session two um that one is more of a Hands-On Interactive Workshop where we'll be working with collaborative groups you guys will have a chance a chance to go around and uh visit various I would say like rooms of um of data collection groups that have data that they would like to share with you all some collaborators that are looking for different Partnerships um so our goal is not a poll I'm sorry um it's a Google form but our goal is to help you all um get what Rob and Jessica need and if you fill out this form as you mentioned um we can we'll definitely do that do we have any more direct questions for our team or for me just to weigh in briefly Amber I mean I think part of the vision here it's that a lot of these emerging Technologies are increasingly present in our lives think about your interactions with technology on a daily basis the information the data that's coming through the that pocket computer is driven by all sorts of algorithms and machine learning and Ai and you know for good or for bad this is this is what is driving a lot of the data that comes to us right and so I think part of the vision here is that um you know that there could be some positive here and a lot of these emerging Technologies to help improve either how we do research in education um how we build interventions um our students are increasingly interacting with technology um is it being done in ways that are sort of Cutting Edge um can we leverage these Technologies to certainly improve education but also improve how we do research in education right so there's lots of different angles here I think you know NSF intends to be at the Forefront on a lot of these issues and so we are trying to push the envelope around what is possible with respect to education research now you've heard some examples here today um and so you know these are um just some examples of how these emerging Technologies get instantiated in different kinds of learning Technologies um and so our vision or our hope is that we can continue to build capacity in this space so that um The Learning technologies that are present and being used are smarter uh more capable more efficient Etc um so I don't know if that's helpful at all or if Jessica if you want to add anything at this point but um it's a new initiative I think we are trying to um you know we're all a lot of us are education researchers and we also don't understand a lot of these emerging Technologies and so that's why we're trying to bring teams together to figure out what is possible yeah exactly Rob said it perfectly it was very well said um and again I hope you all get a lot out of this um and the recording and the slides will be shared with you all uh to the email that you use for registration um usually within the week of the presentations and I also share the link for the registration you will get another link for registration for season um for session two it'll come to your your email um as a confirmation that you attended today so if you didn't get it out the chat no worries um I sent it to you in a follow-up and it'll also be with the slides and everything the next session is November 9th from 12 to 1 30 Eastern Standard Time working on that one is November 9th and move it to a Wednesday a little bit earlier for our West Coast uh folks going to be well for 1 30 Eastern Standard Time thank you all so much please reach out if you have any questions oh go ahead let me just make one quick comment Amber because I'm seeing a few last minute things come in the answers to a lot of the questions that folks are posting regarding the Big Sur competition and the dear colleague letter are going to be provided in the links to those particular announcements so definitely please do read those and if you continue to have questions follow up with us but there is a lot of information in both of those documents awesome awesome thank you all so much enjoy the rest of your day thanks for showing up thanks guys thank you
2022-11-09