001Hello everyone. Thank you for joining us. I'm going to stop sharing. Quick moment just so I can pull you all up again and then um I'll get us started. All right. Hello again. Thank you for joining us. This is our 2627 HB146 ratio webinar for vendors. Thank you for joining us on this Halloween. I will start by introducing myself. I am Alicia Garcia. I am the Texas uh tutoring and content specialist. I am joined by our tutoring team's director um Dr. Kobe Self. Kobe, would you like to introduce yourself? >> Hello. Welcome. Thank you for joining us today. >> Yes. And we have a very special guest joining us today. Um we have one of our expert reviewers that's joining us. And I'll let Heather introduce herself. >> Yep. Hi everyone. I'm Dr. Heather Greenhal
002Spencer and I lead the team of evaluators. So happy to be here, happy to answer specific questions about the evaluation processes should they come up, but also know that Alicia and Colby do an excellent job of like giving you all the information beforehand. Good to see you guys. >> Thank you. All right. So, today we're hoping that we can provide you with some guidance to help you with your um application submission. We're going to provide you with a brief overview of H31416. We're going to also provide you with um a brief overview of vendor requirements, uh the application guidance that we've also provided to you, the product evaluation criteria. We'll go over the timeline, and then we'll give you um a date for the scheduled office hours. If you have any questions, um, you can
003drop them in the chat. I don't think you could see the Q&A, but if you drop them in the chat, we'll try to get them. We'll try to get to them in a timely manner. But if we do not get to your question, also, we're going to ask that you could um, email it to us just to ensure that we give you the appropriate guidance that you're needing. A copy of this uh webinar and a copy of the slide deck, a recording of the webinar and a copy of the slide deck will be posted on our HP1416 racial waveless product web page um probably early next week. So let's start by giving you the HB1416 racial walist overview. Texas law requires that students who do not receive approaches or higher on a star or starc
004administration receive accelerated instruction through either a TIA designated teacher or through completing a set of hours of supplemental instruction. If a student falls in the high does not meet category, the student is required to receive 15 hours of supplemental instruction. If the student falls in the low does not meet category or has a consecutive school or failure in the same subject area or if it's a grade three failure in math or reading, the student is required to receive 30 hours of supplemental instruction. HB1416 requires that the agency approve one or more products that use an automated computerized or other augmented method for providing instruction in lie of some individual or group instruction that um students are provided. The agency can approve products that provide evidence that indicates that the product is more effective than the
005individual or group instruction that school systems provide students. It is important to note that school systems may choose to use any curricular tool that aligns with HB1416 requirements to continue to support the accelerated instruction they provide students. However, only products that are specifically mentioned on the HB1 1416 ratio waiver list qualify for the waiver of the 4:1 studenttoteer ratio or student totutor ratio. So, HB1416 ratio waverless products consist of products that use automated computerized or other augmented methods for providing accelerated instruction. These products do not require a live in-person or online educator to deliver this instruction. products that are approved on the HP1416 ratio waiver list qualify for the waiver of the 4:1 uh student to teacher or tutor requirement. If you're uh wanting more information or if you're wanting an application, if you haven't
006received one already, we have um all that available on our HU146 racial waver list page. And we have announced again that the application window opened on October 13th and it will close on December 12th. There are a few requirements that vendors who are approved or who are interested in applying must meet. So on our web page we have these requirements listed and these resources listed. Um so one of the uh requirements I'm sorry I'm going to go over some of their requirements. So first of all, [clears throat] a vendor who is approved for the racial wavelide outcomes data through an annual star analysis. Additionally, racial [clears throat] walist vendors must meet SBOE suitability requirements and furer requirements. These products must undergo EMRA review if applicable. Supplemental math materials should have EMRA approval in order to
007apply while supplemental reading materials will have to undergo undergo through the EMR process during the EMRA 26 2026 cycle which begins in May. But I believe that application window opens in January. Um these require the EMR requirements must be met to remain on the racial waiver list. Now I'd like to emphasize that our tutoring team is not in charge of EMRA. That is a separate process that TEA has um works with the SBOE to uh provide but again we are not in charge of EMRA. We do um collaborate with them but that's not a process that we are in charge of. And so if you have any questions regarding EMRA we strongly encourage you to go to the web page um they have information regarding the timeline regarding the SBO meet SBOE meetings um application
008window and then the different cycles that Emmer's going to go through. But again um we have a link on our web page and also here um H waverless materials are required to meet um suitability requirements and they must also undergo through the EMER process if applicable. Uh again, math products should have undergone the process this past cycle and um on November 21st, we'll know which products have or have not been approved. And those that are providing reading products, you will have to go through the EMR through the upcoming EMR cycle, which um starts well the reviews in May 2026, but I believe the application process starts in January. failure to comply with the stated requirements in these terms, conditions, and affirmations result in automatic removal from the TEA HP1416 racial wave list. So again, it's
009very important that as you're going through these under requirements, you pay close attention to everything that is listed there. And if you have any questions regarding any sections, please reach out to us and we will provide you with any additional information you may need. We've also created for you some guidance and um to help you complete the application. So on the link for our application resources, we provide you with an application preview of all the questions that you're going to have to respond to. This gives you an opportunity to work with your teams to provide us with the best responses that you can. Um, the application guide also gives you guidance on what can or cannot be submitted. For example, you know, you're completing the application in Smart Sheet. Smart Sheet does not accept embedded
010hyperlinks. You must submit full URLs. Also, Smart Sheet has a 4,000 character limit, including spaces and punctuation, so you cannot exceed this limit. Vendors are able to apply for multiple subjects and grade bands. However, each submission is limited to one subject and one grade band. The products that you are submitting must be used for supplemental instruction and should not be classified as tier one materials. When you are looking at the studies that you're submitting, first note that you are limited to two studies. So, we strongly encourage you that you provide us with the best that you have available. Secondly, studies older than 8 years will not be accepted. So you have to make sure that you're providing us with the most current information you have. I'm also providing some clarification regarding what we mean by
011peer-reviewed. And so peer-reviewed means that it has undergone independent evaluation and critique by scholarly peers that hold no conflict of interest. And we do understand that sometimes companies have to fund these uh studies, but it means that the reviewer does not stand to personally benefit whether it's socially or otherwise from a positive review. Um later on during this presentation, we're going to dig a little deeper into that. The studies that you submit um should be intended to support efficacy for a specific grade band and must include treatment groups that align with that grade band. However, a study will be accepted for the full grade band if it includes treatment groups that cover at least half of the grade levels within that band. The research studies provided should also indicate that the product is more effective
012than individual or group instruction. Again, this product is going to replace uh an in-person or individ or an inerson or virtual tutor that's providing instruction. So, it's very very important that the studies that you submit to us prove that your product is more effective than the live in-person or virtual tutor. I've provided you with a screenshot of what the application looks like in Smart Sheet. It's important to note that the questions are displayed based on your responses. So, there is a display logic to help you ensure that you're providing the proper response or the appropriate response. So if you answer a question that doesn't align with what we're looking for, the application will notify you that your response indicates that your product does not meet HB1416 racial wablous criteria. So it's a pre-screener for both
013you and for us. For example, if you answer, you know, you will see the question asking you if this is a tier one material and if you respond yes, it is, then it's going to tell you, okay, thank you for your interest, but this does not meet our criteria because again, we are looking for supplemental materials and not tier one materials. So that's how the rest of the application is formatted. It's formatted to let you know when your response is showing you and us that your product does not meet what the criteria that we are looking for. Please ensure that the links that you are providing as part of your responses are accurate and functional. This is very important. Links that are not accurate means that they lead to a page what where it is
014not clear what you're trying to share with us or that do not work which leads us to a error message will keep a product from being approved or cause a delay in the review of the product. Linked documents with screenshots should be no more than one page. If it takes the reviewer, the review team more than 10 seconds and multiple clicks to access the content, then that specific content will be considered non-existent. And lastly, we do not accept demo accounts. Um, we just don't have the the capacity or um just definitely we we don't want to guess on what it is that you're trying to have the reviewer see. So, it's very important that what you provide to us in your submission is accurate and it's showing us exactly what you're wanting us to see.
015So again, this is your opportunity to show us the best that your product has to offer. So it's important that you provide us with accurate um information. I'm going to briefly go over the product evaluation criteria. And this is um so the the rubric has two parts. We at TEA look at part one. >> Shut that up. >> Yes, ma'am. >> Okay. Sorry. We have uh we have the first part that we look for and one is just that we want to verify that you've either attended the webinar today or that you've at least looked at the recording um and that you've also taken a close look at the application guide. And again, this is our way of trying to ensure that you you are um familiar with what it is that we're looking for.
016And again, we're just trying to set you up uh for success. We're also looking at product alignment with statutory uh requirements and with strategic focus areas. We're also looking for text alignment per applicable subject and grade level. We're also wanting to ensure that you're providing biling emergent bilingual resources for our students. Most importantly, we are looking for presence of evidence that indicates that your product is more effective than individual or group instruction. And this year for math only, we are looking to make sure that you have EMRA approval. If all the boxes check yes for this section, then you move on to part two of the review, which is what Texas Tech University completes for us. And in this second part, they're looking at um progress monitoring, progress monitoring tools or reports um effective tools
017that support student attendance. And then they'll rate these areas as high level, medium level, or low level of supports. They're also looking at a recommended research-based usage metrics for the product. And then they will score them as strong, moderate, or weak to no evidence. They will also look at rigorous research studies evaluating your product efficacy. And again, these will be scored as strong, moderate, or weak to no evidence. And lastly, they're looking at the uh evidence indicating that the product is more effective than individual or group instruction. Now, we want to remind you that we are scoring your product based on the evidence you have submitted, and that's what we're limited to. So, again, we're strongly encouraged that you provide us with the best that you have to offer. For the student independence piece, we're
018looking at what your product provides. In other words, how does your product mimic what a live um inperson or virtual tutor would provide to the students. So, we're looking at does your product uh provide lesson adjustment supports that allow students to get individualized to retach without teacher intervention. And in this area, your product will be scored at high, medium, or low. And again, we have the definitions of which what each category consists of. We also look at whether your product provides adaptive lesson assignments based on student data. And lastly, we're looking at whether or not your product provides feedback that supports next step for students. We also look at the type of evidence you provide regarding student usage targets and product efficacy across populations and grade levels. Heather, I will let you um speak a
019little more to this. >> Uh that's right. Uh sorry. And I I just wanted to make sure that people are looking at the chat too, just in case there are additional questions there. But yes, that's exactly right. It's really important to make sure that you focus on the uh evidence um that Alysia just said. Um and that evidence is what gives the justification for the evaluation scores that we give. Um, when Alisi was talking about peer-reviewed, as she mentioned before, peer-reviewed means that it has undergone independent evaluation and critique. Now, we recognize that sometimes your product might have been chosen by a group of of researchers that were looking for publications, and so you'd have um, I guess a a sort of truly independent review process. But we also understand that for many folks this
020will involve contracting with a review team with a a team of researchers to do the studies for your product. That's fine as long as none of those researchers stand to personally benefit. So economically, socially, whatever, by giving you a positive review. Um, and this is where if you have any sense that this might be second-guessed, um, I I might suggest that as you're submitting that research, you have a sort of note that just like we do as researchers where there is a conflict of interest statement. In fact, most researchers will add this if they have been paid to do something. Sometimes it happens at the beginning of the document, sometimes it happens at the end, but you state here's how we were contracted. Here's our conflict of interest statement where we we testify basically that
021we don't stand to gain personally from the review that we have given. So there is no sort of sense of I must be giving a positive review. And Alicia, did I address the question that you brought up? Sorry, I was I think I was reading the text rather than doing some of the other things. >> Yes, ma'am. All right. Thank you. This is also yours. >> Yes. Okay. So, this is another thing that is different this year. Well, different from the context. So, if many of you are familiar with what works clearing house uh the the what works clearing house framework used to include four tiers. They have just this year got rid of one of the tiers. So now What Works Clearing House only has three tiers. That's because basically they got rid of
022tier 4. Um so no longer can you qualify for What Works Clearing House unless you fit that tier one experimental design, tier 2 quasi experimental or tier three strong correlational statistical connections. Um but we believe that it's still important to recognize that that tier 4 exists. that there are sometimes when you have studies where really what you're you're analyzing is the theory of action or the logic model. What you're analyzing might be data from other sources that you're compiling together to sort of make your case. Uh we have seen studies like that. Now to be clear, tier 4 will always be sort of worth less points than tier one, tier two or tier three. But because within the field of educational research, tier 4 types of studies are still um done often. We found that
023it was important as a team to still have tier 4 in this particular context. So you will still see just like last year we're naming tier 1, tier 2, tier three, tier 4. We are keeping tier 4 even though what works clearing even though what works clearing house has got rid of tier 4. and let me know if you have any questions about that. >> Thank you. So, here's our timeline. [clears throat] The application window is now open and it will close December 12th. Now, on November 21st, EMRA will have a final vote on supplemental math products. This will determine whether or not for those of you that are providing a math product, whether or not you can apply. So, again, you must have EMR approval in order to be able to apply for the
024racial abil. And for this year, again, this is only applicable to math. We will announce uh we will notify vendors of the review outcomes in the spring of 2026, including next steps for those who are approved for the 20 uh sorry 26 27th school year. It's important to note again that uh the EMR cycle for 2026 will begin the review in May and that will include supplemental reading products. This year again, EMRO will announce um its or will complete its product review and announce the findings on November of 2026 and next year we will adapt our review schedule based on EMRA. So we were anticipating a fall or winter 2026 uh racial waver list window. We will have office hours to answer your questions. We will host this on November 10th at 11:00 a.m. and
025we're going to address any questions you may have. So, I strongly encourage you to please please email us your questions so that we may have a response for you by the time we have the office hours. And some of these questions, especially those that are more commonly asked, we will uh create a PowerPoint and we'll have this on the slide deck with the responses. So, again, we work uh closely with the Texas Tech team to help us provide you with the best guidance possible. If you have any additional questions, again, please email our accelerated instruction email address, but please do not reach out to our third-party reviewer. The questions do come straight to tea. If we need to collaborate with uh Texas Tech, we will, but please send them to us first. Reaching out to
026our third party reviewer can result in being removed from the review process. So, to ensure that we're being um transparent, that we're just following any any rule, whatever, just make sure that you're emailing your questions to us. Um, if you're needing additional guidance, you can also um click on this link for the what work clearing house procedures and standard handbook. Again, our process is closely aligned with what works clearing house. So, this may also give you additional context. Do we have any we have time for a few questions? If not, I'm going to go ahead and show the link um so that you can provide us first with some feedback on how we can uh improve the support we provide to our vendors. And if you have any questions, you can continue to post them
027on the chat or you can come off mute. >> Alisia, in the chat, we'll go to the top. There's a EMR question. >> All right. >> If you could I'll let you find it in the chat and read it. There was a EMR question when you were covering the EMRA portion. >> Okay. So, it says if a product is currently under EMRA review and does not become approved in November, can the product apply for the cycle that is open now and reapply for EMRA? So if you do not get approved in November then you cannot apply for this current cycle. You can reapply for EMRA for the cycle of 2026. So and then you can reapply next cycle which would be next school year. But you cannot apply for if you do not for math
028only. Let me clarify. This is for math only. If math products do not get if your product does not get approved you cannot apply. >> Yeah. And for future iterations, we will look to um align the process. So there's still there's still time to apply um because reading will be next on deck and reading will be handled as math was this last cycle of uh the EMRA November uh I think November 22nd. So we'll account for that for reading as well for future approval cycles as subjects come up for that. Lisia, this is Liza. Um, I just wanted to make sure everybody understood that if they were going to try and apply for IMRA 2026 for supplemental RLA, the first deadline for IMRA 2026 is today to submit a form A. So hopefully everybody is
029aware of that because if you miss today's deadline, I don't think you'll be able to have the opportunity to participate in EMRA 2026. >> Thank you, Liza. Thank you for pointing that out. And just so everybody knows, just because you apply for Emirate 2026 does not mean you'll be selected for review because that they'll get hundreds of applications and then they've got to narrow it down to like between 250 and 300. And each grade level is a separate program review in IMRA. So a grade K5 program is six six separate reviews. Thank you, Liza. >> Josh, I think you have a question. >> Yeah, I just want to clarify. Well, I have a question, but also want to clarify for the EMR 26 RLA review because only K5 RL products are being reviewed. uh you
030were only speaking to for the next cycle for 14 16 uh fourth and fifth grade reading uh or students who failed third grade star or fourth or fifth grade star. Correct. [snorts] >> Correct. >> So it would still be open to any non-imber products for middle school or English one. Okay. Um will you elaborate a little bit more on uh the automaticity of the product meaning no teacher involvement because just as someone who just went through the very rigorous Ibra 25 cycle for math and we we're very helpful for our scores I half of the rubric is about teacher supports and embedded supports like during instruction and I just don't my I just can't imagine a product that's automatic without teacher involvement even getting approved as supplemental math [snorts] because the rubric requires so much
031to get approved by6005 it's like you you need to be supplemental for the teacher not an independent product from the teacher that you can just handle it on its own [snorts] >> all right okay so let me clarify so to HB14 a racial release product is not meant to have just a student log onto the computer and then the teacher not do any type of progress monitoring ing or um not be involved. So the the school districts or the school systems that have been very successful in using racial waveleness products do have the teachers involved and so they'll have them very um actively engaged in supervising the progress monitoring. They're very intentional on what lessons they are assigning to students to ensure that it aligns with what's happening in the classroom as well. Um, so
032I thank you for giving me the opportunity to clarify what we meant by it does not require a live inerson or virtual tutor means that a student is able to log onto the computer and complete the lessons and then the program automatically like can identify when a student is struggling. So you have some type of video or something pops up to retach a student or um it adapts the lessons that the student is completing or the next steps based on student performance. And the teacher of course is involved in adjust making adjustments um ensuring that the student is getting the support he or she would need because we all know that teachers have a lot of and they can't go home with the student to provide them with the additional support. So, as a former
033educator, you know, I was in the classroom for over 16 years. Um, I had to create the ex additional lesson so that the kids had the extra practice that they needed to succeed in whatever it was that I was trying to teach. So, if I would have had a racial this product back then, I would have used it to say, hey, this product has already created the lessons that perfectly align with what I'm trying the students to master in the classroom. And so, that's what these products are meant to do. They they give teachers already lessons that are ready and also they provide that intervention that a teacher cannot give a student for example at home because students are able to access these products remotely and so they provide the kids the retach or or
034the support they need at home where a parent can intervene which they shouldn't right uh or where a teacher cannot be there with them. So that's what I mean and and regarding the EMRA uh rubric there are different ways that a racial waiverless product can fulfill some of these requirements. it's just not going to look the same as a traditional uh pen and paper or traditional, you know, materials. So, it's going to look a little different. And we have had the conversations with the EMR team just to ensure that as they're looking at our products, um the lens changes a little because again, these products are meant to remove the need of having an imperson tutor. Kobe, is there anything you'd like to add to that? No, I I think you're you're framing that correctly.
035And I put the chapter 104 uh rule. It goes in a little more depth of the responsibility of the district uh that's using these products to monitor the usage fidelity um to meet the criteria for this because there's it's not a Instapot. I say it's not a set it and forget it type product. There still has to be an active role from the district involve specifically the teacher and involving the monitoring of this. It's not, you know, to an extent where you said it and just it's going to go and and do all things needed to support the student in the accelerated instruction piece of supplemental accelerated instruction. But to Lisia's point, we've we've talked with the EMRA developers part of that of saying, you know, supplemental independent lens and and I've even shared the
036rubric we're using with that as well. >> Thank you, Joshua, for your question. That's a great question. >> No, thank you. That was a very excellent response. It makes a lot much more clear and hell. I do appreciate it. [snorts] There's a there's a question in the chat I wanted to speak to. They asked about the final scored rubrics. Are those going to be publicized anywhere? So, if for some reason a product were to be rated as weak in some area, would districts be able to look at that? We do not publish the final scored rubrics anywhere on a website uh available, you know, publicly anywhere. Now with that said, that does not mean that we do not get uh public information requests filed. Um in that case, we do um ask for exceptions because
037we do know that there is uh sometimes a competitive advantage for competitors to get that information um and do various things with it. Whether that's granted or not is often out of our hands. it goes to the uh attorney general's office where they look at it and and evaluate it and determine what can and cannot uh be released. That goes for application information as well because that came up before. You know, sometimes there's internal documents that are shared as part of the application process. And a lot of times they'll reach out to you, the attorney general will reach out to you and say, "Is any of this information proprietary? Can it be released as part of a public information request? And many times that's up to you to determine what what is proprietary, what is
038public publicly available, but TEA does not publish anything on a website about the scoring criteria. We give the rubrics up front for transparency for the vendors. We want y'all to be able to submit that. A a caveat to that too is many times we'll uh talk with vendors that did not make the list and they'll want to visit and come to find out they have a disconnect internally with communication of what's submitted. They may have a salesperson not knowledgeable in research submit their research examples. That's probably not the best strategy or approach. Um, or they may have a study that's 15 years old and and I realize RCTs are very costly and difficult to to stand up and you may have a great one that happened 10 or 15 years ago that showed efficacy of
039your product. That's not that's not the best research to put forward to show efficacy of your product in the current state. and come to find out, we're talking to a different party that didn't attend this webinar or didn't come to office hours and didn't wasn't part of the submission and they're like, "Well, we had our sales team submit the research and we don't know exactly what they submitted and there's this disconnect and there's this aha moment like, oh, we didn't we didn't submit the best research to to demonstrate our product's efficacy and our reviewers are going to see what is submitted and and that's it. And it's not saying your product is not great, it doesn't have great outcomes, it's not well known and revered and and push students forward. It's just what was represented
040and what was submitted did not not exhibit that. So, I would just keep that in mind as you're planning this out and communicate within your team. Ensure that you're getting the best information submitted for it. I know you have to be selective in what research studies you submit and ensuring that the links are correct and that that that information is there. Th those are probably the biggest ahas and the biggest things that we've received feedback on. I just want to be sure we're upfront about that. Uh those tend to be the biggest things that derail um our providers when they're submitting their applications. And Heather, you may have a comment on that. I just I see you shaking your head, but >> no surface that >> Hobie is bringing up real life examples here where
041somebody submitted something because they thought this is what they were supposed to submit where the researcher side of the group would have told them that's not that's not a good study to submit. Please submit a different study. But because of that disconnect, the reviewers can only evaluate what we were given. And so if you are not submitting your strongest study, that is what we will evaluate. So please make sure as an internal team that you have that strategic conversation that everyone is on the same page about what actually got submitted because to Col's point that that has been an issue in the past. Darl, you asked if the recording of this session will be shared with the vendors. Alicia, I think you mentioned it from the beginning. You're going to post this. Is that correct?
042The recording of this? >> That is correct. Once we're done, um, I'll submit it and it may take us a few days, but I believe maybe I anticipate by midweek to, uh, late next week. >> There's a question about, uh, cover at least 90% of the TESS and work towards 90% ELPS coverage for all selected grade levels. Does this apply to all requirements or just the items we submitted for EMRA review? Do you understand that Alysia from Laura was asking that question? >> Let me look at it. >> So I can give you some context. If you're a supplemental pub, if you are a publisher who submits a supplemental math or RLA program, you get to tell TEA these are the specific student expectations within the TEICSS that our program meets. So the minimum is
043one student expectation. So a program could be approved through EMRA if they meet a single student expectation for math for for the entire set of maths or uh reading. So I think the question what they're inferring is let's say they submitted a supplemental math program and said we meet these three T or three student expectations you have to meet all of the breakouts for those. So if we indeed did pass the EMR review meeting those three student expectations, does that meet the we meet 100%, you know, 90% or more of the standards? And if whoever asked that question, if that's wrong, please jump in. But I'm assuming that's what you're getting at because a supplemental program will never meet 100% of the TEICSS and ELPS. That that that would be a core program. >> That's
044exactly what I was asking. So for for racial waiver list because we're looking at star performance and we're trying to support um students and star we're looking at the 90% of text coverage. So I will look at what you the alignment you submit uh license. So, I hope I'm answering the question >> and Alicia, I can send you the verbiage out of the EMR, the EMR RFIM where it talks about a publisher self- selects the the standards they're going to cover. Um, so you guys can like look at that and kind of internalize that because it is a way, you know, it's a different kind of thing. >> Okay. And I'll I'll go back. I've captured the question in the chat and I'll come back to you. do some more research on that one. Laura,
045>> thank you so much. >> Thank you. >> And then Amy, you have your hand up. Did you have a question? >> Yes, thank you. Um, so it's great to hear that you guys are collaborating with the Emirate team um and providing some guidance around products that are 1416 approved. My question is, does that go um the opposite way? Um, so as part of your review process, is an I know it's not on the rubric officially other than the yes no question, but are your reviewers looking at the EMRA 1605 scores for the products that you're reviewing and taking that into account um as they're doing the review. And the reason I asked this is because we're limited to the onepage PDFs and sometimes it's a little bit difficult to get our point across exactly
046um to make sure we're checking all of those rubric boxes. So I think that that would be helpful if that was part of the process. >> That will be part of the tea portion of the rubric, Amy. Um but I will I will look into that a little more just to provide the additional context. But on my piece uh for the first part for the TA part I will be looking at that not as in depth but I will definitely look at um and this is our first time having that information available. So I I'm just excited to see what what I get to have access to. Amy, >> great. Thank you. That's helpful. I mean, that's that's a good point because we're getting to our first round of IMRA potential EMRA approvals with the
047mass supplemental and then subsequent subjects with reading. So, uh that that could also be part of the um integration to your point of adding that additional context for for the review and could could give that as well. Uh a question uh Daryl asked about needing to identify stuff on applications as confidential or proprietary. That's not necessarily needed. You could do that, but just understand if there is a PI, we will uh submit it for review. And if it is reviewed or it is in question, they do reach out to you as a vendor or whoever your contact is and say what information is available for release and they have you determine what is confidential or proprietary um out of all the information for your application um before it can be released is the process that
048has happened uh with previously previous uh public information requests. Just so you know that. So that that's kind of how that process has been working on that end. Joshua, >> I don't know if that's from before or a new hand. >> It's a new one. It should be my last one. So thank you. >> No, you're fine. That's what I hear. >> Um we uh I to just be transparent with everyone. I mean LA we submitted for this last year there was one section the research sources where we were allowed um two non- yeses and we got a no and an unclear so we made sure this time that unclear section was was very very clear however we thinking if we could just submit it again with more clear evidence it did seem like with
049or it didn't seem like we noticed this year's rubric had added in parentheses for student independence and student usage targets It's I'm sorry product efficacy uh that high level was required where in the past we we had moderate. Could you tell me and maybe this is a question for Heather is there an example of what you're like what's the difference between moderate evidence and and high level evidence like what do we need to put front and center so that um it's clear [snorts] >> and I'm going to share my screen so you >> I was going to say let's give a concrete example because I I did not follow which question you were talking to talking about >> and I think he's talking about this piece correct >> yeah where that yeah that's was new
050this year to us we didn't see that requirement last year where it says yes plus the strongs required. >> Yeah. And and just to be clear that this has to do with the fact that um as evaluators we really are trying to give the benefit of the doubt but we are also really trying to abide by the sort of letter of the law of can we see evidence of it and so if I can see a um so this is part of so we're specifically talking about across uh variable populations. So this is where if I'm a researcher and I I take the time in my research study that I have submitted to actually do the plots with different demographic groups and I've done my statistical analysis with multiple different demographic groups I would say
051that provides strong evidence. If on the other hand I have maybe I have one study that that you know has that plot but another study that doesn't really have that plot and I talk about it in the narrative but I haven't really given you concrete evidence that I have done the statistical analysis across different variables that might be moderate evidence. If on the other hand, it's not even brought up. I just have here's my whole population. I have done the statistical analysis on my whole population, but I have not taken the time in the study to even talk about subgroups. That's probably more like weak to know evidence. Um, does that get at your question? >> It does. You prefer to see like tables with the labels and the percentages and the points. Yes. Okay.
052>> Pretty much always. But yeah, basically in general, we're looking for evidence that you did the analysis for that question and and that's what we look at. Where can I find the evidence that you did that kind of analysis? And so you you will notice, thank you Josh for pointing out that we did have some changes in the rubric, but that's just because we're trying to make the process more transparent and and provide you with clearer guidance. And you know, we have had vendors reach out and say, "What was what was I supposed to get in this section or that this other section?" So that's why we have we've added the additional information in parentheses to let you know this is what's required for you to um make it on the list. Um the only
053section that does not have that are the bot the last two. It used to be called research sources. We've divided it into two um for more transparency and we have quality of research when we're looking at research execution. And the other quality research piece is research design or research sources. Um but we did not specify how many yeses or nos. Uh you know the team will look at it and then they'll give you a score on that one. I I see in the chat there may be a error with our application preview. We'll have to check that out and get that updated if there's >> okay >> an error with the file. I'll have Alicia will look into that. Kellen, we'll see what may be wrong with the file and get that updated as soon
054as possible for you on that. >> Yeah, thank you. I uh downloaded it a couple of times and had some of my teammates download it just to make sure that I wasn't uh seeing the wrong thing or that it wasn't just my computer being weird. Um but it seems that the section for progress monitoring and student independence has the exact same questions and so uh I just didn't think that was correct and when I saw your screenshots I >> okay >> definitely saw that. So thank you. >> Well we'll look into that sometimes. Um, sometimes that happens and it doesn't doesn't format correctly. So, thanks for pointing that out. Apologize. >> Oh, I completely understand. No worries. [laughter] >> Check and check and double check and >> I appreciate it. Thank you. >> Yeah, we'll
055we'll try to get that updated as soon as possible for you on that. Yeah, the Smart Sheets application has the correct questions. So, you can get those if you need them right away from the Smart Sheets application. So, we'll get that updated for you. The other thing is uh we we've increased our transparency in the process. This is our third round going through that and that's to help you hit the targets. We want y'all to have, you know, the best opportunity to provide the highest quality research on your product that shows efficacy of what your product can do for student outcomes and and overall it's I think we've seen higher quality come in for submissions, which is great. Um, ultimately our our end goal is student outcomes, help student outcomes and and help you as
056a provider uh to hone your product and and find, you know, areas. And we we've talked to our current ratio waiver list providers and the districts they worked with in our our tutoring conference. the districts came and presented and it was it was great to see how that they they've been able to integrate the ratio waiverless products for student success and how it's fit in with their tier one instruction and other other needs that they may have. So, part of that transparency is we've introduced you to Heather uh with Texas Tech University. And I know Alicia may have mentioned this or alluded to it, but that's not any license, shape, or form to reach out to Texas Tech University as part of this process or email Heather questions about this process. Uh that could put
057your application at risk or your consideration for this process at risk. So, please do not do that in any shape, form, fashion. Um, you know, Heather personally, that's a different uh personal relationship with Heather, but uh otherwise about this process, that's not uh that would not be appropriate in this case. Any questions about this process? This review needs to go through uh the accelerated instruction email help desk or you can email Alicia or myself. But direct questions about the application, the review process can come through office hours or us directly. Um but please refrain from directing any questions, concerns about your application. um the Texas Tech directly uh they work they work through us for us in that in that capacity and we oversee this process totally. So I just wanted to add that caveat
058there with this as well. >> Thank you Kobe. And again anytime you email us questions um I don't make up an answer. I I always reach out to to Heather and we work very close together. She's taught me a lot regarding I'm not a re expert researcher so she has really taught me so much. I've learned hobby and I have learned a lot from her but um yes please email us your questions. We we try to respond in a timely manner unless if you provide me with a question that I think oh wait we can address this like in an upcoming webinar. Um then that's the only reason why you won't hear from me. But the questions you send me now I'm going to assume that you would want me to answer in office hours.
059U that's what we're trying to do. That's why we set up these um additional supports for you so that everyone can hear the same uh guidance that we're providing you. So again, just again to ensure that no one says we're providing more support to a specific vendor. Usually I'm like, "Oh, this is something that everyone can hear. I'll wait and then in our office hours is when I'll address some of the questions." So, as you're working, as you're looking at these questions, um, and you have something that you need guidance with, email us the question and we can definitely address in our office hours so that everyone can hear. But if you have a question, more than likely another vendor might have the same question. And again, I apologize for the repeat of questions. I
060just saw that. Um, I'll get that fixed. When you convert documents, sometimes these things happen. Do we have any other questions? Otherwise, I would gladly give you 5 minutes back since we were scheduled to be on until 11. All right, then we'll wrap this up. Thank you again for joining us and