Virginia Board of Education Work Session 02/26/2025

001e e e are we up okay good morning my name is Grace crey I'm the president of the Virginia Board of Education I'd like to call this meeting to order at this time uh the chair will entertain a motion for the board to go into executive session um the chair recognizes Mr Hansen I move that the Virginia Board of Education convene in a closed meeting for the following purposes discussion or consideration by the Board of Education of information relating to the denial suspension or revocation of teacher licenses subjects to the exclusion in subdivision 11 of section 2.2-cu pursuant to section A40 of the code of Virginia and consultation with Council regarding specific legal matters requiring the provision of legal advice pursuant to Virginia code section 2.2 3711 A8 of the code of Virginia and the

002legal councel to the Virginia Board of Education and staff whose presence will Aid in the board's consideration of this matter participate in the closed meeting is there a second motion has been made and seconded by Mr Hansen seconded by U Miss Northam all those in favor please say I any oppose no okay the motion carries and the board is now an executive session all staff members of uh and members of the public who have not been authorized to participate in this executive session we need you to exit the room at this time please um the board will reconvene a public meeting immediately upon the conclusion of the executive session thank you e e e okay um we are back and um the chair will entertain a motion to certify the purpose of the meeting and

003closed session the chair recognizes Mr Hansen we convene today in a closed meeting pursuant to a recorded vote on the motion above and in accordance with the Freedom of Information Act section 2.2 3712 of the code of Virginia requires at the board certified that to the best of each member's knowledge only public business matters lawfully exempt from Hope and meeting requirements under this chapter and only such public business matters as were identified in the motion by which the closed meeting was convened were heard discussed or considered any member who believes there was a departure from these requirements shall so State prior to the vote indicating the substance of the departure that in his or her judgment has taken place the statement of the departure will be recorded in the minutes is there a second okay

004a motion has been made and seconded this will be a roll call vote all those in favor will indicate by saying I miss Kilgore Dr Northern Miss mcferson Miss Holton Mr Hansen I and chair votes I okay we officially out of closed session um and I would like to bring us into the work session but before we start that I just would like to make it um aware give the public a little bit of notice that today we're kind of um jumping around on our agenda we had to take some cases early this morning um we are coming out of Clos session and beginning the work session in order to address a couple of our speakers who have come to to um grace us with their presence today in order to provide us um their

005National expertise in a couple of different areas that the board is examining currently um and so we are thankful that they are here with us today and uh our first presentation is uh standard setting definition of proficiency in scoring assessments but what I would like to do is take a moment to remind the board and the public that we discussed this in August and again in December that the board will be discussing the return of proficiency standard setting and ensuring our expectations are on top in the nation Virginia believes that Virginia students deserve an excellent system of public education that prepares every Virginia for success and to provide the best in-class education system that every student deserves it requires an increase in rigor and accountability for all students under the Administration the board voted to

006lower the proficiency cut scores on the state assessment and lowering those cut scores reduced the number of items that a student must respond to correctly to be categorized as demonstrating proficiency this changed this change caused the Virginia standards for reaching proficiency to be among the lowest in the nation for reading and Mathematics um we will see here as we discuss in the next couple of um presentations that um the nape State mapping study demonstrates the reduction in rigor that occurred here in Virginia governor yunan announced that Virginia's lowest in the nation Mastery standards would be raised to align with the demands of the knowledge economy as part of our commitment to Virginia's Event in May of 22 Virginia Mastery will be benchmarked against employer higher education knowledge and skill demands Nate efficiency and the highest

007state definitions Across the Nation Governor yunan is committed to restoring Excellence to education which includes more rigorous standards in the four core subjects and greater transparency and accountability and a new assessment system the evidence-based process requires standards to be raised first assessment items to be aligned to content and rigor and Mastery setting based on the results of the new content and questions in the assessment Governor yunan directed the board to raise Virginia's expectations to be the highest in the nation by aligning them with national assessment of educational progress Nate proficiency standard and that is our goal in launching this process today after this presentation we will hear from additional National experts on these particular topics and what I would like to conclude Within These remarks is to say that today is the day for questions

008to be asked and answered and for public comments to be made on these items tomorrow is our business meeting um and we will proceed with business in an efficient and effective Manner and so I would ask that if you have questions concerns comments that they be aired and discussed today rather than um spending hours on in tomorrow uh during a business meeting uh so with that Miss Cooper please proceed Madam president miss Holton just before if I may respond briefly to your opening remarks before we hear from Miss Cooper briefly um I just want to note uh just for the board's Clarity the Constitution gives this board the very explicit responsibility to make stand make decisions about standards and standard setting and uh uh we certainly welcome the governor's advice about aligning with Nate proficiency

009and that's certainly an option that the board may choose to do but it is not the governor's decision it is our decision that would be my first point and number two unless I am missing something we have not yet made that decision we've talked about it a lot in a lot of different ways but I think that decision is before us on what's being presented on first review tomorrow and presumably final review at our next meeting so that is a decision point for us going going forward not a done deal thank you Cooper good morning president cruy Dr Coons and members of the board today I am presenting our proposed proficiency standard setting process to seek approval and input on key Steps From the board this ensures that we are responsive to the feedback from

010the field and Community about a transparent process it is important to note that previous iterations of the committee selection process have not been brought to the board in prior years as president Cy mentioned Virginia has the lowest standards in the country in ela and math two key actions occurred several years ago to focus on increasing rigor and accountability the first one was in 2022 Governor yunan directed the board to raise Virginia's expectations for students from the lowest in the nation to the highest by aligning them with the Nate proficiency standard as part of our commitment to Virginia which you can see on our website two House Bill 585 of the 2022 legislative session required a workg group to assess the current state of Virginia's assessment system and to develop a plan for the future state

011of Virginia's assessment system two of the areas that the work group recommended were clearer and more rigorous standards and more rigorous assessment items over the past three years this board has prioritized raising expect expectations for students through the adoption of more rigorous standards of learning and through the approval of the new school performance and support framework vooe has also redesigned summi of Assessments to more fully measure the rigor of the new academic content standards building on this Foundation the board must Now set new assessment cut scores on these Summit of Assessments to align proficiency to the increasing demands of the workplace and Benchmark to ensure Virginia students perform best in the nation I'd like to revisit the honesty Gap in Virginia's student performance for over a decade the definitions of proficiency have been far lower

012on Virginia test than that of the nation's report card and Virginia has had the lowest definition of proficiency in the country for the past four years as you could see our state's so te test scores tell a different story than what national proficiency scores tell us what we have communicated to teachers students and families is not an accurate picture of student performance and as of 2022 there was a 40 percentage Point Gap Jim Cohen who has been the national expert around the honesty Gap is here to chat more about this topic and answer questions after this presentation here you can also notice the misalignment of Virginia's standards to National standards the national assessment governing board approves the standard setting process for nape and a state mapping study was released comparing State's proficiency standards to nape

013proficiency standards this study is another demonstration of the reduction in rigor that has occurred in Virginia our proficiency standards map to nap's two lowest categories that are not proficient below basic and basic this in addition to Virginia students continue to struggle on nape as reported in last month's nape release highlights why we must adjust our cut scores the board must ensure that Statewide definitions meet National expectations to ensure that Virginia students can effectively compete in a national and Global workplace we have brought Leslie malun executive director of nagby here today to share more details and address questions you may have as well to achieve AI the best ination expectations the implementation timeline continues through this summer The Board of Education will be highly involved in the decision-making process which includes methodology timelines and Committee Member

014process as mentioned earlier once new rigorous standards are approved and the assessments are aligned the last step is to complete the proficiency standard setting process the board approved new standards and those assessments have been aligned this fall the TAC recommended that vooe complete the standard setting process at the end of the spring assessment cycle so that the most current data from our students is used in this process this timeline allows for the consideration of the 2024 Nate data in establishing the updated proficiency standards this month the board will vote on the standard setting committee process as part of the board overseeing the process and making critical decisions around the setting of cut scores this spring students and teachers will receive score reports that use the current Mastery levels that are already in place in addition

015the standard setting process will occur over multiple days at the end of our testing window this process will allow the committee to use student performance data to understand how the standards and test items are providing rigor how the cut score should match Nate performance and align to the Knowledge and Skills of Virginia as employers and admission standards to higher education the board will vote on the proposed cut scores recommended by the proficiency setting committees and the technical advisory Council in June and July the updated performance standards will not be applied until spring 2026 this timeline allows the process to be completed prior to the next school year which allows for transparency and preparation for the new year to Educators and family amilies and provides an opportunity for school divisions to receive training and resources on

016the new cut scores the standard setting Review Committee reviews data current performance levels and assessment item types to make recommendations on future content area cut scores for the board with the goal of so cut scores being aligned to nape the national assessment governing board recommended that Virginia mirror the standard setting process that is implemented for Nate the proposed Review Committee process has four key phases during phase one committee selection the department will hold a call for nominations once the board approves the process for selection we recommend that this committee be comprised of 70% Virginia educator experts and 30% community members such as higher education experts Workforce and Industry leaders and other community stakeholders about 200 members would be selected over about 20 committees in phase two committee training it's recommended that all selected committee members

017go through several training steps to ensure that members are prepared to participate in the committee meetings in an exemplary way next in phase three committee convenings we're proposing that the committee would review specific grade level in content specific data the vertical band of performance levels current performance data from grade level students and assessment item types to make recommendation on Cut scores they would be responsible for recommending cut scores that not only reflect students achievement status but also promote the high expectations for all students in Virginia and then in the Final Phase board and stakeholder feedback our plan is to have cut score recommendations presented to the board for first review and feedback in June proficiency cut scores would also be released for public comment and then VOE would provide review that public comment in coordination

018with the technical advisory committee and proficiency cut score recommendations would be presented for final approval in July this proficiency setting proposal will not only ensure that every student in the Commonwealth is held to high standards but it will also address the honesty Gap we are so privileged to have Dr Chris domaleski here today Dr domaleski serves on our Tech and is a nationally recognized expert in assessment and psychometrics there is no one better in the country than for him to be here with us today and as you can imagine he is in high demand so uh Dr Diles we are so grateful for you being here and rearranging your schedule to speak with our board he is going to share more Det details around the standard setting process to obtain cut scores so I will

019turn it over to Dr domaleski thank you very much um it's good to be with you again today thank you so much for being here I know you had to do a considerable rearrangement and we are so so thankful to have you here thank you my honor thank you for the invitation uh president KY superintendent Coons members of the board thank you for having me back um glad to be here today and I'll keep my remarks succinct because I do understand that we want to leave time for some interaction and I'm happy to uh to to entertain uh questions and discussion um at the pleasure of the board uh I think I was with you in August last and at the time I had chance to tell you a little bit about myself and the

020center for assessment so we're a nonprofit nonpartisan organization we work with the majority of the states to help uh provide advice on development validation of Assessments and accountability systems and I'm also here in my role as a member of Virginia's uh technical advisory committee I did wanted to point out that the technical advisory committee all reviewed the slides that I'm going to speak to today and have knowledge of the remarks that I'm going to share obviously any limitations are are mine alone but I want to know my colleagues on the TAC who I have great regard for all contributed uh to this information and stand behind uh the message that I'll share with you today so and the TAC will continue to uh to to work with uh the department in refining the standard setting

021methodology and ensure that we follow established professional practices and the guidance of the board uh it's just a reminder what is standard setting um it's the process by which we're uh determining cut scores for Virginia's state assessments um that's the the shorthand that most people use cut cut scores are how those performance standards are operationalized um and many people might be surprised to know that that's not entirely a technical Enterprise it involves judgment from appropriately qualified experts and panelists as the process that we're going to speak to today certainly does and um I think I emphasized this last time but it's worth repeating that the there's no right answer there's no truth of the capital T when it comes to what are the right performance expectations for any state assessment we we strongly believe that

022the credibility and the trustworthiness comes from implementing a strong process implementing that with Fidelity and attending to uh best professional practices and so that's what I'll emphasize in my remarks um I I noted last time and uh and again I'll amplify the point that the standard setting process is a necessary but insufficient overall condition to operationalize the desired rigor of the standards for the Commonwealth um the it starts and I heard Miss Cooper uh mentioned this just now so we're totally aligned it starts with implementing strong content standard standards very clear definitions for knowledge skills and abilities that students should know and be able to do uh in the State uh the assessment needs to reflect those academic performance expectations so certainly we need items that run the full range of uh cognitive complexity and

023really probe the breadth and depth of the standards and substantial work has been done uh in order to uh ensure the test does that and it's reflected in the blueprints and the test in the test specifications and then of course and this is the reason for um my being here today and for your consideration uh the policy priorities policy priorities absolutely inform where standards are set it's really the combination of the content standards the assessment the standard setting process and the policy priorities that go into determining what those performance expectations are and I really appreciate I mean this with all cander I really appreciate the level of scrutiny that this board is giving uh to that those policy priority sometimes that's um an afterthought um and and I really appreciate the board is really on

024the front foot and And discussing very explicitly uh what those priorities are and how to make sure they're instantiated in the process performance level descriptors this a term that I'll use as we talk about uh standards uh the performance level descriptors describe um uh those knowledge skills and abilities associated with each level below basic basic proficient advanced um there are different uh types of performance level descriptors and I won't go into detail on all of them but you see them listed here in the slide um it starts with policy definitions these are usually statements from the education leaders in the state about the kinds of characteristics or benchmarks that should accompany those performance expectations often you see phrases like uh adequate to ensure that students are ready for post-secondary success or on track for post-secondary

025Success so there are various distinct statements of what the policy Ambitions are uh they're more detailed at range and threshold plds range plds as the name implies uh should cover the full breath obviously there's some room say in the range of basic or proficient um for uh students who are just at the threshold all the way to the kinds of knowledge skills and abilities that are associated with students that are right up against the next threshold so the range PLS try to describe that Continuum of expertise uh and how that is demonstrated a threshold plds uh very obviously describes that uh in the case of sayfe proficient that just proficient student and really hones in on uh that U that borderline level of performance and for some standard setting methodologies that's a particularly important plld

026to develop and then reporting PLS is how most of the public interacts with the performance level descriptors these are the statements that usually about a paragraph or so long that accompany the isrs or the individual student reports or other public uh statements about those performance they're intended to be accessible so that a full range of constituencies can appreciate what's being described in those performance levels so there are oh my goodness there are whole uh volumes uh courses uh dedicated to standard setting methodology there's no way that I could very briefly give a full account account for all the kinds of approaches that are in use and the nuances associated with each but I thought it might be useful for me to just pick um three very commonly uh implemented standard setting methodologies and Linger on

027each for just a moment uh the modified angl this was one that was used a lot more uh some years ago it was one of the initial standard setting methods that uh that gained in popularity and um and what I've tried to do with each rather than cover everything is just really focus on what the central task for the panelist is what what are we asking of the panelist in a modified Ang off and and that methodology the panelists are considering the U the probability that a student who is at that performance level be it basic proficient Advan um can answer questions so they go through each of those items and they try to make a probability statement sometimes it's modified so that instead of making a probability statement like other student has a 50%

028probability of answering the question uh correctly or 75% probability they would just answer that in the affirmative or the negative yes or no um and there are different modifications of angol in fact angof almost always is modified to some extent when we see it in practice that is the Legacy method I understand that was used in Virginia so again very common has a strong research base uh The Bookmark method this really started gaining more traction I'd say in the um late 90s and the early alts and the the essential question here the panelists are looking at a list of ordered items ordered by their difficulty um the probability that a student's going to respond to them correctly and they're um placing a a marker in that ordered list that differentiates uh when a student is

029more likely uh when a student at that next level would be more likely to respond affirmatively to those items so um you take those ordered item lists you put those bookmarks um where you start to see the the shift occur from say basic to proficient to Advanced and then there's some statistical uh connections between that item and uh and how it gets translated into a scale score in order to determine where the cut scores are um that's probably a a trail we don't need to follow too much but it has to do with h um determining a response probability and then using item response Theory to translate that to the scale score with ID matching it really has some flavors of each of the two but the central question shifts a little bit for panelists

030now uh panelists are matching items to specific performance levels in this method so we're not so much considering uh would a borderline student answer this question correctly or what would the probability of that student answering the question correctly now it's really more of a matching task now we're saying let's look at what the demands are in the performance level and let's look at what the Knowledge and Skills and ab that the item is intended to elicit and let's match up those items with the performance level in a way that makes sense it still operates in an ordered item booklet so we're still going through uh items in order of difficulty and we're still looking for those transition points when we do that but many have U argued and I'll make this point a little bit

031more on the next slide that that is a much clearer task for panelists that's much more cognitively clear uh we know from research that for example that panelists just aren't really aren't good at all at predicting um student uh probability on on items so this really just focuses on the content and their match to the items to the performance levels and it centralizes the demands of the academic content U we've seen the ID matching uh grow quite a bit over the last 15 years or so there's a seminal publication I think in 2008 that really sparked uh the growth of the ID ma matching method and in my experience Most states are shifting to ID matching now not any way suggesting it's popularity contest that vinia should pick the one that's most popular but there

032are some advantages in having uh a widely used method uh and so far as there are really good methods and we get Lessons Learned and systems in place that help us to implement that with Fidelity when there are widespread practices so but again I'm not ruling out that any of these methods are necessarily insufficient um I I'll be quick here I've already kind of named some of the pros and cons associated with the I I talked about the limitation with Ang off and how difficult it is for panelists to make judgments about item difficulty um with the bookmark you have sort of a less um that's less of an issue about the Judgment task um we do bring in the importance of the ordered item booklet when you go to a bookmark method which of

033course carries over to ID matching um so it requires that there's stable uh and trustworthy data uh available before for the standard setting so you can have confidence in those item parameters both for the um the construction of that ordered item booklet and for the translation of that um marker in the booklet to the scale score so it really matters that you have trustworthy information about that and as Miss Cooper mentioned in her remarks um being able to construct that after an operational or a full census Administration gives us a lot more confidence in our ability to do that with Integrity than say relying on um a pilot or a field test for that purpose so from a technical standpoint it's very um desirable that the oibs can be constructed based on good data whatever

034the method is uh is selected uh I believe these characteristics accompany any method that we're going to call defensible uh we need well-qualified well-trained panelists and uh and you heard some remarks earlier about how uh Virginia is moving forward to Implement that um want to centralize judgments um on content-based expectations uh described in the performance level descriptors uh we want to include efforts to promote coherence because obviously this is being done for every grade content uh intersection and you want an opportunity to look across uh grades for example to look for uh coherence and uh I think we'll have a chance to speak more about that in just a moment and um and with any Sen set method we want to ensure that the do the methods and the rationale are well documented uh they're

035reviewed by appropriate technical experts such as the state's technical advisory committee uh and policy leaders so uh always advocate for full transparency uh throughout the process I think that uh adds to The credibility of what we're doing and how we're going about doing it as I noted uh the proposal is to trans transition from uh an Ang off based method to ID matching method method uh in the past Virginia's imple implemented the modified angl uh and we support uh moving to ID matching we being the technical advisory committee um for again many of the reasons I think I've already um enumerated um it certainly matches up with those quality criteria that I mentioned um strong research base the panelist task being very clear that was something that was particularly important to Virginia's Tac um so

036that was something they elevated in the discussion so uh we think that that is a strong methodology to implement uh to uh establish those performance expectations a note just a bit more about the um the process itself and I believe I also provided a a one pager that covers some of this board may have had a chance to look that over um the I understand the uh the specifics of the process are still being developed part of which is contended on the guidance of the department in this board but the general procedures follow this Arc uh panelists will review the test in the plds in multiple rounds they'll evaluate this ordered item booklet and uh make those matches the matching those the items to the performance levels based on the extent to which those knowledge

037skills and abilities are represented um and then after each round they get an opportunity to see feedback across um all participants and see kind of how their ratings matched with everybody else ratings and the whole idea is that the panelists can learn from one another if somebody is say um more rigorous or less rigorous than their colleagues then they can learn what their perspectives were and they can take that into account as they um complete the task in subsequent rounds and usually the process will provide more information information about impact and as I'm going to talk about um in the present case information about um its correspondence with uh nape data um after that's done U then you get these candidate um cuts that represent the places in that oib where the performance levels shift

038and you sort of see a visual representation of that and the at the bottom of the slide there you see the items are basic basic basic proficient proficient proficient so forth there's going to be sort of a Zone where that transitions you find the median of that zone that gets translated to a scale score um and then uh the final uh process before documentation and review from the policy leaders in the state is to U do a vertical articulation which is to look across grades in content areas and determine if any adjustments are needed for the sake of coherence so a feature that is it's it's not um unheard of but it is less common is um in order to to to make sure that we're aligning the policy priorities of this board and the

039department into that process um we're proposing to bring nape data explicitly into standard setting so the idea is to uh map the nape standards to the state tests and uh come up so you see in this graph I can see it much better over here you see in this graph and this these are remarks I know the board has heard already that there is a very different distribution of impact across those assessments when you look at the nape data this particular illustration shows I think it's eighth grade reading you see about a third of the students below basic you see 37% in that basic range and then you see the remainder of students at are above proficient looks very different over on the left with the Virginia state assessment where you see uh only 9%

040below basic 20% at basic and um More than 70% at or above proficient so the idea here is uh to map nape to the state assessment um so we're really answering the question uh what would be performance on the state assessment that would produce similar impact to nape and then once you do that it's a fairly straightforward statistical um question to answer then you can get these um we're going to call them reasonable ranges we're going to get some Zone around those plausible cut scores and be very transparent very clear with the panelists that in that range we think are sort of it's an attractive area it's a plausible area to locate the cut score um um so you map it you provide those reasonable ranges and then you're just fully upfront with the panelists

041and you say if you I think of it as kind of like shining a flashlight um we're going to point the flashlight in this region of the oib we think that this is a promising area for you to focus in finding uh those transition points from performance levels and um doing so will uh ensure some correspondence between the state assessments and the nape um but at the same time and you can see the balance that we're trying to strike here at the same time you're not obviating the central task of keeping the academic content and the the central question that the panelists are wrestling with at the center of the activity we're just asking them to operate in a zone that is um going to produce impact data that's relatively correspondent um using reasonable ranges

042um has been done with many other state programs in the past like I say some some states you do it some don't some do it with nap some do it with maybe a legacy assessment or with some other impact data that's of high value um so uh this would not be um tell you I wouldn't I wouldn't say that it's common um but it is certainly not without precedent in my experience and then I think this is my uh my last slide here uh the timeline um we're very pleased that the timeline allows for multiple reviews of the process and results such as the review that we have here today uh that the assessment data will be collected under operational conditions which we think gives it more stability and credibility um really appreciate the fact

043that Virginia is communicating those expectations in advance of the academic year in which they'll be implemented that actually is not very common so I really applaud Virginia for taking that stance um and so U then after all of that um the department will be able to document everything submit it for federal peer review uh submit it for all the different review uh channels that it will that it will go through so those are my prepared remarks I'm happy to address any questions okay again thank you so much for being here of course we appreciate it um and I certainly appreciate the constant reminder of the process having served on standard setting committees in the past um and understanding the complexity of the process um and also those cons to the current method that we use

044certainly um having been a part of those committees over the years so uh are there any questions I know Dr Northern said you had a few um just in terms of the ID matching versus the bookmarking I guess my main question is why can't the committee members do both because in my understanding you've got this ordered booklet that presumably the TAC um has done a job of thinking okay this is this is how we think they progress in order of level of difficulty and then you've got the panel members saying okay I think this one's pegged to proficient this one's pegged to below and so on um but with the ID I mean with the bookmarking you actually are asking them to say okay here's where I think the shift occurs From Below basic to

045basic for instance in the booklet why is it that the uh committee members aren't asked to do both like I I understand there's an advantage of bookmarking and ID matching and that they both use the ordered booklet which I'd love because I think that takes a lot of the pressure off of the committee members to do a lot of work um but is there a reason and you talked about you know the probabilities are hard for the panelists and so on but can you talk more clearly about why it is that it's better to have them identify at the item level to the plld versus having them identify the shift that takes place in the ordered booklet that it's an excellent question they are very close and again the the real difference and and this

046really turns out to be one of the most important things about standard setting is in the ask of the panelist the central question that the panelist is sitting with and there's there is a very real difference in bookmark and ID matching on that question in bookmark they're asked to think about a student we're we're giving them a threshold plld and we're saying think about that student who for example is just barely proficient now um address whether or not that student is likely to get this item right or not and if the answer is yes then you mark it so if it's no you mark it so and then you look for kind of where the yeses and NOS start to um transition across panelist the central question we're asking in ID matching is we're not

047asking them to imagine student we're say look at the plds look at what the item is supposed to elicit and see which see if they match and I would imagine you're going to see some correlation between the task and bookmark and ID matching but it's our judgment that the ID matching is going to centralize the academic content and keep the pressure of having to make any kind of predictions about students off their shoulders and we think that that is the uh the the the preferred um task of the panelists um but as you noted um Dr Northern the uh uh the methodology from there forward is very coincidental it's very similar um and so once you get those transition points in the oib the process of finding a cut score becomes very straightforward and very

048simle but is it my understanding that then you guys the TAC would then be the the responsibility then Falls to you more to decide where those shifts occur in that itemized booklet after you've gotten the feedback from the panelist um they really once you get the feedback the methodology is very similar usually you're you're looking at a sort of an uncertainty range because it never lines up you know everybody's rating similarly and you get a very clearcut you're always going to get the sort of fuzzy Zone and then you're going to come up with some methodology like selecting the median um from all of those ordered item booklet locations as the place and then translating it to a scale score so um once you get those judgments the methodologies are very similar just out of

049experience I would just say that I guess you have various you have teachers who were serving on those panels that have varying experiences obviously with different um different different students from all across the state you know and and so that's supposed to yield you that that opportunity to say yes or no and often times it's I mean we've spent hours in discussion on those things and so um I don't know I guess I just feel like the the more objectivity available the better um Dr Madam president yeah I appreciate that I want to reiterate some of the things that mem mentioned earlier one of the things that we're recommending that gets added in is an additional set of training so that there's dress rehearsal opportunities for the committee members to get very comfortable and have

050feedback loops so we want to make sure as um president crey said that there's opportunities for practice and recognition that this is a process and we have built in additional um recommended um times to make sure that all of our people coming across the Commonwealth to have this conversation have the opportunity to practice before they work through these um scenaries Miss Halton thank you do Dr domaleski we're really uh appreciative of your expertise and your help both with helping understand but also serving on our Tech uh am I right you've been involved with our teac for many years or is that more recent Advent I believe I started in 2018 okay uh so uh so this I'm going back farther than that but I'm sure you're familiar that Virginia's passing cut scores on the SOS

051have been more closely um aligned or or corresponded more closely to nape basic or even below nape basic for going back really two decades am I right about that you understand that that's my recollection as well yeah um uh so this uh what the administration cause an opportunity Gap is a longstanding choice in Virginia that as as to how we were um uh assessing U what we wanted to do with our um with our test scores right or wrong and now a recommendation is being made that we make a different Choice um and you're here to help us with that and I appreciate that the the difference switching the first recommendation you have is switching from angof to ID matching and I just want to understand that process a little better um as I understand

052it the impact the first step of that is compiling these ordered item booklets and that is done by psychometricians staff the testing vendor some combination of them is that right the the doing the specific operational work is going to be the test in Vendor and their staff and and and that is done based on what you described as a a full sensus of administration of the test is that right so they'll get item calibrations or difficulty estimates um from the operational Administration which is best case because it's census test of motivated students and they um those would be used for the order item book list and and I heard you say that that's better than doing it on field tests for instance so so help me understand how I just am a little bit concerned

053about whether we've got a timing schedule here that's realistic so I'm we're uh the schedule we've been presented is would have the standard setting committees meeting in late may but many of our divisions some of our biggest divisions don't complete testing by then um and am I right that the compilation of those ordered books takes a little bit of time so how how does how is that workable yeah and and I'll and I'll say it's not immediately clear to me whether those operational calibrations come from the most recent event or if there because sometimes what we do the operational calibrations come from prior events and they're carried over so how those item which event is being used to construct those oibs um that's not a detail I have real deep insight into but I I

054can assure you Pearson has real visibility into the timeline and into the expectations and so um I'm not I'm not immediately worried that we're um proceeding in some kind of Reckless way there so just to follow up on that a little bit as you know we have changed our content standards um and have new test items uh that are being used this spring for the first time and I heard you say that it's a good thing that we're trying to I I I think the EXP we're Expediting the schedule with the goal of having the divisions know what they're up against before the the school year starts but that that's uncommon and in the past we've always done I don't always typically done standard setting in the fall so if if you had to if

055we need to delay it a little bit so that we can use the most recent test data would that make more sense than trying to rush it through Dr Coons yeah member Holton what Dr domaleski said is we are running it this year so that school divisions know what those proficiency scores will be those won't be operational in a year so we want based on feedback in the field the field has appreciated having a full year of knowledge so we are being responsive to the field and the timeline and the request of the field which Dr domaleski has worked with us closely on that I I have a number of questions for Dr domaleski because I really do benefit I think we really benefit from your expertise and I understand your schedule is short so

056to the extent I can hear from Dr domaleski and hear from others later I I would just ask for that Grace given that I understand his schedule is is tight um but so at some point we'll have to make a DEC decision whether to proceed ex proceed before we have the full results from this Spring's test Administration if we decided to push it a month later say uh for whatever reason and that means the uh School divisions don't know the final T scores until September maybe is that a a problem or will that um does that work based on your experience in other states yeah I'm not really sure how to respond I mean it is um like I say it the fact that um Virginia is announcing them in advance at all is itself

057a feature um it's very common for um for states to say that we're we're going to wait until the administration then we're going to set standards and then sometime in the summer or fall we're going to sort of retroactively report how students did that obviously has its own challenges so I think you're asking me to speculate on whether doing that sort of in the summer versus in the fall has some benefits or drawbacks and at I'm not sure that I have strong insight about it either seems preferable to me than doing it retroactively gotcha thank you okay so now once the once the ordered books have been put together the Committees come together to review them those committees uh uh the proposal we've seen um is that 30 70% of those be teachers or educating

058education experts and 30% be community members um um that's new for us here in Virginia to have U relatively new for us here in Virginia um do you have any guidance or wisdom based on your experience in other states on what the criteria should be used for uh selecting those non uh education expert members of the panels we we certainly want them to be familiar with academic expectations we want them to be familiar with the uh population the students that are assessed we want them to be familiar with the impact and use of assessment scores we want the panelists to be broadly representative of the different interest in the state so all of that goes into account When selecting a panel it's um not uncommon to bring in uh outside community members into standard setting

059panels I've seen panels have uh IH Representatives institutions of higher ed um what is I iig is uh institutions of higher education oh I'm sorry it's not uncommon to have um I've seen panels with Business Leaders and uh advocates for uh different student groups such as uh students with with special needs or language Learners so um mostly States will work to identify what interests are most prioritized to be represented and they want to make sure that everybody's got good understanding of the policy priorities and of the academic content expectations in order to do the task because as I was discussing and reviewing that sort of central ask of the panelist ultimately whatever your background is everybody's going to be doing the same thing everybody's going to be making judgments about the alignment of the plds

060to the academic standard so having the expertise to fulfill that task becomes Paramount and I heard you say so a cross-section really representative of various uh backgrounds highered business Advocates uh would you also want a geographic cross-section parents uh demographic cross-section and I want to be careful not to misspeak I'm not necessarily advocating for any of those groups I was trying to be illustrative in describing what I've seen and other standard setting events um what I would I sort of go back to the process being really important the process of uh education leaders here in Virginia working with their testing contractor to determine their priorities and making sure that they're represented Geographic proximity Is or geographic representation is almost always one of the factors that that are that states look at so I think that's

061a great example ma'am and demographic representation as well that's very often a priority as well so now you've got your panels and they're looking at this ordered test and they're um I don't understand as well as my colleague Dr Northern the difference between bookmark and ID matching but I I get the general idea that they're looking at questions looking at the plds and essentially drawing some lines am I right that the nape piece bringing the nape data in is a separate step in the process and that that would happen after the initial um ID matching process has taken place no ma'am it's it's it's woven into the process so and and I don't know that we've gotten down to the operational details of how that'll be done but an example of that could be that

062maybe round one is an independent rating round two they get the reasonable ranges we could off we we could had that at round one and so they could operate right from the start with that but it would be woven into the process and in fact it would be operationalized as guidance um to the panelist as they work through the oib so we could just say here's a lower bound and here's an upper bound um like when I did this in another state years ago um it nowadays everything's digital but at that time it was a print um oib and they just literally put like sticky notes like this is the page where the lower bound starts this is the page where the lower bound ends between these markers um we think is um where you

063should focus your scrutiny because those are the sort of reasonable ranges for us to locate our standards so it gets woven into the process that's helpful to understand and just uh so when you say reasonable ranges basically the panel would be told well if you were trying to align with Nate proficiency you would want to draw the cut score between this and that Mark this and that percentage essentially um um and that there would be a range given for the difference between fail and pass and between Pass and Pass Advance right all the panelists would see is they're going through the oib and they say here's where the lower bound starts now that's where the flashlight shining this is where I'm going to um focus my scrutiny and I'm going to do that as I

064go through it until I reach the upper bound um so it just becomes it just literally emphasizes a Zone in the oib where they're operating they don't really have to think about um the impact of what they're doing and the percentages because again that's really not their purview their purview is making good decisions about aligning items to academic expectations but but they're going to be encouraged to aim for cut scores that are within those reasonable zones that have been determined by some combination of the tack and the administration in giving them guidance they're going to be invited to focus their scrutiny in those ranges I I think there's some confusion here and correct me if I'm wrong wrong they are looking at the question they are these panelists are looking at the questions as to

065how difficult or how easy they are not assigning a cut score for the end of the process so assigning what you're saying as they are doing the cut score work as assigning a total end you know whether they are pass pass advance or fail that is not what's happening at that moment they're deciding which questions fit into which category is that correct okay so thank you for thank you for thank you for that clarifying but but regardless they're being told at that later rounds anyway if not the first round they're being told that if you um you're being invited and encouraged to align to the nape scores uh and such that if these questions uh this many questions should be in this category versus this category if you want to align to the nap's course

066is that right right I mean there's um no no desire to be anything but transparent with the panelists and we say these reasonable ranges were constructed for this reason and um to satisfy this policy uh priority and so they they would know how they've been constructed and and again I don't want to misspeak because we haven't gotten down to A fine grain operational level and all this and so um I've seen when reasonable ranges are used in standard setting the kind of guidance that can uh accompany that changes or can can be modified depending on the interests of the department so one example of that could be we invite you to focus your scrutiny in this range um but you can go out of the range if you can make a compelling rationale for that

067um in other cases operating outside the range um is is more discouraged so that that level of detail I I don't think it's been worked out for this particular methodology yet so that's helpful and I'm almost done I know there are others the uh U the the uh but with that advice and invitation to align with Nate scores essentially we're um working to ensure that the ultimate outcome is that our uh our scores next we'll know we'll know ahead of time that our scores the foll year they're going to align much more closely to the nape uh cut scores because we have defined that into the process we've built that into the process so it won't be any surprise next year when our Pass rates drop dramatically because our current Pass rates are built around

068Nate basic and now if this process goes forward they're going to be around aligned around Nate proficient and so you know pass scores barring students um getting radically smar over the course of one year our cut scores are going to drop from the 70 percentages down to the 40 percentages percentil am I right about that you're right that I would expect impact to be more correspondent with davee Miss mcferson so what what I want to know is based upon the fact that we are switching and we're raising it so that we can have a more rigorous process what is the expectation of the length of time that the school systems and divisions will need for students to achieve the the goal that we're looking for so obviously we're making it more rigorous we're going to

069implement the testing what has been historically the length of time that school systems have needed in order to see the Improvement to get to the point where we are in in line with the N process I think that's a great question and I just want to be honest with you I just do not have an evidence-based way to answer that question um I I don't know um that I could fairly predict that Dr Ken is there a state or a couple of states that you can and you don't have to have have the answer now that you can point out to us that move from a system similar to ours with scores similar to where we are that moved into the ID matching that we could look at to see how long the performance took

070and maybe some of the operational things that were put in place to help the students in the school system that's Dr g i I do want to mention Chris said this before but we are doing a whole year Gap so that we have that transition time that is not common among my peers in other states is that we are giving per the ask of our school divisions that it is noticed in the fall and then they have a whole year before those new cut scores become operational so we we are spending that extra year as a response from our school divisions and their request Dr Northern um one just sort of factual question and one that's a little Broad I think getting to um Miss mcf's question will we have released nape items to be

071able to put into the mix um as I understand the process we're we're not putting nape items um in standard setting it's all um Virginia state assessment items okay yeah um and I guess Miss mcferson I would just say to your question um Louisiana is one that's been held up across the country as a state that had um just an an Miss M Cooper can can discuss this if she'd like um but had very low literacy rates um and instituted many of the changes that we're talking about here in Virginia including um you know recommending and making sure that the early early Elementary grades are using high quality instructional materials is key all the things that we've done with with Allin and the high-intensity tutoring all these things um that states like uh Louisiana and

072Mississippi have done and if you looked at the recent nape uh Louisiana is one of the few states that actually held pretty firm um and made made some some made some gains and Mississippi has been held up for years and they were really low kind of like we are now um and they made some pretty um pretty substantive um changes to their accountability system and to their state policies and are seeing in my opinion the fruit of those labors so I can only hope um that we're going to see uh similar changes uh as to Louisiana and Mississippi given that we've taken a couple Pages uh from their playbook thank you and I know Dr domaleski has uh another presentation that he has to do at this time so we wanted to just thank you

073so much for um coming today uh for the time that you have given us we are always grateful uh for your presence here and your continued advice and counsel so thank you you thank you president Cy thank you members of the board thank you um we do need to take a five minute break given the technology um piece here to switch to the next presentation um so quick five minute break thank you e e e okay thank you ladies and gentlemen we are back um and we are on to Item B uh in our National presentation Series this is um it's actually My Hope here that some of the questions that are raised about uh Proficiency in our cut scores can be discussed and answered by the two experts who are following um here and

074so I would like to begin just by saying I'll go ahead and do all of my introductions Jim if you want to come out too um Jim Cowen is the executive director of the collaborative for Student Success a nonprofit advocacy organization defending policies strategies and initiatives that accelerate learning and improve curriculum choices so that K all kids are better prepared for college or career Jim is a proud formal Naval Naval former naval officer and toned his Communications and public relations experience as senior vice president at ogal public relations worldwide he is a graduate of Marquette University and holds a master's degree in public policy from Georgetown University he lives in the Commonwealth in Alexandria with his wife and two daughters our other presenter here today um is lesie moldun since 2019 Lesley has served as

075the executive director of the national assessment governing board an independent nonpartisan board created by Congress to set policy for the nation's report card Nate as executive director she leads uh implementation of the governing board's strategic policy priorities manages its staff of 12 and oversees an annual budget of $7.8 million Le joined the governing board with the charge to enhance nap's value to the nation as a catalyst for Action by policy makers and practitioners to improve student learning and to innovate nape and maintain its relevance in a changing education landscape during her tenure the go the governing board achieved bipartisan consensus on the first major updates in more than a decade to the na reading mathematics and science assessments launched a new strategic Vision to chart the future of nape in an era of rapidly changing

076technology and societal change as well and released four iterations of the nation's report card in 2022 and 2023 which served as the nation's most authoritative and widely covered study on the impacts of the pandemic on student academic progress malun graduated with the Bachelor of Arts in English and history from the University of Maryland and a master of public policy from Georgetown University she is is a New Jersey native and the proud parent of three Elementary School students in the DC public schools and also a former swimmer thank you all for being here today um as we heard from M earlier we've lowered our proficiency expectations in reporting student performance and this has created an honesty Gap here in Virginia so lesie you have worked on nagby and we see the dramatic decline of the many

077states and their performance I am happy to turn over uh the presentation to you thank you so much and um good afternoon I'm honored to be here and appreciate the opportunity to meet with all of you um okay but now my slides have disappeared hold on there we go um so as as you said I'm the executive director of the national assessment governing board the board was created by Congress as an independent nonpartisan entity to oversee and set policy for the nation's report card and it is comprised of uh state and local education officials teachers principls testing and curriculum experts and representatives of the general public including parents and Business Leaders so I'm here uh and really pleased to be here to share some of what we know about Virginia's educational trajectory relative to the

078nation over the 35 years in which the nation's report card has been producing data for all 50 states so before I dive into the data um I just want to make sure to share some important background information about about the national assessment for those who may not be familiar uh the nation's report card or nape is the only nationally representative assessment of student learning in K12 education it assesses uh fourth and eighth grade reading and Mathematics for the nation for 53 States and jurisdictions as well as 25 large Urban school districts every two years and it allows us to understand how school systems are serving students across States and over time we also provide National level results in other subjects such as science US history and Civics uh which is really critical because we know

079those content areas are also crucial to students uh Mastery of you know core academic knowledge nape is considered the gold standard in assessment with trend lines going back decades which allows us to see how our us students today are doing relative to their peers over time and how different groups of students are making progress over time as you know uh nape has been used and can be used as a catalyst for change it is intended to help shine a light on how students are performing and allow state and local leaders to use that information to inform policy decisions thus today's conversation I know there's a lot of discussion in Virginia and nationally around what does it mean to be Nate proficient and so I'd like to share some examples before I dive into into results

080yes do we have these slides in our book I'm not finding them I'm looking for them right now and do we have them somewhere I can see them there but not that well we don't we do not have them okay I'd be happy to make sure that can you forward them or do you want would you like me to forward them okay wonderful thank you thank you um these are uh just some examples and uh of what it means to be at each of the npe achievement levels so this is mathematics and then I'll talk about a talk about reading um these expectations are anchored in the Knowledge and Skills that we know students need um as determined by panels of experts and practitioners um the nape achievement levels I think it's important to note

081date back to the early 1990s when the Nate program um when the the national assessment governing board was first established and when nape first began assessing um and Reporting data at the state level until that time it had been only really a national assessment and the goal at the time was to make sure that we were reporting results not only as arbitrary numbers on a scale but Anchored In what students should know and be able to do to be successful in their Futures and so the governing board establishes these levels um they have remained very consistent over time which is a very important feature of nape as state assessments change you know regularly across the country nape is a sort of more of a constant to allow you to Benchmark um your progress relative to

082other states so um it is as you know Nate proficient and state proficiency are not always the same um Nate proficient means that students have shown uh competency over challenging material including subject matter knowledge and the ability to apply that knowledge so this slide has some examples um for both fourth and eth grade mathematics of what a student performing at nape the nape basic level might be expected to demonstrate relative to nape proficient and relative to nape Advanced so here in fourth grade mathematics students at nape proficient should be able to add and subtract multi-digit whole numbers fractions and decimals in both single and multi-step problems the npe basic example is you can see a simpler item this is having uh we expect students at this level to be able to demonstrate that they can

083locate numbers on a number line um in eth grade mathematics you'll see students at nape proficient should be able to apply the Pythagorean theorem to solve equations whereas students at nape Advanced which is a really um sort of trick uh more certainly more more advanced and more rigorous level we would expect them to be able to um look at that figure and sort of restructure it and apply their knowledge of formula to um a formula for finding the area of a figure to identify or to calculate the size of that blue um figure there so you can see sort of the progression um and the scaffolding of the complexity of student knowledge and skill in these examples so now I want to uh show you I'll do one slide for fourth grade reading and one

084slide for fourth grade uh excuse me eighth grade reading so here and these are actual um actual items from the nape assessment so I can tell you how uh students performed how many students got this item correct in Virginia relative to the nation so this is an example um based on the text the tale of despero by Kate D CIO um at the Nate basic level we'd expect students to be able to recognize the reason for a character's action and in 2024 uh 70% of public students nationally answered the question correctly in Virginia 75% did at the Nate proficient level we'd expect a fourth grader to recognize an inference about a main idea in a story so rather than just you know locating something this is more about um sort of interpretation comprehension uh in

08524 54% of public students nationally answered this correctly and 55% in Virginia so that's not a statistically significant difference just approximately the same at the nape Advance level we would expect a fourth grader to be able to select uh text from the story that supports an idea about a character's feelings and here 22% of public students Nationwide answered it correctly and in Virginia 25% did so let's go to Nate Prof uh excuse me let's go to eighth grade reading so there were this was an item where there were two passages so eth grade students we would expect them to be able to read two different reading passages and be able to answer questions about them um and the in this case students scoring at the nape basic level in eth grade um we would expect

086them to be able to identify the main character's motive correctly 65% of eighth graders nationally answered this correctly and 69% of Virginia students did for Nate proficient uh we would expect that students would be able to answer question about how the main character's statement applies to another character and to be able to fully you know explain themselves here so 28% of Virginia students answered that correctly and 32% of students nationally did and then finally for nape Advanced um this is a question asking students to provide their opinion on the effectiveness of comparisons in the poem and it is possible to get the question to answer the question um in either direction and and as long as a student justifies their argument well they would get full credit 9% of students answered this question correctly nationally

087and 8% in Virginia so you can see that there's you know overall in those examples the scaffolding is sort of the first point I want to make the the level of complexity increases and the second point is that Virginia students scored about at answer those items correctly at about the same rate as students nationally so you know State um standards as you know well vary um and that is by Design and by intention that is congress's goal right with each state having its own State assessment system but nape allows us to compare the relative rigor of States uh where states have set their proficiency bar so this chart I I realize it's a little bit hard to read but it is available in a report that we um hopefully you have but if not we'll

088make sure that you have a link to it and here um my colleagues at the national Center for Education statistics uh who which is the federal U Federal app ation statistics body conduct a study every 2 years following the nape assessment Administration to basically put States proficiency levels on their own assessments onto the same scale as the nape assessment so that we can compare um in each of the four grades in subjects that Nate possesses where States fall relative to each other this is not about State performance on nape this is about the rigor of the performance standards and this first slide here is fourth grade method matics so you can see that Virginia's proficiency level in fourth grade mathematics is set within the range of nape basic and that is true of um many

089states right Most states are falling in this range between npe basic and Nate proficient in fourth grade mathematics Virginia is also however the lowest on this scale for fourth grade mathematics um Puerto Rico is far lower uh down on the the very far left of the screen um well below Nate basic and then Virginia is the the second lowest there are states that have established their cut scores at at or above the Nate proficient level and you can see them on the right side of the screen um Illinois Maryland Alaska Colorado North Dakota DC and Rhode Island are just above that Mark and I will show you this for each for fourth grade math fourth grade reading and eth grade reading because of Virginia's end of course exams in eth grade um in end of

090course exams in mathematics we can't compare eth grade nape cut scores to the eth grade Virginia uh end of course exam so this is the slide for reading and here you'll see that Virginia's um cut score for the state Assessments in fourth grade reading maps to um below the nape basic level and here we um we see that Virginia is the lowest in the nation uh in terms of where it sets its cut score and again at the on the the right side of the screen you can see that Mass Rhode Island and Oklahoma had set their cut scores at above the Nate proficient level and the remainder of the states are falling um sort of in the middle there with the exception of Iowa which is also below the Nate basic level and here

091we are on eighth grade reading um and again here Virginia's standard for proficiency is below the Nate basic level and um joined Again by Iowa as the only other state in that range with Kansas Illinois Rhode Island and Tennessee uh have set their cut scores above the Nate proficient level so with that context on the proficiency standards I want to now turn to data and actual data on student performance um and I'm going to start by looking at average scores for students in Virginia relative to average scores for students nationally uh and then I want to break that down for you so you can look at your high performing students versus your low performing students this is a a very important Trend that we've seen on nape over the last decade and one that we

092strongly encourage states to investigate as well so this is um student performance over time uh going all the way back to 1992 we did do a couple of State assessments um uh one in 92 one in 2000 and then regularly every um two years starting uh in 2003 so um you know back in 1992 you know Virginia was scoring just above the national average in fourth grade mathematics we saw tremendous growth for both the nation and Virginia over the next several decades and uh then started to see a bit of stagnation nationally at about 2015 we see here that Virginia's trend line started to grow the the gap between the nation and Virginia grew so Virginia was outperforming um the national public school average in fourth grade mathematics for um much of that time period

093from 2003 to 2017 uh excuse me 2019 and then then we see a a significant decline for Virginia students uh such that today Virginia students are performing at the same level as the national average rather than having this advantage over um peer States so that's the average score and then I want to show um trends for students who are at the higher and lower performing ends we Define higher performing as the 75th percentile uh on nape and we Define lower performing as the 25th percentile on ape so here again you'll see different um different rates of change and different directions of change for higher performing and lower performing students um we see a steeper decline for lower performing students in the last several years um and than we do for higher performing students and that

094is true nationally and in Virginia and again we see the gap between the national public average and the Virginia average has shrunk so that students both at the high and the lower end of the distrib are scoring at the same place in Virginia as nationally okay I'm going to try to move a little bit faster because I but I wanted to provide a little bit of context on how to read those slides so now this is e8th grade mathematics and you're going to see a very a very a similar story here to fourth grade although I would say that the Virginia still maintains an advantage over the nation in terms of eighth grade mathematics scores um but similar to mathematics we saw really significant um steady increases over several decades uh and then some stagnation

095in the middle of the last decade about um from 2013 to 2017 a little bit of a bump and then a steady decline since 2017 um and that is true both nationally and in Virginia for your higher performing lower performing Trends we see an even Starker decline here for lower performing students than we did in fourth grade mathematics and this is a national Trend that we're seeing it's a national pattern that's deeply uh concerning right our lower performing students have been losing ground for the last decade and it is uh predates the pandemic uh substantially reading um so this is uh fourth grade reading the national story in fourth grade reading is one of sort of steady or small but steady increases for a time and uh and then stagnation for at least a decade

096Virginia's story is uh was a little bit better for a time you were making steady gains um had a significant advantage over the nation and then we start to see declines in about 2017 in student performance and at this point uh Virginia's Advantage relative to the nation has has erased um and students are scoring at the same average level as students nationally and again as in eth grade mathematics this is largely driven by the students of the lower um the lower performing students in the 25th percentile and Below all right eighth grade reading my final uh final one at uh looking at the data in this way eighth grade reading is um nationally has been a real concern since um about the 2015 to 2017 range we've seen steady declines overall and we see the

097same in Virginia um and again as I've mentioned in the other subjects we see that Virginia's Advantage has has um eroded and and been erased and again driven by lower performing students overall our our higher performing students are showing some declines but not nearly as steep as for lower performing children and if you think back to the examples I shared about what it means to be scoring at the npe proficient versus the npe basic level you know we're seeing the lower performing kids are well into the uh below the nape basic performance level so they are really struggling with some of those foundational um Knowledge and Skills in reading comprehension all right so now I want to look at the data in terms of performance levels um and here you will see that um I'm

098so sorry yes this is the right side okay fourth grade mathematics so now we're looking the green bars are the percentage of students scoring at or above Nate basic and the blue bars are students scoring at or above Nate proficient and we're comparing um the darker bar is Virginia and the lighter bar is the nation so you can compare Virginia to the National Trend and again you are seeing that we have fewer students performing at or above Nate basic or at or above Nate proficient than in the middle of the last decade um the pandemic certainly contributed to this but some of these declines were starting prior to co ask just interrupt am I right that the decline started pre pandemic on reading but not math um the declines in in scale scores started before

099the pandemic as well so this is um let me go back to mathematics reading yes but math in mathematics it it looks I want to make sure I'm looking at this from the right direction so 2017 is the high point here on for Virginia in 8th grade mathematics on nape um in nationally it looks like the high point was 2013 so slightly different Trend but still pre pandemic and then this is fourth grade mathematics 2019 okay sorry y yeah so 2017 is the high point in both in the nation and 2019 in Virginia no I'm sorry 2017 is the high point for Virginia and 2013 is a high point for the nation yes meaning the drop started in 2019 no mean well meaning the Virginia the drop was after the 2017 Administration so it could

100have been 18 we just didn't assess yeah all right okay let me find my slide here okay um so now again we're looking at this in terms of the percentage reaching those different thresholds the different performance levels um now I want to look at eighth grade mathematics and we see that you know as of last year 29% of Virginia 8th graders were scoring at the Nate proficient level which is um just above the national average um or the national level excuse me I should say and in um we have 63% of students scoring at or above the nape basic level and here you'll see that sort of plateau um in from 2013 to about 2017 to 2019 there's a there's starts to be a slight decline in 2019 um with the number of students who

101are reaching that threshold of Nate basic all right I think the picture is Starker in Reading than it is in mathematics you see that decline starting earlier um for Virginia students uh reaching the npe basic level versus the npe proficient level starts to decline in 2017 and then here in eighth grade reading again we start to see these declines um there's stagnation of the percentage of students who are reaching those thresholds um and then uh the declines start to uh turn down in 2017 okay I'm almost done with data hopefully you love data because I love data um so now we introduced something new on the Nations report card this year and we think it's um a really important uh shift it's slightly subtle but we know you know as um more and more students

102have been eligible for free and reduced lunch um through the community eligibility um standards our ability to report ape results by students economic disadvantage it's gotten a little less um nuanced because so many more students are um could fall into to the category of economically disadvantaged so we have created a new socioeconomic status index measure in um that takes into account other factors besides students family income it includes their um parental education books in the home um the the community's economic disadvantage and then the students eligibility for uh School uh free free and reduce lunch and I think the main thing I want you to take away from this is that both in Virginia and nationally are students of higher socioeconomic status are clearing the bar of Nate proficient and students who are in the

103middle and lower socioeconomic categories are you know um failing to achieve that standard and you see this sort of as you would unfortunately expect um an association between students uh socioeconomic status and their achievement level um so that is I think something that you miss if you're just looking at their their economic Advantage so this is a little bit more sophisticated a little bit more um um insightful and hopefully hopefully a better way to communicate about um the relationship between students um school and background experiences and how that contributes to their um their achievement I want to show the same um so this was fourth grade math this is fourth grade reading and here um we see a slightly different picture right High sees students are just at the Nate proficient level or just below

104whereas um students who are low sces or middle sces are barely getting to the npe basic level so um reading seems to be uh a higher little bit of a larger challenge for students to get to and we see similar patterns for Virginia and the nation although it does look like there's a bit of a difference between um students at the middle and lower SCS levels in Virginia relative to their peers nationally um and then I just want to talk about who these lower performing students are um we get this question a lot and I think it's really important to understand that students who are scoring below the 25th percentile in on nape in Virginia and in uh in the nation are a um a very diverse group of students they come from all racial

105and ethnic backgrounds they have parents who went to college many of them um to be sure students from traditionally um marginalized communities students from underresourced communities are over represented in that 25th percentile of achievers but they are not um they are not homogeneous right there is a lot of racial and ethnic diversity diversity in terms of their socioeconomic status um so I have some some numbers that I want to attach to these you can see here in fourth grade um fourth grade mathematics that students in this group um below the 25th percentile 68% are economically disadvantaged and 32% are not 31% are identified as students with disabilities 69% are not 35% are English Learners 65% are not um 49% are from the middle socioeconomic category and 37% are in the low and among racial and

106ethnic groups we see um 35% are black students 32% are Hispanic and 24% are white so um hopefully that gives you a little bit of a flavor of just heterogeneity here in eth grade um this is eth grade reading and I show this just to show you that it is largely a similar picture regardless of the subject in grade in eth grade we um do get information from students about the level of Education their parents attained we do not ask fourth graders that because they can't tell us reliably but eighth graders typically can um and in the here on the bottom right is the pie chart for Education uh level and I'll I want to just highlight that um 5% did not graduate their parents did not graduate high school 21% had high school graduates

107as parents 39% had parents who did graduate from college so this is you know a population of students who have um a lot of diversity in terms of their background characteristics okay finally um a couple things to keep in mind um nape has no Stakes for uh schools or students but the critical information it provides does help you help you get more context about how students are doing in your state relative to others and we are glad that states like Virginia are using this data to spark important conversations and to take those lessons and figure out what does it mean in the context of your own education system um I want to highlight very very briefly um some research that shows that nape is not um the the nape scores do matter right assessment scores

108are associated with students outcomes in life um there's research from Stanford um Eric hesek at Stanford who estimates that decreases in nape scores equate to lower lifetime earnings for individuals and lower GDP for States um because it affects the the skill level of the workforce so he has estimated that the um losses we saw in eighth grade mathematics during the pandemic equate to about 31 trillion in future losses to the nation Tom Kaine and uh at Harvard and Doug Stager at Dartmouth have shown that Rising achievement on eighth grade nape is associated with higher incomes and better educational attainment as well as lower rates of incarceration and um teen pregnancy so these are not necessarily like high nape scores cause you to have a better life but they higher test scores are certainly associated with

109better outcomes economically and for individuals and I would be remiss if I did not share that more is coming from the nation's report card these are National assessment results but we think they are really important to add to the picture of how our students are doing this summer we will be releasing the nation's report card for 12th grade math and reading um this is the first look at our um students at the end of their K12 experience since the pandemic from the Nations report card and in the fall we will have 8th grade science scores which um we know you know science and is so critical for students um educational Futures as well and we want to make sure that we're adding to that picture um students this time next year in 2026 will'll be

110taking the next Nations report card in reading and Mathematics and by the end of the 20 6 calendar year we will have updated data so you can see where um Virginia students are scoring in about a year and a half so thank you so much for the opportunity to be with you today and I will hand it over to Jim thank you so much Jim welcome thank you all um this is why Leslie should go first um my name is Jim Cowen I'm the executive director for the collaborate for Student Success I appreciate the opportunity to be here today I've I have my my colleague uh Adam ezring here today to keep me honest and be ready at spreadsheets if if questions come up so we can we can handle those for you um for

111a little over 10 years now my organization has been focused primarily on holding students to high expectations to helping students Educators meet that bar and uh following a a strong system of measurement to make the adjustments that need to be done it's a very simplistic way to put it but in essence we are we are a Communications organization and that is the way we have functioned I'm I'm going to talk to you about something called the honesty Gap that has um taken a little bit of root and has become kind of a coined phrase that we're we're happy that it's there we'd like to talk a little bit more about it nationally and how it looks in Virginia for you today first off like I said the h Gap is a Communications exercise we're not

112trying to make this sound like it's more than than what it is this is not a highly uh analytical tool this is a look at a a simple principle which is parents deserve the truth and so we have um worked closely with with Leslie and the team at at nape because we do believe that that nape is a strong bar and a strong um system of measuring progress across across the country so the idea of of the honesty Gap is based on kind of a line of parity if you will where we have how States declare proficiency compared to how npe is showing proficiency of course we would hope that states would be as proficient or even more proficient than than nape um unfortunately lot of times it's it's lower than nape for different reasons

113so let's take a look at the first roll out we did on the honesty Gap the first one first honesty Gap rolled out in 2015 it was based on 2013 data from achieve at at the time what we saw were that there were 14 States that had gaps between nape proficiency and state proficiency that were 40 points or higher Virginia was not one of these groups they had the honesty gap of 27 points minus 27 points so they were not perfect but they were not in the worst performers this is in fourth fourth grade reading in 8th grade math in 2013 again we had 12 States that had gaps that were 35 points or larger that came out Virginia also not in this group they had an Hy gap of 29 Points again not the

114best but definitely not in in the worst group Ty that we saw so let's Le forward to 2024 our most recent honesty Gap we now we we said initially that we had 14 states with gaps that were 40 points or higher in 2024 we only had two states that had 40 points or higher and it was Iowa and Virginia Virginia had an anony Gap minus 42 points this is in back and breing and in 8th grade math initially we said we had 12 state with the biggest gaps of 35 points or more and in the most current one we had three states that had 30 points or higher and it was Iowa Missouri and and Virginia with with minus so we started looking at what what what happened in this time frame to move us

115to this to this point and I think we can say a lot happened um there was a pandemic schools shuddered kids went home kids learned remotely um kids came back uh we started to see drops on nape scores we started to see drops on state summi of tests we saw drops on formative tests we saw um grades coming home from school including from my own two daughters that were A's and I'm very happy that you had our friends from learning heroes in here not too long ago to testify because what they showed was pretty startling which is at 8 10 parents think their kids on grade level but only four out of 10 are actually proficient based on na scores that's important to us and it's something that personally I I I feel and if

116you have kids you you feel that as that as well so we recognize that and what that has created is is this difficulty in staying true to the honesty Gap it has created a level of of of opportunity to lower cut scores to to make yourself to mask differences and make yourself look more proficient than you actually are it's becoming harder and harder to hold that that higher bar a couple weeks ago I put out a piece in Forbes that looked at Virginia in particular because I was curious like how did Virginia get next to Iowa on this and so I began to look at at what was going on in in in Virginia I see that things happened in 2019 I see that there were there were changes in accreditation for schools I see

117that there were lowering of cut scores uh before the pandemic and the pandemic hit and it was literally a one two three punch on the state of Virginia which caused I believe the state to drop to to where it is today I also noticed that you're not running from it and that was the whole theme of the article and I thought that that is a it is an example for the rest of of the country I think to follow that the only way that you're going to deal with this is to not shy away from it and what I saw was very public and open uh acknowledgement of the issue um a lot of movement around uh accountability movements working towards chronic absenteeism dealing with a lot of the problems that we know can make

118a difference and so the point of my article was to applaud that now again this is in Communications exercise our our job is to take the extremely sophisticated work that is done at nape with those slides and turn it into something that average parents can understand and I think they understand this you don't run from a problem you deal with it and I think that's what Virginia is doing and so that that was what my uh my intention was here today that was much easier than what Leslie went through I'll stop there are there questions comments questions yes ma'am suton so uh thank you both for your presentation very very useful I know you both have a lot on your plates right now especially we're we're thinking of our our our friends at u u

119u in your world Miss mun and and we're very very grateful that you making time to be with us uh today very helpful um um Mr an honesty Gap question and then I have a question for Miss malun the honesty Gap question as as you've labeled it I hear you that it's really primarily a Communications tool and I appreciate that uh I think I am right that Virginia I I see you show that Virginia was not as bad by your honesty Gap metric uh um in 2013 and as it is as you would characterize it being now but we've been uh we've been aligned our our cut scores have been aligned much closer to Nate basic than to Nate proficient for two decades am I right about that my understanding is is correct so they

120were aligned there even in the years when we were doing High when we were having high performance outcomes on nape and on our state so so uh so the honesty Gap issue that you rais is primarily about Communications U it is a it is a simple differential between what nape shows as proficient and what a state declares as proficient and above yeah and the the changes that were made in 2019 were really fairly modest the the the longstanding Virginia choice to align our cut scores our Pass rates to Nate basic is what's responsible for what you characterize as the primarily responsible for what you characterize as the honesty Gap we we haven't done like a differential analysis to determine the exact cause of why the dropped but we're looking at things that would suggest reasons

121why they they would fall and so when least when putting this other article together I I I did look at like what what actions were taken over and the changes I'll tell you because I was here part of it the changes in 2019 were very minor um so I posit that there's other things going on uh in terms of student performance and as we all know what really matters for student performance is teacher quality curriculum quality standards um and parent engagement and Communications parent Communications can be a tool for parent engagement but really the if if we're going to help our students actually do better we've got to address all of those uh what's going on in the classroom and we're working on that absolutely I I am just happy to show how this pairs

122across you know the national landscape and snapshot Y and then my question for Miss malun and then I'll um yield thank you madam president is um um we we were shown studies back when we started this process um um I think it was fall of 23 or maybe even sorry would you mind speaking closer to the microphone I'm just having sorry now it's too loud uh but we were shown studies uh back uh when we started this process uh Matt chingos who does I think great work at Urban Institute I know you know and I believe there's also o some folks at education next uh Daniel Hamlin and Paul Peterson who've done work showing well as you know a number have St a number of states over the last decade Plus have realigned their cut

123scores many of them and and and it shows in Mr C's charts too were aligned closer to Nate basic as we are and have made big big movements realigning their cut scores closer to NP proficient over the last couple of decades and the analyses we were shown showed that that alone did not move student outcomes in other words it's a Communications tool but just realigning to Nate proficiency instead of to Nate basic doesn't in and of itself makes students do better am I right or am I understanding that research right or do you have contrary research well I'll say one caveat just the nation's report card is better at telling us what's going on than why um that is I think just important to to um say second I would say that from the I

124think Jim you know and the work that they're doing on the honesty Gap that I would argue is is very much about communicating and helping get the attention of parents and community members and um and others who have a stake in the student outcomes I think for nape the achievement levels are and for any state assessment achievement levels are more than a Communications tool they're intended to articulate the um what students should be aiming for what teachers should be aiming for and the what set of skills the state or in the case of nape nape say students need to be able to demonstrate to be successful at the next level whatever that you know whatever that is and I think that if um setting high standards for kids whatever that means is a necessary but

125not sufficient step to help them be to help the system right help teachers help parents help schools uh get students to that level right all students to that level well necessary but not sufficient you mean if you have great uh uh uh great teachers and great curriculum and allocating resources to your uh low low U economic disadvantaged students etc etc you can get great results even if you have cut scores aligned with Nate basic instead of proficiency am I wrong I think that in that case you might not be telling parents whether students are achieving at the level that they need to be successful you might have the communications issue but it in other words raising the uh the cut scores alone does not move nape scores at all I I don't think we have

126conclusive research that shows just changing cut scores but I think that's the same as just raising your academic stand standards just changing changing academic standards in the state we know is not enough it it's down to implementation and alignment of the systems but what we have is research to the contrary that raising cut scores alone is not going to make students do better on nape or in any other test uh at least that's the research I've shown and so if you've seen some different that's what I'd love to know but so I think I'm under so then I'm trying to understand what is the theory of change for moving our cut scores from a land with Nate basic to Nate proficient and I think it's really primarily as a Communications tool which is back to

127Mr CO's purpose which can help with parent engagement is that the is that the theory of change um well I certainly can't speak to the theory of change for for a state I will speak from the the Nate perspective and why we set the achievement levels at the level that we have for the last 30 plus years and that is to articulate what Knowledge and Skills students need to have mastered in order to be prepared and so that is and again as I said before I think it is about communicating and that's why I showed sample items right there's a whole lot more there's a lot of research that goes on behind that to articulate what is the test show students know and can do um so it's it is about Communications but it is

128also about sending important signals to Educators to students to families about the body of knowledge students need to attain so I would I would argue that's more than a Communications it's about um it's a a very clear articulation of the values and the goals towards which the system should be working right so just for one more that comment that I I I think the choice Virginia has made in the past whether right or wrong was not a choice that we didn't want our students mastering challenging material but rather that we were aligning our cut scores to what we understood to be the minimum that a student needed to move to the next level whereas the choice to align to Nate proficient I'm hearing you say Nate proficient is more has a student mastered challenging material

129for that grade level and so that's really the difference between basic and proficiency the minimum necessary versus the challenging M mastering challenging material and and States you know can choose to align with those definitions or not and I I think our my colleagues are ready for us to move uh to to aligning to the N definitions but it's all definitional really um okay one for for just for just one second um true or untrue did Dr James Lane when he was superintendent um recommend to the board certain cut scores that the board decided to go below his recommendation true or untrue we went slightly Below on one or two tests it was not a reduction from a prior test because it was a new test but it was one or two very minor differences it's

130nothing comparable to what we're talking about here where we're where Virginia has made a choice to be align to Nate basic not to Nate proficient thank you raising a valid Choice it may be the wrong choice and this Bo it may not be the choice this board wants to or should make now but it was it was a it's a longstanding thing not none of that affected 2019 npe results or the 2022 n results I believe what Miss malun was talking about was the fact that when we raise cut scores we raise expectations and we believe that both our Educators and our students are seeking the challenges of raising those expectations and I for one believe in both our Educator's ability to do that and our students ability to meet those challenges number one um

131so Mr Cowan can you answer a question for me are you guys doing some work in the area of this this this big pervasive question which is how many parents are okay with their child or their children simply being at basic we have not done research on that yet I know that there are some some of that is in in progress right now I'd like to go back to one of the points that were made earlier about the difference between basic and proficient too as a as a parent I go back to the slide that that that Leslie showed where you show like what does that mean to be basic what does that mean to be proficient those items are important to know I'm throwing out an example but if a if it's considered basic

132to be able to understand a point line System point line coordinate system at a certain grade and and suddenly we're changing that I'd like to know that as a parent that we have now altered what we're considering to be proficient because I think I think that's critical for a parent to have that understanding thank you Dr Northern so I'm going to clear the deck here because I'm the researcher on the panel um and there's been plenty of research in terms of what Nate proficient means in terms of long-term outcomes which is all I frankly care about like I don't care if you know if if it's only going to correlate to you know your next year's GPA that's one thing but I want to know that it correlates to real long-term outcomes that matter for

133kids in real life and here's the research a standard deviation on nape Improvement in eth grade math math achievement on nape so in other words you've you've you've you've you're at nape proficient right and okay like now now what does that mean in terms of long-term earnings in adult in earned income 1.6% decrease in the present value of Lifetime earnings but if you're actually Nate proficient you have about a $19,400 per student increase in earnings if you're Nate proficient that means that Nate proficient actually matters to your longtime earnings number two students born in states with larger increases in in in math proficiency at 8th grade saw higher rates of high school graduation and increased College enrollment this is not a randomized experiment study we don't have those but there are strong correlational studies that

134have been year after year after year employment so okay um this is this is really interesting to me so kids who are proficient in in higher math scores than a proficient they're less likely to be unemployed they are less likely to um be teen girls born in States proficiency less likely to become teen mothers so they're actually social outcomes proficiency on boys you're less likely to be incarcerated at ages 18 to 24 um lower and then if you're not proficient you have lower decline in high school graduation lower declines in college enrollment more increased likelihood of being pregnant as a teenager and more increased likelihood of a arrest so long-term incop Dropout risk I could go on and on and on and on but my point is that Nate proficiency matters in terms of long-term

135outcomes for kids I know this because I study it and so this doesn't mean it's not just like blah blah blah let's just blow up our skirt and say we want to be Nate proficient it matters to kids in realterm longterm I'm repeating myself but that's where all of this should land I don't care about the politics I don't care about like oh we did this and we did this and who I mean that's called Missy nobody knows why we are in the situation we're in we just know that we're in it and we're not about pointing fingers we're about saying okay this is what Nate proficiency does for our kids and we should actually have that as our goal to do right by them M president may I ask Dr Northern a follow-up question

136Miss Holton thank you I I I totally concur with your summary of the research based on my reading of the research which I've also spent a lot of time with over the years uh my issue is whether there is any research that have so so npe scores correlate with good outcomes so scores correlate with good outcomes but having whether our so cut score align with nape cut scores or not I think there's no research the research shows there's no impact of realigning our cut scores we need our students to do well on the tests but where the line is is irrelevant that's the research that mat Chingo and all showed us but I'm just saying Miss Holton they're synonymous no they're not no no no no if our so cut score was aligned to Nate

137proficient that's what we that's this is so for instance on Miss mun's charts we are at the bottom of the pack no question about that and and should we move from that absolutely we should but we are at the bottom of the pack on our what you call what they call the honesty Gap the how our cut scores align to npe but on our npe performance we are average now we want to be better than average but our performance is average so we are not bottom of the pack on our performance the research shows that performance on nape outcomes cor correlates with all those life outcomes but there's no research that I've seen that where you draw the cut scores correlates with nap outcomes which again doesn't mean we shouldn't make this change but it

138just makes the point that you know we're going to declare well essentially the change that where we're heading is we're going to declare twice as many students not passing next year as this year that doesn't mean our students are any Dumber it doesn't mean our teachers are any worse it just we have declared twice as many our failures because we've moved the line uh that doesn't mean the students next year are going to have twice as un twice as high a likelihood of failure in all those life outcomes they're still going to have the same project of Life outcomes we've just moved the measurement at and the moving the measurement alone does not produce outcomes I don't think it probably does any harm from the research I've done seen but it doesn't make any positive

139changes what makes positive changes is work like what we're doing with the literacy act the work we desperately need to continue to do on having a quality teacher in every classroom especially for our high poverty students who are not currently well being served they're being served by provisional licenses High turnover etc etc so that's what we really need to do to move the needle Dr Northern of course I understand all that it's not the point we're making we all know all we need that the answer high expectations are are part of the larger puzzle of all of this but I can guarantee you there's no state that has cut scores in the basement like ours in the basement like ours on Cut scores has low expectations for kids because low expectations are like a domino

140right like you have low expectations here you don't magically have them somewhere else and actually the the the test scores are where the rubber meets the road right where all of these things that we know come into play the disadvantage so on and so forth but at the end of the day it's the same test and it's holding all kids to the same rigor uh and and level of achievement and I actually think that if we say okay we can move the bar really really really low and still expect that our kids are going to raise up and do all these great things when we're not expecting them to do it it makes no sense it really doesn't make any sense so part of this is about Nate proficiency and the research and part of

141it is about common sense and what signal do we send to kids when we actually set the cut score so dang low and you see a slide like that and you see Virginia way at the bottom on the cut scores it's embarrassing frankly we we need to do better as a state for our kids it's a Communications tool Miss hon uh Mr Hansen do you have comments okay thank you ladies and Gentlemen We Appreciate uh your participation today your expertise um we thank you for being a part of our national expert series um you all have brought new fire to the discussion as you can see uh and we are grateful for for your um contributions here today so thank you um we are GNA go well I need you to go ahead and make

142a motion here um we are going to switch back into our um our iure cases and so what we are going to do now is I'll entertain a motion to go into close session Mr Hansen I move that the Virginia Board of Education convene in a closed meeting for the following purposes discussion or consideration by the Board of Education of information relating to the denial suspension or revocation of teacher licenses subject to the exclusion in subdivision 11 of section pursuant to section 2.2 3711 A40 of the code of Virginia and consultation with legal council regarding specific legal matters requiring the provision of legal advice pursuant to Virginia code section 2.2 3711 A8 of the code of Virginia and the legal councel to Virginia Board of Education staff whose presence will Aid in the board's consideration

143of this matter participate in the closed meeting is there a second a motion has been made and seconded all those in favor signify by saying I I any no okay motion carries the board is now an executive session all staff members and members of the public who have not been on e e e okay um the chair will entertain a motion to certify the purpose of the meeting in closed session Mr Hansen we convene today in a closed meeting pursuant to a recorded vote on the motion above and an accordance with the freedom information act section 2.2-inch that to the best of each member's knowledge only public business matters lawfully exempt from openland meeting requirements under this chapter and only such public business matters as were identified in the motion by which the closed meeting

144was convened were heard discussed or considered any member who believes there was a departure from these requirements shall so State prior to the vote indicating the substance of the departure that in his or her judgment has taken place the statement of the departure will be recorded in the minutes uh is there a second motion has been made and second this will be a roll call vote please signify by saying I miss Kilgore I Dr Northern Miss mcferson Miss Holton Mr Hansen chair votes I okay that motion carries okay um as we took up a number of cases today um I will ask for the board um to um make such motions to follow in the case uh 2024-25 the board um the chair will entertain a motion to take no adverse action move is there

145a second Dr Northern Miss Kilgore hi Mr Hansen hi Miss mcferson hi Miss Holton abstain and chair votes I uh in case 202-0324 the board uh the chair will entertain a motion to issue a written reprimand to Joshua David hearnen is there a motion move second okay it's been moved and seconded Dr Northern Miss Kilgore H Mr Hansen hi Miss mcferson hi Miss Holton I and chair votes i h case one chair will entertain a motion to take no adverse action second okay um moved and seconded do Dr Northern Miss Kilgore ah Mr Hansen Miss mcferson Miss Holton I and chair votes I in the case 2025 d05 the chair will entertain a motion to issue a license approv second okay moved and seconded Dr Northern I miss Kilgore I Mr Hansen I miss mcferson

146I miss Holton I and chair votes I case 2025-the Mr Hansen Miss mcferson Miss Holton and chair votes I okay we are done with that portion of our meeting so now we need to go back to item D or C okay um based on the fact that we have some people waiting in the waiting room um to be a part of item E I would ask that um the board board entertain this opportunity to skip over items c and d and move to item e we will go back to item C and D after this presentation uh I believe Dr Armstrong this is you no not this one okay the first part is North Dakota so um I guess um Dr Armstrong will come up later but in December board members requested National feedback on

147seat time flexibility and to that end we have asked knowledge works and a member of the Mastery based learning team with the North Dakota Department of Education to present for us today uh then we'll hear from Dr Armstrong and Dr Coons about Virginia's notice and action kit in preparation for first review tomorrow at our business meeting um so I think we need to admit yes please good afternoon thank you for being with us today so sorry that we're running a little bit off time schedule here um but are so glad that you're able to be with us um to speak to us this afternoon we are ready for you to introduce yourselves and proceed whenever you are great thank you um this is Anne ellson I'm with the Department of Public Instruction in bismar North

148Dakota I want to confirm that our slides are currently being displayed or would you like me to share my screen good they're good yeah they're up sorry this is getting really loud I'm sorry if I busted your aard drums what um scre yes that would yes thank you yes thank you I will share my screen how is that okay perfect um so happy to meet you all um Ann ellson as I said I'm the director of academic support in the North Dakota Department of Public Instruction we want to thank Dr Coons and members of the Virginia state board for inviting us here today to share with you some of our strategies Lessons Learned um and different perspectives as we've pursued personalized learning throughout the state of North Dakota um as director of academic support in

149the DPI here in bismar um we have a variety of projects and initiatives and one that I oversee is innovative education and the roll out and practices that we have within our schools and districts when the first piece of legislation was issued in 2017 to do this work we recognized that the state could not do this alone we had limited capacity to support our 148 school districts and um developed a strong partnership with knowledge Works in order to support our schools and districts so my knowledge Works partner in crime is Lori Phillips and I'll let her introduce herself thank you superintendent Coons um chair Cy and members of the State Board of Education um for having us I am a grateful partner to Ann in North Dakota and I am a senior director of teaching

150and learning um at knowledge works and lucky enough to work across the country and have been working in North Dakota alongside an and her team for this is year eight so grateful to be here thanks Lori so as we prepared for today's um session with you all we thought we would go ahead and share North Dakota's approaches to Innovative education what that's looked like throughout this journey and then also kind of in tandem between Lori and myself give examples of what this work actually looks like occurring in schools and districts across our state so we'll share some information we'll have time for questions at the end but should you have questions as we Navigate please feel free to to interrupt so as I mentioned um the partnership between knowledge works and in particular Lori Phillips

151and the Department of Public Instruction really grew out of an organic place um we recognize our schools and districts needed support that was beyond what we at the state department had capacity to provide or understand through our eight years together we support each other through thought partnership um as we write grants or develop policy or practice within our state um we really work hand inand with each other I will say that the knowledge Works team brings together a policy a practice team and also some visioning that has been a benefit to the state of North Dakota our also our assessment offices and other offices within the Department of Public Instruction have also utilized knowledge works as a sounding board for design and development we've also provided design facilitation for district level supports so schools and

152districts that have pursued Innovative education models what does that road map look like and knowledge Works has helped to guide that road map through Innovative education we do recognize that District leaders need to behave differently and receive supports differently so knowledge Works has been able to also provide additional support coaching mentoring on District leadership really I think when you get into Innovative education and schools you need an an entrepreneur of Education sometimes more than a superintendent of education so um you really have to have that entrepreneur mindset or be a superintendent and an entrepreneur at the same time so we've been able to provide our school districts with that leadership support and development there are also some logistical pieces that we'll get through such as our learning Continuum where knowledge Works has been instrumental in

153the design like reaching to other states and facilitation of bringing North Dakota Educators together we've moved through this journey and now are in a place where we have a Statewide Network where we have North Dakota Educators North Dakota leaders providing strategies to spread and scale personalized learning to each others to help lift and support so the journey with knowledge Works has been robust and I will say that North Dakota would not be where they're at without a partner like knowledge works at um at the table so now a little bit about our Statewide flexibilities in North Dakota we have the Innovative education program in place my understanding is in Virginia you all you all refer to that as your seat time flexibility North Dakota has um its innovative education so when I say that um

154that I believe is the context that you all look through it when we work with our schools and districts often they will want to find the one piece of Statute that might be in their way the one piece of law that might be their in their way and that really is not the approach that we take with our Innovative education program rather we ask for our schools and districts to design a comprehensive plan a systems level plan that looks at what's their rationale and and vision for Innovative education what does this look like at the systems level how do they know that this is what their stakeholders want want and need stakeholders include their teachers their students their Community parent members board members and so forth there is a level of board approval because um

155we know that pursuing Innovative education is a it's a tax on time in development of staff and we really need to to behave differently in schools that that are that are moving towards Innovation also the level of support provided for professional development for educators has to be articulated and how is that supported throughout the process and then we look at implementation and continuous Improvement what we've learned is that some schools that have had an Innovative waiver in place since 2017 they've iterated along the way you only know what you know as you design your system and so we've had to be responsive to that to their continuous needs on an ongoing B basis there is also an element of evaluation and sustainability where we report to our state legislature on an annual basis regarding progress

156and outcomes for the Innovative education schools as we've learned since 2017 these schools it's important that they have a community in which they can learn from one another and with one another um often it's an isolating experience if you're trying to innovate a system all on your own the other thing that North Dakota has done uniquely we believe it's uniquely is that the state has approached this work to learn from our schools and districts that have pursue Innovative education so the state is in a true partnership with these schools and districts and then we leverage that partnership to adjust our practices and inform our policy we've seen changes to our North Dakota Century code and administrative code as a direct result of practices that are happening in classrooms schools and districts that are practicing innovation

157so we're able to address those barriers that were on their original list um with more context and true examples of what it looks like so that's a little bit of ground setting for the next part of our slides on this screen you'll see our Innovative education timeline so I did begin explaining in 2017 when our partnership really started with knowledge works but prior to that in 2015 superintendent Kirsten Basler was elected to the to the superintendency of North Dakota she was um a a fresh person in that position prior to that Dr Wayne sandstead served for 28 years so Dr Lisa Coons you have a little bit of time if you want to beat way and stay instead's record 28 years as state superintendent um and so superintendent Basler stepped in with her ears wide

158open and many of our schools and districts in 2015 to 17 said here's the flexibility that we need and we in North Dakota we have everything from one room School houses to large comprehensive high schools with thousands of students and their needs were vastly different but we recognized we needed to provide some form of flexibility for that work to look different in these different environments so with um listening gathering information from our schools districts Advocates from 2015 to 17 we've then pursued this journey with knowledge works at at our right hand in this timeline that you'll see so we'll be referencing back to this slide or to especially the titles and colors on this slide as the next few um as the next section section pursues so our first major legis piece of legislation was

159Senate bill 2186 which is our Innovative education bill this was passed in 2017 and allows for that school or District to design a comprehensive plan for improvement it allowed the state superintendent to wave a laundry list of sections of Sentry code and every every other year we meet on a um on a biannual basis our legislative assembly this section of code is updated with additional areas where superintendent Basler May way may wave statute and Authority districts that do pursue the Innovative education program have one year of planning so recognizing that it takes time to bring teachers community members and so forth a minimum of one- year planning to bring those individuals together to design and then they're they receive a waiver for five years of implementation under that plan as I mentioned we have some

160schools that started this in 2017 and they're on their second set of five years for ongoing implementation I also mentioned that we collect reports on an annual basis from these schools and districts and Report regularly to our legislative assembly lur I'm going to share it with you for a little bit and then I can provide some further context on what this looks like in North Dakota absolutely so Empower Ed is a program within the bismar um City Schools or public schools um one of the largest districts in the state um the Empower Ed program started in 2020 um their mission is to empower Learners to thrive and take control of their own learning so they really started small they took advantage of the Innovation waiver um and then also the innovation academy which is in

161alignment with that waiver and it provided support for bismar leaders to come together and to be able to design and iterate with others to develop this program so out of the innovation academy they started small with a small group of kids in 2020 and that has continued to grow over time and now the Empower Ed program is at two different high schools two different campuses um it really embeds Community projects it is Competency Based um we have had the privilege of engaging with many Learners within the empowered program and the level with which the Learners can describe what they're learning can drive what that learning looks like they drive you know how they spend their time when they need to show up um and what they need to work on and what they're trying to

162accomplish and they're very articulate um about all of that so I think empowered is certainly delivering on the mission they iterate constantly um and try to meet Learners where they are it is open and accessible to all although they say maybe not for everyone you know everyone may not want to take advantage of the style of learning um but it is open to all so that's one way that I would say the Innovation waiver has taken root within an actual school and District an what would you add there yeah I would add um you're right that it began small almost like a school within a school type of structure it's important to note that the original design is in a Career Academy that does not issue its own high school diplomas so it is a

163separate um structure a CTE a career and tech ed building within the bismar public school districts where the comprehensive high schools have kids that attend there what the school um knew and noticed is that their kids were sometimes very successful in their CTE programming where maybe they may be less engaged in their content level programming back at their regular high school so the empowered program allows them to bring both of those pieces together and fold in community um work out community outreach and work outside of that school Lori did indicate that they started small um I believe the original design they're like hey can we do this with five kids and then eventually it moved up to 25 that very first year and now they have over 100 kids per semester and also have a

164second site within a comprehensive high school campus so again getting the learning to look different and really the adults in the system to behave different the instruction has to look different for students the learning has to look different for students and for that to happen adults need to understand that learning at a different level because it's not the same delivery as um high schools most of us probably had participated in I also think it's pretty cool that the kids have renamed this program they've designed the logo for this program there's a lot of local ownership for the empowered program that's generated directly from students our number one customer in public education in 2018 as we mentioned the law passed in 2017 in 2018 I was fortunate to receive a bunch of phone calls from schools

165and districts that said hey we want to do the Innovative education program Where Do We Begin and um you know that's that's really hard because where do you want to begin what are the needs of your community what um what are your stakeholder needs where are you currently at what is your design and vision for the future so other than providing deep thought questions and coaching with schools and districts the state agency didn't have capacity to figure out how do we support schools that want to Design Systems differently when our expertise was within our current Sentry code our current instructional understanding and so forth this is where we developed a partnership um a m a really multi-partner approach knowledge works is our longest standing partner that came to the table but we also need to

166shout out the bush Foundation Who provided Financial Funding behind our personalized competency based learning cohort this was a five-year commitment between North Dakota knowledge works and the bus Bush foundation on how might we provide supports to Educators across North Dakota in personalized Competency Based learning so the application process aligned very clearly with our Innovative education application under 2186 the schools and districts that were interested had to agree to multi-year Commitment for implementation as well as systems level changes so um that looked very different in our largest Comprehensive High School which has 11 to 12,000 students to some of our smallest schools that have K12 maybe 500 students in North Dakota when we roll out pilots and programs we feel it's really important for our schools to see themselves within the model schools that are there

167I would imagine in your state as well you have large schools and small schools if you only provide an opportunity to your large schools those small schools are like well that's easy for them they have coaches they have a building principal and our small schools are like my building principal's driving the bus and coaching volleyball tonight um so you have we had we intentionally provided environments in which other schools could see themselves some of our larger school districts down to smaller school districts so um this began in 2018 and lasted actually six years because of the pandemic um slowed our progress it didn't derail the progress but we had a lot of virtual meetings for a few years there so um it did slow that progress Lori I'll let you talk a little bit here

168about what the cohort looked like thanks Ann so this was really when knowledge Works began I would say to have some District level Partnerships um always in tandem with an and the department um we had districts apply we put out an RFP um I think it's important to recognize that there were five districts um who began this journey and talked about it earlier leadership development and Leadership capacity is really important so we had four districts who persisted um various size districts um West Fargo being the largest District in the cohort um they are consistently adding schools and growing so upwards of 12,000 kids Northern Cass it's important to say that they were out in front Northern Cass was probably about five years into their Journey at the point that they started um this cohort and

169due to a really strong relationship with an of the department she said this is going to be good for you you should join this um Oaks has around 550 kids they were starting at the beginning and then the Marmont wolves in the lower leftand corner had our heart from the beginning beginning that is a high school within the Department of Corrections um and that's the youth correctional center so they brought a unique perspective to this cohort as they are meeting kids' needs in all areas um not just including academics and they had a lot for the rest of us to learn from um so it's important to say when you're bringing districts together that are at all different entry points you know Northern Cass had a lot to share with the rest of the group

170but they also needed to feel that they could learn and grow and they were going to be stretched and challenged and so we needed to walk the walk and say can we personalize an experience for adults in a way that we're asking and we can model that or we want them to do that with kids in classrooms and Educators in buildings you know from the leadership level so I would say one of our biggest challenges was we were looking at the sustainability and the long game here to say we want this not to just be five to six years of really great professional learning we need them to build Collective efficacy we need them to depend on each other value each other begin to share resources and recognize that they can't do this alone and

171there is strength in numbers and so would they continue to collaborate long after this cohort ended and an and I can be really transparent to say that we were probably four years in still you know kind of scratching our heads to say West Fargo as a large District really wondered how they could learn from a small District that was Rural and vice versa they're different they have different challenges we're unique um they all feel that way and they're all correct but they can still learn from one another and so I would say one of the biggest levers um coming out of covid was we started to have the district's host site visit so each of them agreed that per year they would host two to three site visits within the cohort and they started to

172recognize as they sent educator teams to each other you know West Fargo sent a group of Middle School Educators down to Oaks and they came back and said okay we're going to take more they really didn't feel it was going to be relevant and they learned quickly that small school large school we all have kids in classrooms and we're all trying to innovate and personalize that experience and they had they they really started to recognize they could learn from each other and they didn't had to re have to recreate the wheel all the time and so that was huge um we also this cohort was about transforming systems and so systems change second order change is something that's not intuitive I think you know many of the districts wanted to close their doors and they

173really wanted to focus on building up their own skills and creating their own capacity and said wait a minute we don't have time right now to worry about what anyone else is in the state is doing so it took quite a while for them to recognize that they needed to help pave the way for others and that was a responsibility being a part of this cohort yes deepen your own system transform your system pave the way for others and so I think throughout covid a little bit of a happy accident in some of those virtual meetings we began to utilize them as thought leaders so they helped to create a personalize learning framework that you're going to see um coming up next they started to think about what are some of those tenants you know

174that that we believe play out differently but they unify us all and so some of those pieces have helped get us a little bit closer to this idea of sustainability and once the official cohort ended last fall the districts are through the network continuing to reach out to each other continuing to ask questions share resources um and it's it's incredible to see and I I don't know that Ann and I thought always that was going to happen so that's I would say a big highlight an anything you would add thanks um Lori the other thing I would say is from the state perspective it was really important that we designed and the we have these um personalized learning framework elements with some key definitions and look fors within them um on the surface um let's

175say build relationships with students um it's easy for schools to say yeah yeah we already do that we do that that's a part of our um rtib that's a part of our mtss but really when you look at it we're looking at those meaningful relationships between the adults and Educators or adults and students students to students um we really believe in a of personalization especially when it comes to their work so their work can um support that relationship development um throughout so what I would say especially as we've pursued Innovative education being clear and articulate on your state structures not necessarily as a you know know your mtss system doesn't necessarily do it exactly this way but allowing people to see where see them M eles within your framework um and understand how they continue

176to learn grow and deepen within that framework that already exists another piece that that's been very important in North Dakota's work has been a portrait of a graduate and you'll hear about portrait of a graduate when you hear from a lot of states that pursue Innovative education or personalized learning in North Dakota this was developed under Governor Doug beram and what we found was as our districts worked through their local processes on personalized learning they went through a process to Divine their local portrait of a graduate and um as we continued to have conversations on state policy or at that level above the school districts having a structure and a framework and a common language for those other entities was critical one of the things in North Dakota the portrait of a graduate is not

177required it's not required in statute it's not required that anybody has to have a portrait of a graduate what it does provide is it provides a North star for the state of North Dakota for educators and Learners and our school systems to aim towards uh this was very helpful especially in the legislative session to have that common language and those stakeholders whether they be K12 Advocates business and industry all kind of speaking to those same pieces on this slide you'll see the 2020 one porra of a graduate that was designed and developed we had over a hundred stakeholders um everything from business and industry to non-public schools to students to um educational leaders a part of this process so this was developed under the governor's office but it's become a foundational element that we've been

178able to build into our other practices such as the northa learning Continuum so in 202 21 that legislative session our schools and districts said the Innovative education program is great we'd like some flexibility on a graduation pathway that might look different than your traditional four courses in English let's say so Senate Bill 2196 provided the state superintendent the authority to create a learning Continuum and it also provided flexibility for a Mastery framework we'll talk about what that means in a second but for the learning Continuum we felt that this was a critical piece for our schools and districts in order to fully transform their systems to a personalized Competency Based system it provides a structure for integrated and engaging learning opportunities allowing for mastering Essential Knowledge skill knowledge skills and dispositions that align to the

179students post-secondary career or military interest North Dakota has a vision that all students will graduate Choice ready with the knowledge skills and dispositions to be successful that's in postsecondary career or military so this portrait of a graduate on the prior slide became a foundational element for our North Dakota learning Continuum our learning Continuum has elements that are as I said aligned to to the the essential skills of the portrait of a graduate also we have those overarching skills in English language arts math science and social studies Lori I'll let you speak to a little bit about that process when we actually designed it in 2022 thanks Ann so in 2022 again it was a you know we thought we could do this in person and realize that actually virtually was a little more efficient so

180knowledge works tandem with GPI we led a process to bring Educators from all across the state not just Educators who were in the original cohort um we were really intentional to say we need Educators from different regions um we had the Reas which are Regional Education associations um be a part of it we had when it came to the attributes from the profile of a graduate there were some folks who were counselors some who were maybe coming from social work um brought stakeholders together and said okay what are the attributes here that we believe are essential and when we talk about the attributes we're scaling up to be more Broad in this from the standards realm so standards don't go away but we really needed to help Educators and leaders understand the difference between the

181granularity level of a standard up to a priority standard up to then a competency and and so thinking about St a lot of people had fear that standards would go away standards still have a place they are the vehicle that Learners will actually use to practice and show Mastery of these skill-based competencies but a competency really is meant to be a crosscutting skill or concept so as Ann said if you look at the North Dakota learning Continuum you will see headings which we think is a baby step a training wheel some may always leave it there you will see social studies as a category but if you look underneath social studies at those attributes if you removed social studies as the topic area they could be practiced across the board in all content areas so

182we really feel as though the Educators and you know other stakeholders that came together it was such an aha for them to think about what do we truly want kids to know and be able to do and then when they walk across that stage we're saying that they have mastered all of these they might practice them with different standards but we're saying that being an Engaged citizen or engaged citizenship is important for all and so it was um I think it was one of an and I we walked away one of our proudest moments um to be candid I think districts are still struggling to implement there were some really great deep conversations and now putting it into practice ractice looking at these grade banded progressions um is is a different matter and so thinking

183about what comes next in order to help them operationalize a document such as this thanks Lori so the piece that puts this into action in schools and districts was the second part of Senate bill 2196 that went along with the learning Continuum and this allowed for schools and districts to create a Mastery framework policy so local school boards could create a policy that really provides that alternative graduation pathway different than what's required currently in statute however that Mastery framework policy must meet or exceed the North Dakota learning Continuum so as Lori said this is an area where we see our districts maybe struggling on what does that look like and it might be easy to identify what it looks like for a certain group of kids or um a a certain student within their system

184but we're really thinking about the systems level so what does Mastery look like at an opportunity for any student and again we believe that instruction and learning has to look different in order for the ma Mastery framework to be used to its full potential um as a line to their local Innovative education plan so um the Mastery framework we do see it occurring in one of our our schools in Northern Cass that was one of the schools that was a part of our pcbl cohort they continue to be a leader in this work in the state and in the nation and Lori I'll let you explain a little bit about what we see happening in their Studio models really where that learning looks vastly different than a Traditional School model absolutely so Northern cast decided

185they they're always a how do we get to yes District um and always looking to put learning in the hands of their Learners and kids and so they learned I believe Ann it was I Iowa I want to say Iowa Idaho um they went and visited they learned they saw you know that District implementing or that state the studio model and came back and said you know our middle school learners are languishing a little bit we're we're looking for something different at the middle school level so that's where they started um they started to say you know what we still do have foundational skills that kids need to learn um in some standards that are those priority standards that are not going to go away but can we make space within the middle school day

186where kids can learn in a studio and right now it's still pretty I would say educator Le where the educators are creating Studios that are more of a performance I would say leading into a performance assessment where there's a scenario and so there may be car crash physic physics but wrapped within that six week experience for kids there are identified competencies and there are identified standards that kids will be able to master as they move through that studio um a piece that I want to make sure that you know is that the learning Continuum really was meant for districts to adopt if they needed to adopt it and didn't have the capacity to go through that productive struggle and create their own nor cast is a district that had done some of that leg work

187so they did not take the North Dakota learning Continuum and just adopt as is so Dr Cory Steiner and team they wanted to match local language they wanted to align to their focus areas so they have taken you know and an said it has to be um The Learning Continuum is the is the floor you know so they had to go above and beyond that so they have their own take on the learning Continuum um and another piece of their studio is there has to be an an expert in expert out so they really are bringing in community experts and Community Partners that have to be a part of the studio experience for kids kids get to join and decide which one they want to be a part of and now kids are starting to

188recognize I could have a different end product than you know someone working next to me in this group and because of that I might bring in a different competency I might bring in a different standard and so there are some baked in but there are also room for kids to have voice and then you have some Educators who are working with the kids and this has now flowed up to the the secondary level and it plan is for it to flow down into the elementary but they're working with kids as coaches to say you are in need of this this and this competency how could we put a studio together for for you that is something that you're interested in and that it's something that you can work through and you will Master the competencies

189that you need to master while also being in a preferred learning experience that that aligns to your goals so they've seen a lot of success with that even in remediation at the secondary level and so a lot of room to grow but they've been the first district to to take advantage of Senate bill 2196 in this way thanks Lori and then lastly what we like to talk about is kind of as this work has evolved since our partnership with knowledge Works in 2018 we brought these schools and districts together they are part of a cohort um the sense evolved our funding with the bush Foundation is done and as Lori indicated you know we kind of were like is anyone going to still be engaged after this work what it currently looks like is our

190cohort has merged into a North Dakota Network for personalized learning we still recognize that districts need support beyond what the state agency can provide they've leaned on one another to receive that support and have a place for Thought partnership for expertise for other North Dakota Personnel to share their learning and experiences in our Network for personalized learning we really focus on a few areas in particular what does learning and growth for the educator look like um what does learning and growth for students look like how do we share progress and spread the word of the personalized learning happening throughout the state provide leadership coaching and development as we indicated we believe that we need entrepreneurs within Educational Systems um so that needs to look a little bit different especially in Innovative environments and then always

191focusing on continuous Improvement for the network for the schools themselves and on a broader scale so throughout this journey um we recogn ize that you know our school systems they're not starting brand new and starting from scratch um on this work um I'll often say like it's trying to compete for that top chef competition between 5:30 to 6:30 at night in my kitchen like that's probably not the best time for me to become the specialized at my craft if I'm trying to focus on that systems level scale others will say it's like remodeling your house while still living inside and having all those functioning parts and parties that you may be hosting um we're asking our school systems to live in the current and provide something different so supports resources technical assistance um and being

192really adaptable in what that looks like is very important we find some of our smaller districts they might not have coaching capacity um as I said their person is wearing 14 different hats much L A coaching hat versus our larger districts might have coaching supports and structures but how do you ensure all 15 of your coaches are on the same page so the needs and supports look different between all in that personalized learning cohort um Lori I'll let you speak a little bit about this this next slide and then we'll be ready for questions okay the North Dakota Network for personaliz learning launched last year um as Ann said it really did evolve out of the cohort and so a lot of the local leadership um I will say that an and I are behind

193the string the scenes still maybe pulling some of those strings as the handoff isn't completely ready yet or clean um and so as a knowledge Works representative who feels like a local um in North Dakota I still am directing this network but very behind the scenes at this point you're not going to see knowledge Works name on something at this point you're not going to see something likely coming from Lorie Phillips when we talk local leadership we decided that the network could focus Fus on learning opportunities opportunities to share districts and Educators always going to see what does it look like and they feel they have to travel across the country to see something great in another classroom and they don't things are things are great things are happening even within their own District maybe

194they don't get the chance to walk down the hall and it's definitely happening across districts in the state and then this idea of impact and Improvement how are we measuring not only the progress of the network but how are we measuring the progress of personalized practices so right now there are three coordinators that are heading up those learn share and impact Improvement teams and those are coming straight out of the districts um we have Tom clap out of Northern Cass who is leading the learning opportunities he is still doing his current full-time job but believes in this and his superintendent Dr Corey Steiner said we I would be a fool not to want him to do this work because he's going to bring as much back to Northern Cass as he's contributing and this overlaps

195with the work he's doing for our district so absolutely it makes sense um so we have people from bismar from West Fargo from other districts who are doing the the the leadership and they're doing like the contribution into the network um instead of it continuing to be the lorian and show who then are just providing opportunities that may not meet the needs um but it's coming straight from those who are looking for the impact um so lots and lots and lots of iteration and room to grow today as we speak there were two um learning workshops happening in Fargo and in bismar um and an and I were not a part of them so years ago we would have been leading them creating the agendas doing all the things and now they're happening um and

196we are guiding you know from the side so it's exciting it's exciting to see where it's going to go and exciting to see how North Dakota can continue to work towards that sustainable Network um where they can learn and grow within the state thank you Lori so um superintendent Coons and Madam chair we will gladly stand for any questions that you might have or additional information that you would like to learn about North Dakota knowledge works or the support we to provide to our schools and districts thank you than you so much for your presentation we really do appreciate it are there any questions from board members Dr Northern why did heads turn down here I know I did I did ask for the presentation um all right I've got several questions but I will

197be brief and hopefully some them won't be too long um one I'm just curious what was waved right you said at the beginning that you know there were certain things they wanted wave so so what is it that has to be waved that's getting in everybody's way the most the most cumbersome legislation um or or guideline to is when you're talking about what um you know what do you have to do necessary for graduation let's just look at the portions of the learning Continuum my concern was that you are um prioritizing skills over content so just kind of make me feel better there um because I think it's easier to prioritize skills when you're talking about um a framework like this versus particular content to be mastered and I'm curious whether you had to actually

198alter any course requirements for graduation so we're all familiar with four four courses of ela and three of math and three of science but what we're seeing across the country is that like you can't fit it all in um and so I'm just curious like do you have like you know opportunities where they can finish all their course requirements by 10th grade for example and do Innovative personalized programs whatever in 11th and 12th grade so I'm just trying to figure out how you're cramming everything in because it seems to me like something's got to go and then my other question is on the bush Foundation honestly just was curious like um you know it is your sense that all of these programs really did require foundational additional Foundation support and I promise my last one

199I hope you wrote These down it's just on quality control I've been concerned that um we are all about Innovation and we are but we also need to make sure that these um programs are producing outcomes for kids so talk to us a little bit about how you view quality control and honestly if that's too much we can do some of this off you know thank you for those questions um I didn't catch your full name but um I will go ahead and answer um what we're what we're able to hear so when a school writes its comprehensive plan I will tell you most of them like it's like well it's seat time seat time is my problem I need to figure out seat time it's the number of hours that they have to have

200for the unit of study and for CTE it's higher and um in my experience that's too narrow of their thinking so if they think about the statute that's in their way they um they think too narrow so when you really challenge your schools and districts to think about like what would your ideal School System look like do the kids have to walk through a threshold to be counted in attendance to receive learning instruction and ultimately payment like so they think of this broad plan that comes into the state agency and then people like myself who read the Sentry code on a regular basis and know all the different numbers that a school district doesn't like they know the repercussions of seek time but they don't need to know the Sentry code elements so to answer

201your questions specifically what we after I review the plan we call out in that plan and literally Redline it and say Well they're going to need seek time flexibility here they're going to need teacher licenser flexibility here they're going to need flexibility on learn we call it learn everywhere but where like learning might happen at Boy Scouts and they bring that learning in and work with maybe a literacy coach maybe it's a personalized learning coach where the teacher of record might not be the master of English language arts but they consult with the master of English Lang language arts we see um flexibilities in internships and externships and apprenticeships the other thing that we're seeing some flexibility that comes up naturally after they're implementing we have some dual credit some higher education requirements and you

202have to be this this many years old to be able to participate we're able to accelerate that because we're able to prove the students are ready for maybe some dual credit access courses or even beyond what our typical English 110 might be and they're on a more accelerated path so the flexibility that our legislature was able to provide to superintendent Basler um at first it was drafted that she could wave anything in statute like including the speed limit um so we had to revise that but there is flexibility that even for or something like dual credit that is not overseen out of our K12 education that's our higher education institutions um and through the university system we do have authority to wave that section of Statute if it's in the way of kids so I

203think that's the number one there that's great I'm going to talk about quality control real quick so when we um when we pass the bill for Innovative education there is an element of quality control from local districts reporting their progress um and you know they're busy they're busy teaching at the same time they're trying to figure out their evaluation and their progress what I'd like to say that has been probably one of the biggest impacts of that bush Foundation investment and our pcbl work was a longitudinal study so there are multi- Partners knowledge rcks is primary partner number one and our my best friend Lor is my best friend in work regarding all of this work but one of the partners that came to the table is West Ed so West Ed has done a

204comprehensive study during our six years under this work and actually today I had a call with wested they're not under contract with the bush Foundation they're not under contract with knowledge works but they've come back to the state of North Dakota and said hey we want to update on our data we're two years past their research they want to do a Refresh on that data to look at the longstanding impacts of this work so again it's about Partnerships and relationships um not necessarily contracted services from vendors to get this work done so there is an element of quality control um we have positive progress we had some Flatline progress in in some of this work uh what we did see out of the research is students that participated in districts that had a personalized Competency

205Based learning approach they attended School more often that includes during the pandemic and they're more engaged in school and we saw some academic indicators as well I do want to put an aster on that though if we look at the school sizes like the N size is tough our ycc school which is the adjudicated youth they had very small numbers so they were not in that and then our largest School West Fargo is a really big school so like they could easily shift our data one way or another but um we're continuing to monitor those Trends in that broader research as well as local districts reporting what that looks like um graduation Lori I should pause do you have anything to add on the waiver or the quality control heard yep what I've heard so

206far is that um and westad evaluation was that something your general assembly put in put in to fund that evaluation where did the fund are we going to continue to fund that evaluation yeah who funded the westad evaluation so the original funding for the westad evaluation was part of the Bush foundation's expectation and work so that was provided originally I believe West but but again keep in mind that work ended in oh 23 I think spring of 23 and so as I said today we're still having meetings with them even though they're not in under contract to provide a continued report for North Dakota thank you yeah and I would say I don't have anything to add um in what you've already talked about but I can certainly begin to speak to the prioritizing skills

207over content and the idea that that potentially is easier um and so I think when you look at the learning Continuum and you see the skills that are listed there what isn't listed there are still those priority standards that kids are going to be working towards you know no one is going to say that kids still don't need to know how to read to write to there there are math standards that are really necessary um what I think that the learning Continuum does is prioritizes application and so when you think about potentially traditional math classes and I'll speak for myself you know I was able to do all of the discret skills but I couldn't apply it when I got to college I received a D minus in a stats class and the feedback they

208gave me was cannot apply and so I think it's putting kids in situations that when you ask kids and we talk to the kids in the bismar empowered program they're going to say you know what this is harder and they don't choose to take all courses in Empower Ed they might choose to take an economics course but they go ahead and go back to their home school and take social studies and when I asked them why they said because it's a lot more work so I'm only going to do it in something that I really care about and I'm really interested in because I have to come up with my plan I have to figure out how I'm going to show and demonstrate Mastery here it feels a lot more cumbersome to kids but they'll

209also say it's a lot more meaningful and so I think this idea of you know easier an and I certainly with Senate bill 2196 um we had a lot of conversations behind the scenes that said you know could there be some districts who graduated kids for just being really good people you know and how do we how do we keep that from happening um and so what I would say is you can't and I think you can't now in the current system um but Ann I see that you've brought up the learning continuum um you know piece here and so I think it really is still a combination of those standards with the skills being if you could forgive me and I can use a sports analogy you know I think the content used to

210be the end zone and now we've made this idea of kids being able to problem solve that's the end zone and we're going to use the standards as the way that they get there and practice that um and what will you add from this yeah I just thought it would pull up this slide that we often share as we talk through the learning Continuum and um in our traditional model we have North Dakota has standards and it's required that our courses must meet or exceed those standards often times schools deliver the course right it's an English nine English 10 English 11 those four English courses and I have two high schoolers right now if they sit long enough if they jump through the groups if they do their homework they get that unit of credit

211the learning isn't necessarily driving the issue of the credit or what that looks like not in every situation I must say but in a in our traditional model that we see in many of our schools under the Competency Based learning and Mastery framework the bases yes the standards are there the the the this the learning of those um of those standards is still a piece but the focus is on that essential overarching element in that learning Continuum I'm going to flip to another slide real quick that shows this is just an example from the learning Continuum where critical reading might be that competency that we're aiming for the learning might come through Ela it might come through social studies it might come through my Eagle Scout badge it might come in a lot of different

212ways that's for a local District to design and develop like what what is a Sandbox in which this can operate that then is captured within that Mastery framework our law does not require a district with an approved Mastery framework policy to back map it to four English credits in a traditional model they have to back mmap it even a competency based system often has to say well you did English 9 10 11 12 our system including the state scholarship allows for Mastery to be documented according to that learning Continuum and then that that credit that recognition um occurs the other thing that I think is important is like the example that I had back here for grades six through eight that doesn't mean you master this in eth grade I mean good good gracious I'm

213probably still working on some of these and I'm a well seasoned adult um so we have many times throughout our educational career that we would focus on and I'll maybe this slide would be helpful for you to see too those overarching academic attributes what they look like in practice and multiple times throughout our career what we and what we are not seeing even in schools that have the Mastery framework we are seeing that their kids are coming with the ability to apply those standards in real life settings and in ways that make their learning more relatable more actionable more meaningful to themselves and learning becomes much more fun and much more engaging and something I think we want to also like reiterate is that there really aren't big swings here someone is not going to

214take you know this learning Continuum in Senate bill 2196 and just go Implement all of it you know all at once so when when Dr Steiner and Northern Cass decided to begin down the road of the studio model they took some small steps you saw the empowered program started with you know 20 25 kids so really trying to I think get some proof of concept there before so start small scale fast you know iterate iterate iterate and so there really are small iterations that have a large impact you know that that we're going for and I believe there was one more question on the bush Foundation but I don't recall what it was it's okay it's you guys have done great yeah I'm thank you guys thank you all so much we really do appreciate

215you um presenting for us today um so thank you for taking the time that you did with us um and we will certainly be in touch as we move forward um to continue to seek your advice and your counsel as we move through this process so thank you yes thank you very much we appreciate the opportunity thank you thank you okay so as promised Dr Coons and Dr Armstrong will be presenting if you look at your um SharePoint it's the second presentation and I think this one's going to be brief and after this presentation we'll take a very quick break so that we can determine whether or not we have to leave here at 4:30 okay thanks well I really want to appreciate knowledge Works in North Dakota who as you saw in their timeline

216has been doing this for quite some time working on Innovation working on iteration some of the things that they shared with us were some of the things that we heard as we've gone through extensive feedback this start small scale fast the the thoughts around networking I think starting with needs instead of starting with statutes and waivers has been a really clear focus of ours as they said they're calling this personalized learning as we looked across the country some people are calling this Mastery based learning some people are calling this Competency Based we have given this the umbrella seat time Flex because that's where the legislation landed but we're really excited to talk to you about today is what we've heard as we've done extensive stakeholder engagement feedback sessions listened and talk to our lab school

217networks we really are excited about spreading and learning from the innovation have and I think they talked about a network and um we are excited that there is an innovation conference April 22nd through the 24th and one of the key pieces of that conference will be design thinking and any school any School Division can go to that conference and start this opportunity for design thinking around um house bow 1477 and Define design thinking and C templex I'm going to turn it over to Dr Armstrong to take us through good afternoon Madam president board members I'm GNA start talking there we go thank you we as you all know uh back in December the the board is aware in our discussion we talked about HB 1477 which is a law that uh provides us with this

218opportunity to implement seed time flexibility I do want to orient our work around the requirements of that legislation 1477 requires Board of Education to provide Local School Board 's maximum flexibility to wave existing instructional clock hour requirements by developing alternative instructional time models in accordance with the board's regulations and guidelines relating to instructional time waivers and directs the board to provide flexibility for school boards to develop programs that provide for acceleration remediation and multiple Pathways to graduation that permit students to demonstrate competency at different rates um just wanted to make sure that we were all oriented around that being the legislative directive and we use the term seat time flexibility interchangeably as we just mentioned with personalized and compet based Competency Based learning which is designed to prioritize student outcomes and students advancing at their

219own pace to engage in Innovative experiences and we going to let Dr speak for a moment oh it might be maybe that's my problem yes it is thank you Dr C um I'm gonna let Dr Co speak for a moment about uh some of the example that that we have developed really a library of examples from across the state as we discussed back in December as well as our stakeholder feedback we've received I really appreciate the opportunity I think when we came and chatted about models and examples in December one of the feedback that you gave us was what is happening in the field in Virginia so we can't go through all of these today nor do we want to because we may have a very brief amount of time left for the day so

220really encourage you to click on each one of these links we have cultivated examples in Virginia of models so when we think of alternative hours Hampton City Schools implements a 4515 model that changes that clock hour and that seat time to provide Academy opportunities Hopewell City Schools does interner sessions which allows alternative hours where they go in a balanced calendar approach and have two weeks of inter session where kiddos get Innovative courses maybe not by their fourth grade teacher per se but it also reinforces those fourth grade literacy and math skills and students who are struggling come during intercession for that Innovative content and have more time and more approach before the end of the year that is a K12 model in Hopewell it's not just a school it's a large scale model Chesterfield offers

221High School courses from a high school schedule and fits it into a middle school model and changes the course design we have flexible learning settings you guys have heard from Louisa several times they actually take their teachers for tomorrow and put them in a different location where they get job embedded support in a child care facility for employees but they get their cours workk in the same place so it's a different flexible learning setting and then we also have Blended learning courses in Virginia Beach which Dr Armstrong and I had the opportunity to be at Virginia Beach two weeks ago and see students in this biofit freshman course actually in health facility so they were with community members who were working on workout facilities in the pool having their Fitness classes and their biology course

222was in an off-site location inside that fitness facility so we have a lot of opportunities for flexible learning and we have the opportunity to start small but talk about what knowledge Works in North Dakota talked about where there are examples within us that we need to network and learn from so we're really excited that these are the four models that have itated to this point where we are in front of you today on first read for notice and really excited about the level of feedback and the level of Engagement because what we're seeing as we listen to stakeholders that our leaders are excited they are tired of that 150 hours every kind of kid starts the day at this time ends the day and this is how it works but they are requesting it to

223be simple and start small and how do they get the guidance one of the things they said when we looked at guidance is give us the first 10 steps we need to know what the steps are to get us there and I think you heard a little bit of that when knowledge Works in North Dakota were talking about how they iterated the struggles they had so one of the things you will see in the package for first read is you will see not only the notice the examples of the notice but the actual action kit and that came out of our feedback that opportunity to really get those first 10 steps and how to get our school divisions comfortable with and our principles comfortable with what some things are happening in small spaces to spread

224across our state and then most importantly they want a video library they want a video library of examples so they can sit down with their team start designing and thinking looking at what other things are happening so those are exciting things that we've heard um Dr Northern you asked about what are some of the flexibilities what we've heard is that teacher record and that teacher lure and how they manage that we've also heard um some about assessment grading and Reporting and what kinds of flexibilities are already happening in some of those models and how they replicate those and make sure that their teachers are comfortable with that space and now I'm going to turn it over to Dr Armstrong to do the technical pieces thank you Dr Coons as mentioned we have uh provided within

225the action kit responsive to the feedback we've received a notice form with accompanying anti ated examples to guide their implementation of these flexible models in schools they are included in the action kit which is a clear step-by-step phase in guide for schools to support this streamlined student outcome centered logistically manageable plan I think that's a a barrier we see with principles and superintendents just having this come to life the video library the action kit does that for them so we we have examples that I'll just scroll through these very quickly the notice form in addition to a lot of the sort of boil PL demographic information about the school and the setting uh requires narratives related to the uh selection of the sea time Flex model as well as the desired outcomes and rationale for

226students so you can see this is an example of the narrative section of a notice form with a Balan calendar model which could apply K through 12 we have another one for a middle school model which would be a fairly typical situation where an Earth Science High School Credit course is offered at both middle and high School in the same division but due to the schedules being different maybe an8 period day at the middle school seven period at the high school it will require some flexibility in time and then a high school Blended course model which this is the the very same program we just mentioned in Virginia Beach the bofit class we have double blocked uh co- talk classes that blend content sensibly between PE and biology for example uh in the Virginia Beach

227model that we saw the notice form does include assurances um those insurances are going to to include reporting attendance communication with families and also the requirement to provide support to special populations so that is included within the notice form and then the action kit and I I can't get the link up now but hopefully you've had a chance to take a look at that I'm proud of the work that we've done to create this action kit draft that you'll find link there at the top of this Slide the action kit is a robust resource for schools to thoughtfully plan and Orient their seat time flexibility model and as you walk through that document you'll see we have back ground we have links to the video libraries we have the notice forms um a lot of

228very practical deep information to guide schools and school divisions and implementation so we are excited to continue this iterative and responsive work and and further flesh out the action kit based on the experiences that we're going to hear about in our schools and we will continue our work to disseminate these models Statewide and we're here to answer any questions yeah I think one other thing we want to say about the action kit as those challenges arise that knowledge works and um North Dakota talked about one of the things we heard in Virginia Beach is can you give us some Transportation options can you give us some things and so as we hear about those we'll continue to build out resources both from Virginia and from National models such as not um the things that North

229Dakota and knowledge works that allows this action kit and the appendices of the action kit become iterative and supportive as our schools evolve and start small and scale what they're trying to do I also think it's important to note that several School leaders have told us this allows us to scale a lot of the things we're doing we've started small and we've stayed small the opportunity of House Bill 1477 now allows us to scale into larger opportunities so we're really excited about this work so um just as a comment you know one of the things as I thought through this whole process uh this is really a very unique opportunity for our school divisions to learn learn from each other um and how um you know each individual School Division that is currently using some

230type of model as you guys have talked about earlier in the presentation um you know sort of to have that um Blackboard type resource that says here in Virginia Beach we do X Y and Z here in Chesapeake we do ABC and and these are the ways in which we hand Le these obstacles and so I would just really encourage us to use some opportunities within this particular option to be able to allow the school divisions to have easier communication with each other on how to best um just exercise all opportunities Within These models that we have available so um I really do appreciate this work so thank you all any comments here Dr Northern make it brief um I really like their idea of having some type of reporting mechanism whether that be you

231know General Assembly whether it be you know partnering with the local Virginia University to give us feedback and evaluate uh the program because I'm with you I think that we need to learn from it and that would be another way that the districts could learn from one another if we actually had some kind of a valuative component um to it to to share those ideas and also to inform the program as it moves forward what's working what's not we what we could strengthen okay if there is nothing else we're going to take a five minute break and then come back to Circle back to item C e e e with a presentation along with Dr Wallace take away thank you Madame President and board members just waiting for our presentation to appear and that when

232it does Dr Wallis and I are going to provide you with an update on the agency's ongoing efforts to provide extensive support to Virginia School Division and their implementation of the approved school performance and support framework as the board intended the school performance support framework is intentional in its provision of support to divisions across all dimensions of the framework and its implementation since its approval the vdoe team both in the office of accountability and elsewhere in the agency has provided constant technical assistance organized on demand training for Unique stakeholder groups responses to specific questions and added additional resources in response to the needs of schools and school divisions as the board knows the VOE launched the school performance support framework resource Hub in November where anyone principal teacher superintendent parent or citizen can find not

233only the technical details of the framework and transparent data analysis tools but also important resources geared toward addressing common questions around support for targeted populations High School Readiness and clear materials for parents to support their child at home the resource Hub continues to grow as we engage with schools and stakeholders across the state and you can find regularly updated information on office hours at the resource Hub toolkit page as noted here we go on the slide that I know you all have in front of you right there circled in Red so every week we will update we have been updating that with the statistics on our office hours and participation rates that engagement with schools and stakeholders represents hundreds of hours of in-person trainings question and answer periods and technical assistance sessions across the state

234since last Fall highlights include weekly office hours which attracted 102 participants from 68 School divisions at our February 5th session 16 Regional superintendent trainings 12 trainings for division data and testing leaders and nine trainings for specialized organizations such as vas vassp VA ESP and backi we are also offering dedicated training for divisions on requests 14 of which have already taken place and just last week we scheduled a training for a division on the request of the superintendent to include the local school board to provide a detailed overview of the framework in addition to the trainings noted here our team answers individual questions and takes calls every single day in order to provide our schools with the information they need to support the work of the schools in their divisions I want to specifically thank our

235accountability team and our data team who have collab elaborated to provide exceptional responsiveness to our schools as mentioned our support goes beyond the data and calculations and targets nonacademic barriers that we know affect student performance and we are excited to follow last year's successful attendance webinar and Symposium series with this year's program bright spots and bold Solutions which will be led by the office of Behavioral Health and Wellness Virginia has been noted nationally for our work to move the needle in reducing chronic absenteeism the past two years and this year's attendance series was designed based on feedback from our divisions about what they still want to learn more about to improve attendance in their communities next month series will focus on attendance Specialists Transportation mental health and parental involvement and will feature guests from divisions

236sharing best practices that are showing results to lead the dialogue the webinar series will culminate in the VOE attendance Symposium here in Richmond which is being finalized now for a location and date and before I hand it over to Dr Wallace and provide the board more information about supports Dr I believe you have a few comments You' like to make I'm just really proud of the team and all the work they have done to be shoulder-to-shoulder with our school divisions as um Dr Armstrong mentioned the volume and support of our twers accountability team to provide that level of support and the ongoing support that now we are partnering with school superintendents to support their boards and their board transition in the space I also um just am incredibly grateful for this attendance work I think

237we are featuring a three to four School divisions in each one we talked about the networking and the sharing this is another example where our school divisions who are leading the way are sharing with other school divisions to support the growth and spread of practices and Dr Wallace is going to talk about four literacy networks that we have partnered and our work around zern which continues to expand that shoulder-to-shoulder work in academics I'm just really thankful for all the work our teams are doing to support our school divisions in the field and putting connecting our school divisions together to get resources and supports from each other as well Dr Wall good afternoon president crey board members and Dr Coons I am happy to present with Dr Armstrong today on the supports that we're providing to

238our schools um first we have the comprehensive literacy Network and this is a network of nine School divisions that provide support as they are implementing the Virginia literacy act these divisions have selected 43 schools that will receive direct individualized support which includes weekly support for literacy staff as they Implement their adopted hqm curricula as well as English literacy walkthrough data collection in practice with the division established walkthrough schedules an analysis of student work samples with a deep dive into the walkthrough data and student work samples to provide professional learning for schools for the remainder of the school year we will begin looking at Division and school data and that will drive year two of this initiative which will then expand uh with the existing nine divisions to uh introduce other more more divisions into this

239program next year our English learner literacy Network provides supports to school divisions representing 30 schools and 16 School divisions these divisions applied for grants for their schools to participate in this network based on their schools needs and this support includes three inperson convenings followed by coaching sessions and technical assistance to support School Leaders with evidence-based reading research and to support literacy teams on the impact of structured literacy with English English Learners excuse me um This research will continue for the rest of this year and they will create plans to execute next year to directly support our English Learners to improve instructional growth and graduation rates next our students with disabilities Network provides support to 30 School divisions from 17 School excuse me 30 schools from 17 School divisions as with the El Network School divisions

240applied for Grants to participate and this includes inperson as well as online convenings for school divisions to collaborate among each other with focus on high quality curriculum Staffing and scheduling to meet the various needs of our students with disabilities schools will create an action plan to increase student achievement and growth with students with disabilities all three of these networks will use a common literacy tool as they uh do their walkthroughs lastly I would like to share the supports around mathematics we have been providing supports to students and schools with that are using our zern program through a special program called The Anchor program as you know zern is our mathematics tutoring provider through the all-in programming and these schools which includes 30 schools from 14 School divisions are receiving additional coaching and they are getting

241monthly meetings with division leaders bi-weekly meetings with each School leader and they are doing a deep data analysis to help Implement zern at a deeper level we have already seen a huge amount of success with our zern anchor divisions where we were had some concerns at the beginning of the year with these divisions and now they are implementing as model School divisions for zern all of this work is very important to support our schools as we go through the Virginia literacy implementation act um and then also as we continue to support programming through our all-in tutoring I'd like to uh shout out our vdoe English English learner special education also School quality and Mathematics staff as they are supporting all of this work with our schools and divisions thank you Dr Wallace is there does

242any board member have a question or comment here not they walked off no I'm just kidding questions or comments Miss Holton thank you all for these um uh presentations with the respect with the networks and the zern anchor program you're uh uh can you tell us how the you said you had more applicants than you could serve how were the applicants how what was the decision criteria for who participated is the first question and then the second question is uh do we have a research partner or somebody who's studying outcomes in in the network schools versus the the non- network schools or others in other how how are we reaping the lessons from these essentially pilot work because it's not well this this work that's going on that's not at scale how are we reaping

243the the learning lessons that we can then take to scale common Dr Coons I'm going to answer the research piece and then I'm going to let Dr Wallace talk about how her team work to select schools we have research partners for all of the literacy work that is funded by the clsd and requires a thirdparty evaluation and then with zern we have a thirdparty evaluation that is going to evaluate impact so in everything that Dr Wallace presented there is an outside entity doing evaluation work the literacy networks probably are 1 to three to five years given that our clsd grant is 5 years long um zern is going to do an evaluation at the end of this year in regard to the selection process I'll start with zern um so zern School divisions were selected

244at the end of last school year based on on the needs of the School Division and so we kind of handpicked those divisions and we also uh looked at divisions that had high needs that needed those supports the extra supports um for implementation of the program um and so uh we looked at our zern data as well as the school needs whether they were atsi CSI that that type of uh status that went with the school at that time the comprehensive literacy Network which is the nine School divisions um those School divisions again were kind of hand selected after we looked at school data we looked at atsi CSI and TSI School data for our elementary schools as well as um once the data was released in the fall with the new support system we

245use that data and those schools that would have fallen under needs intensive supports were also kind of targeted uh for those divisions we did um reach out to divisions and ask them to participate and so so um you know that that's how they got into the network and then next year we will expand that Network so that we'll have Regional supports all across the state and the nine that we have this year are going to be used as model divisions um so we'll use the the data from this year to kind of drive next year's efforts for the clsd grants that includes the English learner Network and the students with disabilities networks those um applications came in by divisions they applied by for each school that they wanted to so they could could have applied

246for more than one network we selected them based on a rubric that included their current status as well as their projections of what they wanted out of the Grant and what they wanted out of the network so there was an application review process um so if a School Division applied for a school to be in both the El Network and the sped Network we looked at their data and we decided which network would best suit them and so therefore they weren't duplicative efforts um in a school and more school divisions and schools could be served and so um where you see the the count of schools that would be a duplicative count in both of those that's you wouldn't add those two numbers together for the total number because some of them may have applied

247for both of the networks um and so uh you they got priority points again if they were uh atsi or CSI or needs intensive supports identified schools and so we're working with our needest schools and what their needs are specifically around students disabilities and English learner data thank you you're welcome anything else okay thank you thank you and we are moving to item D School connected overdose parent notification guidelines oh we're more than happy to I'd be happy to do that yep thank you madam president board members I would like to provide an overview of the guidelines for school connected overdose response and parental notification policies today for your consideration on First and final review I will provide background and context to this item including an overview of essential policy and legislation a review of

248the extensive stakeholder engagement in both the development and of and feedback to the proposed gu guidelines and a description of the key components of the proposed guidelines the guidelines before you today are a result of several cascading policy and legislative priorities that began with the governor's 2022 introduction of the right help right now plan to improve Virginia's Behavioral Health Care system through expansion of access to Quality Care in coordination with healthc care law enforcement and mental health organizations and agencies in 2023 as a result of a large scale School connected overdose the governor issued executive order 28 requiring notification to all parents in the school division within 24 hours of a school connected overdose and outlining collaboration between school divisions and law enforcement E28 directed The vooe to provide guidance to school divisions with additional

249detail and resources to implement 24-hour notification this guidance was provided to school divisions in February of 2024 finally the governor signed into law SB 498 in 2024 which required the Board of Education to create the guidelines that are before you today for consideration this Law requires that the board establish guidelines and as you see before in the slide this is the text from SB uh 498 which is now that section of the code which requires the board to establish guidelines to include a model action plan detailing agency and law enforcement Communications as well as criteria to ensure sensitivity to privacy concerns of affected individuals the VOE is under take an extensive input and feedback gathering with both key stakeholders and the public in the development of the guidelines including three town hall listening sessions in

250June agency staff began drafting the guidelines with the input of the Department of Criminal Justice Services Department of Behavioral Health and developmental Services the Virginia Department of Health as well as counseling and nursing leads from 10 School divisions representing variety and geography and enrollment from across the Commonwealth in November key secretariats including Education Health and Human services and Public Safety and Homeland Security provided input to the guidelines and in December a school counselor focus group and school nurse focus group were convened to provide input prior to an initial listen and learn session in December the guidelines were posted to the VOE web page and included in the vde newsletter at the first of the year three town hall listening sessions were held in January that featured feedback from parents and School staff both in writing

251and through verbal feedback during the virtual listening sessions the feedback was valuable and will inform implementation though only minor non-substantial changes were made to the draft document based on the feedback received this table includes the guideline component Associated feedback and the response from the agency as you will see in the guidelines draft that is included in your materials content is organized to include an introduction terminology section information detailing and clear expectations for administration of the parent notification within 24 hours of the school connected overdose model action plans for both response to the school connected overdose and support of return to school following an overdose reporting collaboration requirements between school divisions law enforcement and the vooe and appendices that include sample Communications a sample website template and a curated pre-populated list of resources to support School

252Division divisions and implementation upon approval the agency will immediately shift into implementation and related support to school divisions to include data collection systems collaboration with dcgs technical assistance and training and preparation for the 2025 2026 school year and I am happy to take any questions at this time there any questions Dr Northern I'm just wondering because I'm I've got some educator friends who work in some places where they're experiencing this unfortunately um and you and the guidelines Clearly say confirmed but sometimes obviously it takes a long time for an event to get confirmed and when you send it you know it just it doesn't it takes a while right right are they are they supposed to issue a um just a medical emergency uh notification to parents when the um overdose has not been confirmed

253yet or is that up to them the guidelines you'll find state that we obviously encourage notification upon confirmation but the spirit of this is really to ensure student safety so the time it may take to confirm with medical and law enforcement really could be a window of time where parents can be communicating with their kids getting more awareness about something so the language you'll see is more about suspected so if a principal of a staff member has knowledge of a suspected overdose then they would be expected then administer that notification the confirmation would be ideal but we know that can take time um one example I've talked about as I've had discussions around this as a principal and a superintendent if you were to see a bunch of uh you know ambulances in front of

254a school in a division where I work we would send a notification to families because we know that would raise just some concern on their behalf um if a student had a weapon you would send out a notification about that I mean there there's a lot of process in between but making that awareness clear at the very beginning I think and the timing is important so we're encouraging our school divisions to get confirmation but also to make a notification when it's suspected okay the notification doesn't have to be specific you're saying it can just be we've had a suspected Med medical emergency whatever sometimes they don't want to say suspected when they're not right and in the in the template language that's including the guidelines really captures what you're referencing thank you yes ma'am Miss

255Holton yes ma'am uh there is legislation it has not been has been neither signed nor vetoed by the governor so as far as our work today is concerned we're acting based on the legislation from 2024 which which orderers create the guidelines um and of course if that legislation is signed in law by the Governor I I would assume we would need to to revisit that and reexamine our guidelines yeah and right now um there are consistencies there are areas where it does diverge um we haven't really done a deep analysis on that but again if that were to come to fruition I know the board would need to revisit this Miss hon at to help you with that question having worked pretty significantly on that legislation there is not much difference here it does and

256then we specifically speak to that uh in the guidelines document and and have received assurances I believe it's gone through review even with ogag to make sure that this complies with furpa um and that student privacy is protected appropriately other questions Miss mcferson how the funding models work with the various like the apprenti you know I mean these students are having to go from one place to the other and what Provisions we made for things like that and on the job training programs and certificate and whether we're bearing any but we can we can get we'll discuss that more in the in the next deck here oh okay okay thank you and you're doing the next one too yes ma'am I'm settled in here yeah uh item F yes M miss mcferson this is is

257what you've been waiting for our uh presentational work based learning and Madame President board members I am excited to provide this update to the high quality work-based learning definition experiences and evaluation criteria for the board's consideration for approval on First and final review today I'm sorry tomorrow is the day of the vote Yes my sorry my apologies oh there we go today's presentation is the culmination of months of board dialogue and deliberation related to work-based learning dating back to 2023 I remember this distinctly because the first presentation I gave to the board uh after coming to the agency I was extremely nervous and the board's desire to strengthen the Commonwealth's work-based learning guidelines in order to move forward with thoughtful integration of work-based learning into the state's accountability system I will provide the board with

258context and background and overview of our stakeholder engagement and the proposed definition experiences and evaluation criteria for highquality work-based learning Cod of Virginia speaks to the board's responsibility to Define highquality work-based learning the current definition the status quo definition used for this purpose is that high quality work-based learning uh consists of school coordinated workplace experiences related to students career goals and or interests integrated with instruction related to the core technical competencies of the CTE course and performed and partnership with local businesses the VOE currently identifies 12 experiences as high quality quality work based learning as we work to provide the board with a revised and improved definition highquality work based learning experien is align with meaningful student outcomes and evaluation criteria to guide divisions and establishment of highquality work-based learning the vdoe has collected extensive

259feedback that has been integrated into the edits as the work-based learning materials have iterated the board was presented with draft high quality work-based learning definitions and experiences at the December work session and in the meantime National and Virginia stakeholders have provided feedback to inform the development and revisions of the highquality workplace learning guidelines feedback Gathering sessions have been held across the Commonwealth with various key stakeholder groups including Workforce business industry postsecondary and K12 Partners video we engage with all of the stakeholders you see there on the slide which includes many from various agencies including the Virginia Workforce Development board uh groups of superintendents from certain regions our Virginia works Partners the Virginia Office of Education economics um the list goes on but you can see that on the slide as well as the governor student

260advisory which actually was one of my favorite groups to interact with students from across the state who also were able to give us some feedback on this as well the proposed definition of highquality work-based learning is a school coordinated experience within the workforce that includes sustained or intensive engagement with industry Professionals in high demand fields in which students apply coursework learning to a real world setting while at the same time building critical thinking collaboration communication creative thinking and citizenship 5c skills students practice their technical skills and durable skills on a sustained basis and these experiences accelerate learning toward the goal of earning a national certificate of completion an industry recognized credential Andor attain post-secondary credit and receive feedback and guidance through mentoring and supervision from industry professionals this definition captures the key characteristics of highquality

261work-based learning noted by stakeholders including sustained and intensive engagement with professionals high demand Fields attention to the 5c's earning of credentials Andor post-secondary credit and the critical mentoring feedback and guidance by industry professionals the high quality work-based learning experiences that are proposed include registered apprenticeship internship clinical experiences and school-based Enterprise entrepreneurship these experiences have been identified based on CTE completer data stakeholder feedback and the need to provide maximum flexibility to school divisions while still ensuring meaningful highquality experiences to students that will result in measurable outcomes prior to graduation apprenticeship includes both registered and youth registered apprenticeship internships include immersion supervised agricultural experiences as well as supervised work training programs such as project search and the Wilson Rehabilitation Center postsecondary Rehabilitation transition or perk program and clinical experiences will focus on health and medical science es

262school-based Enterprise and Entrepreneurship engage students in a supervised experience planning and managing their own business venture School base Enterprise also provides flexibility to divisions that may have limited access to business and industry Partners to provide some of those other experiences and we noted that in the feedback as recent as yesterday when we spoke to our rural superintendents the criteria for evaluation aligns with the proposed definition and reflects the key characteristics of a highquality work-based learning experience as you can see the checklist retains criteria for graduation requirements and for 3E Readiness as we have discussed continuing to count the existing 12 highquality work-based learning experiences for graduation requirements while we propose including the 3E Readiness work-based learning experiences apprenticeship internship clinical school-based Enterprise for inclusion in our accountability system ultimately we recogn recommend to the board

263that Virginia's highquality work-based learning experiences merge into those 3E Readiness experiences for both graduation requirements and accountability eventually and we recommend conversation on the timing of that phase in with the board in the coming months as we consider shifting immediately into implementation upon board approval we are already working to better leverage our CTE workflow to support School divisions business Workforce and postsecondary partners with training technical assistance and boots on the ground support to Galvanize connections and communities to move this work forward on behalf of students in the workforce we hope the board will consider integration of work-based learning into the school per performance and support framework in March as well as consider the timeline for the shift from graduation requirement work-based learning to the new higher standard of 3E work-based learning for students and with

264that I happy to take any questions are there any questions Miss Holton again thank you Dr Armstrong first question again I think there is legislation working its way through the general assembly if it is signed by the governor is this uh are the are these actions consistent with those legislation or will we have to redo it these are um they they were before they were introduced excellent um uh second question uh I see we have a letter in the public comment uh from somebody raising a concern that we're not that we've excluded externships we've provided for internships but apparently we've excluded externships I don't know what that even is but what is that right we're excluding them and what are they and are we received that feedback and then Dr Coons and I have had

265that conversation as well as reached out to some of our our experts in this and and populated that and our feedback um an internship is when you are being supervised and and doing the work whereas an externship and I'm not saying it's not valuable but for the purposes of of our 3E read a standard and work-based learning and exter internship is is more of a job shadowing than it is actually doing the work you're you're there learning from the outside but you're not being supervised to actually learn to do the work and and we we see in the data as well that the externships don't translate to those outcomes the way that an internship and those other experiences do so we considered it um is still a valuable career exploration activity which we've captured in

266our career exporation definition but not meeting that standard of the 3D wri thank you that helps clar and so we're for 3E Readiness we're requiring we're authorizing internships but not externships that's correct and then my last question is I'm I'm not still not compar whether where are the what are the differences between our 3E Readiness definitions and our graduation definitions so you'll see why why help me again this is the best SL to illustrate that in the evaluation criteria and there there are a lot there are many parallels that that are consistent between the graduation requirement high quality work based learning our new 3E but you'll see here that there were we have a minimum of 90 hours for 3E Readiness which while lower than what um was there before which is really Antiquated and

267based on the Carnegie 140 hour clock hour requirement is still higher than most States and we don't require that for the graduation the graduation work-based learning it depends on the experience um and some of them require different numbers of hours but 140 is kind of consistent among those uh the artifact soon toe artifact graduation requirement oh so the graduation requirements are higher than the 3 some of them are yes yes which is something we receive feedback from that 3E is higher than many other states do uh then than many states I won't say every state but then many states yes ma'am and you know the high demand field is another um part of our 3E definition I know we've had some discussion about that as well and then we're settling on high demand in this

268um the alignment to CT course is consistent across both and then it's it's going from providing opportunities to apply the 5c's to then completing an assessment that documents your attainment of the 5c's it's just a different level of rigor in the definition between the graduation requirement 3E um you're provide an opportunity to earn these things but to get the 3E credit or to make this a work-based learning experience for under 3E you earn that um you have that that is earned and then instead of providing the opportunity alone for these mentorships then you have document ation of a structured mhip with industry professional in sight so it's really the differences are just that that higher level of rigor and engagement between graduation requirement three I I appreciate it and I apologize for making you go

269through it again no no worries no wor I didn't go through that in detail to begin with so happy to do so any other questions M I'm sorry Dr Northern I was just gonna say thank you I found this um this chart very helpful I think when I was looking at it last time I couldn't get it straight in my mind mind and this is really cohesive and shows nicely how they coordinate um and I really do feel like what you guys have sketched out here with the bullets are high quality work workpace learning experiences um and I just ties into the earlier conversation of how important it'll be for us to have models and proof points um so that kids get enough time right to do a registered apprentiship and I mean these things

270all take time and they're going to be competing um and I would suggest that me uh the Cristo Ray model uh is a really good model for us to kind of learn from I'm actually doing an evaluation of it in my day job um and trying to figure out like how do they how do they make the time for kids to be in these corporate workplaces and still you know check all the boxes on the academic so there are some um models we could learn from thank you Dr Norther and I'll mention too I think SE time flexibility is really the oil that's going to make this portfolio run whether it's dual enrollment work based learning um that that's a a catalyst that's going to make all that happen more effectively thank you Dr Armstrong

271thank you all I think that's it okay so that takes us to um item G and so Mr Hansen that's you we will now go into a closed session which we hope to keep brief Mr Hansen would you make your motion I move that the Virginia Board of Education convenient a closed meeting for the following purposes consultation with legal counsel employed by or retained by a public body regarding specific legal matters requiring the provision of legal advice by such Council pursuant to section 2.2 3711 A8 of the code of Virginia and the legal councel to the Virginia Board of Education staff whose presence will Aid in the board's consideration of this matter participate in the closed meeting is there a second all those in favor um respond Dr Northern Miss Kilgore Mr Hansen mcferson Miss

272Holton I chair votes I we now on closed session thank you e e e Hansen you read our certification to return to open session please we convene today in a closed meeting pursuant to recorded vote on the motion above and in accordance with the Freedom of Information Act section 2.2-inch that to the best of each member's knowledge only public business matters lawfully exempt from open meeting requirements under this chapter and only such public business matters as were identified in the motion by which the closed meeting was convened were heard discussed or considered any member who believes there's a departure from these requirements show State prior to the vote indicating the substance of the departure that in his or her judgment has taken place the statement of the departure will be recorded in the minutes Dr

273Northern Miss Kilgore Mr Hanson M mcferson Miss Holton shair votes I um so that brings us out of close session officially and do you have any comments for tomorrow I mean for today not tomorrow it's not tomorrow yet uh okay so um I will take a motion to adjourn our uh work session this afternoon all those in favor we're adjourned e e